---
license: mit
language:
  - en
tags:
  - generative-ai
  - text-generation
  - transformer
  - pytorch
  - gpt
library_name: pytorch
---

# Nexora-1

Nexora-1 is a compact GPT-style generative language model built from scratch using PyTorch.

It is a Transformer-based autoregressive language model trained on English Wikipedia text using next-token prediction.

The project was developed to understand the fundamental components behind GPT-style language models, including tokenization, embeddings, positional embeddings, self-attention, multi-head attention, Transformer blocks, autoregressive training, checkpointing, and text generation.

## Model Architecture

- **Architecture:** GPT-style Transformer
- **Transformer Layers:** 6
- **Attention Heads:** 8
- **Embedding Dimension:** 384
- **Context Length:** 256 tokens
- **Vocabulary Size:** 8,000
- **Dropout:** 0.1
- **Tokenization:** Byte-Level BPE
- **Framework:** PyTorch

## Training Dataset

Nexora-1 was pretrained on English Wikipedia text.

The training pipeline was configured to collect Wikipedia articles and create a text corpus for autoregressive language-model training.

Articles were separated using the special token:

```text
<|endofarticle|>