| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - generative-ai |
| - text-generation |
| - transformer |
| - pytorch |
| - gpt |
| library_name: pytorch |
| --- |
| |
| # Nexora-1 |
|
|
| Nexora-1 is a compact GPT-style generative language model built from scratch using PyTorch. |
|
|
| It is a Transformer-based autoregressive language model trained on English Wikipedia text using next-token prediction. |
|
|
| The project was developed to understand the fundamental components behind GPT-style language models, including tokenization, embeddings, positional embeddings, self-attention, multi-head attention, Transformer blocks, autoregressive training, checkpointing, and text generation. |
|
|
| ## Model Architecture |
|
|
| - **Architecture:** GPT-style Transformer |
| - **Transformer Layers:** 6 |
| - **Attention Heads:** 8 |
| - **Embedding Dimension:** 384 |
| - **Context Length:** 256 tokens |
| - **Vocabulary Size:** 8,000 |
| - **Dropout:** 0.1 |
| - **Tokenization:** Byte-Level BPE |
| - **Framework:** PyTorch |
|
|
| ## Training Dataset |
|
|
| Nexora-1 was pretrained on English Wikipedia text. |
|
|
| The training pipeline was configured to collect Wikipedia articles and create a text corpus for autoregressive language-model training. |
|
|
| Articles were separated using the special token: |
|
|
| ```text |
| <|endofarticle|> |
