File size: 1,221 Bytes
b4c73ed 35a324f 5e87417 35a324f a8d18a1 5e87417 a8d18a1 5e87417 a8d18a1 5e87417 a8d18a1 5e87417 a8d18a1 5e87417 a8d18a1 5e87417 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | ---
license: mit
language:
- en
tags:
- generative-ai
- text-generation
- transformer
- pytorch
- gpt
library_name: pytorch
---
# Nexora-1
Nexora-1 is a compact GPT-style generative language model built from scratch using PyTorch.
It is a Transformer-based autoregressive language model trained on English Wikipedia text using next-token prediction.
The project was developed to understand the fundamental components behind GPT-style language models, including tokenization, embeddings, positional embeddings, self-attention, multi-head attention, Transformer blocks, autoregressive training, checkpointing, and text generation.
## Model Architecture
- **Architecture:** GPT-style Transformer
- **Transformer Layers:** 6
- **Attention Heads:** 8
- **Embedding Dimension:** 384
- **Context Length:** 256 tokens
- **Vocabulary Size:** 8,000
- **Dropout:** 0.1
- **Tokenization:** Byte-Level BPE
- **Framework:** PyTorch
## Training Dataset
Nexora-1 was pretrained on English Wikipedia text.
The training pipeline was configured to collect Wikipedia articles and create a text corpus for autoregressive language-model training.
Articles were separated using the special token:
```text
<|endofarticle|> |