metadata
license: mit
language:
- en
tags:
- generative-ai
- text-generation
- transformer
- pytorch
- gpt
library_name: pytorch
Nexora-1
Nexora-1 is a compact GPT-style generative language model built from scratch using PyTorch.
It is a Transformer-based autoregressive language model trained on English Wikipedia text using next-token prediction.
The project was developed to understand the fundamental components behind GPT-style language models, including tokenization, embeddings, positional embeddings, self-attention, multi-head attention, Transformer blocks, autoregressive training, checkpointing, and text generation.
Model Architecture
- Architecture: GPT-style Transformer
- Transformer Layers: 6
- Attention Heads: 8
- Embedding Dimension: 384
- Context Length: 256 tokens
- Vocabulary Size: 8,000
- Dropout: 0.1
- Tokenization: Byte-Level BPE
- Framework: PyTorch
Training Dataset
Nexora-1 was pretrained on English Wikipedia text.
The training pipeline was configured to collect Wikipedia articles and create a text corpus for autoregressive language-model training.
Articles were separated using the special token:
<|endofarticle|>