metadata
datasets:
- jgalego/eca-queiros
language: pt
library_name: pytorch
license: mit
model_name: eca
pipeline_tag: text-generation
tags:
- portuguese
- literature
- from-scratch
Eça
Small GPT pre-trained from scratch on the collected works of the Portuguese novelist Eça de Queirós, from the jgalego/eca-queiros dataset. Byte-level BPE tokenizer trained on the same text. eca.py in this repo holds the model and doubles as the loader.
"As enxergas rijas fazem as almas fortes." (A Cidade e as Serras)
🚀 Usage
uv run https://huggingface.co/jgalego/eca/resolve/main/eca.py write "A estação de Ovar"
From Python, with eca.py on the path:
from eca import load, write
model, tokenizer = load("jgalego/eca")
print(write(model, tokenizer, "A estação de Ovar", count=100))
🏋️ Training
| Model | Decoder-only transformer, 5.3M parameters, 6 layers, width 256, 4 heads, context 256, vocabulary 2048 |
| Data | First 98% of the text: 3,505,536 tokens. Random windows of 256 tokens. |
| Steps | 3,000, batch size 64, AdamW, lr 0.001 with warmup and cosine decay, dropout 0.1 |
| Selection | Weights from the step with the best validation score (step 3,000) |
| Hardware | NVIDIA A10G, 0.03 h |
📊 Results
Validation is the last 2% of the text, 205,864 characters (70,135 tokens), scored in non-overlapping windows of 256 tokens. Bits per character is comparable across tokenizers.
| Loss per token | Bits per character |
|---|---|
| 3.3384 | 1.6408 |
⚠️ Limitations
- Trained on about 10M characters, so it imitates the style of the novels and has no facts or reasoning to speak of.
- Spelling follows 19th-century Portuguese, not the current orthography.
- The validation text is the end of one book, so the score reflects that book more than the whole corpus.