| --- |
| datasets: |
| - jgalego/eca-queiros |
| language: pt |
| library_name: pytorch |
| license: mit |
| model_name: eca |
| pipeline_tag: text-generation |
| tags: |
| - portuguese |
| - literature |
| - from-scratch |
| --- |
| |
| # Eça |
|
|
| <p align="center"><img src="assets/logo.svg" alt="Eça logo: a cream letter C with a red cedilla on a blueprint tile" width="200"></p> |
|
|
| Small GPT pre-trained from scratch on the collected works of the Portuguese novelist Eça de Queirós, from the [jgalego/eca-queiros](https://huggingface.co/datasets/jgalego/eca-queiros) dataset. Byte-level BPE tokenizer trained on the same text. `eca.py` in this repo holds the model and doubles as the loader. |
|
|
| > "As enxergas rijas fazem as almas fortes." (*A Cidade e as Serras*) |
|
|
| ## 🚀 Usage |
|
|
| ``` |
| uv run https://huggingface.co/jgalego/eca/resolve/main/eca.py write "A estação de Ovar" |
| ``` |
|
|
| From Python, with `eca.py` on the path: |
|
|
| ```python |
| from eca import load, write |
| |
| model, tokenizer = load("jgalego/eca") |
| print(write(model, tokenizer, "A estação de Ovar", count=100)) |
| ``` |
|
|
| ## 🏋️ Training |
|
|
| | | | |
| |---|---| |
| | Model | Decoder-only transformer, 5.3M parameters, 6 layers, width 256, 4 heads, context 256, vocabulary 2048 | |
| | Data | First 98% of the text: 3,505,536 tokens. Random windows of 256 tokens. | |
| | Steps | 3,000, batch size 64, AdamW, lr 0.001 with warmup and cosine decay, dropout 0.1 | |
| | Selection | Weights from the step with the best validation score (step 3,000) | |
| | Hardware | NVIDIA A10G, 0.03 h | |
|
|
| ## 📊 Results |
|
|
| Validation is the last 2% of the text, 205,864 characters (70,135 tokens), scored in non-overlapping windows of 256 tokens. Bits per character is comparable across tokenizers. |
|
|
| | Loss per token | Bits per character | |
| |---|---| |
| | 3.3384 | 1.6408 | |
|
|
| ## ⚠️ Limitations |
|
|
| - Trained on about 10M characters, so it imitates the style of the novels and has no facts or reasoning to speak of. |
| - Spelling follows 19th-century Portuguese, not the current orthography. |
| - The validation text is the end of one book, so the score reflects that book more than the whole corpus. |
