| --- |
| library_name: pytorch |
| pipeline_tag: text-generation |
| language: |
| - en |
| datasets: |
| - roneneldan/TinyStories |
| tags: |
| - safetensors |
| - custom-code |
| --- |
| |
| # Tiny CED |
|
|
| Tiny CED is a small custom PyTorch encoder-decoder language model trained from |
| scratch on the English [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) |
| dataset. This package contains inference-only weights in Safetensors format. |
|
|
| This is not a Transformers `PreTrainedModel`. Load it with the included |
| `model.py` and `safetensors.torch.load_model` as shown below. |
|
|
| ## Model details |
|
|
| | Item | Value | |
| | --- | ---: | |
| | Independent parameters | 19,667,712 | |
| | Vocabulary size | 8,192 | |
| | Hidden size | 384 | |
| | Encoder layers | 4 | |
| | Decoder layers | 4 | |
| | Attention heads | 6 | |
| | Feed-forward size | 1,024 | |
| | Local attention window | 64 | |
| | Trained context length | 512 | |
| | Weight dtype | FP32 | |
|
|
| The token embedding and output head are tied. `model.safetensors` was exported |
| with `safetensors.torch.save_model` so the shared tensor is stored only once. |
|
|
| ## Usage |
|
|
| Install the runtime dependencies: |
|
|
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| Run generation on CPU: |
|
|
| ```bash |
| python generate.py \ |
| --device cpu \ |
| --text "Once upon a time, a little rabbit" \ |
| --max-new-tokens 100 |
| ``` |
|
|
| Use `--device cuda` when CUDA is available. Set `--temperature 0` for greedy |
| decoding. |
|
|
| ## Training and evaluation |
|
|
| The exported weights come from the existing `best.pt` checkpoint; no retraining |
| was performed during export. |
|
|
| - Optimizer: AdamW |
| - Peak learning rate: 3e-4 |
| - Best-checkpoint training tokens: 40,004,782 |
| - Best-checkpoint step: 5,856 |
| - Validation loss: 1.896527 |
| - Validation perplexity: 6.662715 |
|
|
| Validation used the TinyStories validation split and the same 8,192-token BPE |
| tokenizer included in this repository. |
|
|
| ## Files |
|
|
| - `model.safetensors`: inference weights |
| - `model.py`: exact custom PyTorch architecture |
| - `config.json`: architecture and token IDs |
| - `tokenizer.json`: BPE tokenizer |
| - `generate.py`: minimal generation example |
|
|
| ## Limitations |
|
|
| This is a small research model trained only on synthetic English children's |
| stories. It may generate incorrect, repetitive, biased, unsafe, or otherwise |
| inappropriate text. It is not suitable for factual, safety-critical, or |
| production use without further evaluation. |
|
|
| The source project did not specify a license for its code or weights. Choose an |
| appropriate license and confirm the training-data terms before publishing this |
| package publicly. TinyStories is listed as CDLA-Sharing-1.0 on its dataset page. |
|
|
