|
Download README.md from WithinUsAI/Nano.Deep.Reasoner.11m: direct link, hf CLI and curl.
- Browser
- Download file 2 kB
-
https://huggingface.co/WithinUsAI/Nano.Deep.Reasoner.11m/resolve/main/README.md
- Command line
-
hf download hf://WithinUsAI/Nano.Deep.Reasoner.11m/README.md
-
curl -L -o README.md https://huggingface.co/WithinUsAI/Nano.Deep.Reasoner.11m/resolve/main/README.md
2 kB
| license: mit | |
| library_name: pytorch | |
| tags: | |
| - causal-lm | |
| - decoder-only | |
| - reasoning | |
| - pretraining | |
| - think | |
| - thought | |
| - reasoning | |
| - code | |
| # Nano.Deep.Reasoner.11m | |
| A small decoder-only causal language model trained from scratch. | |
| ## Architecture | |
| - Parameters: 11,229,120 | |
| - Context length: 1096 | |
| - Hidden size: 360 | |
| - Layers: 6 | |
| - Attention heads: 8 | |
| - Intermediate size: 1440 | |
| - Vocabulary size: 4,096 | |
| ## Reasoning tokens | |
| - `<|input|>` | |
| - `<|think|>` | |
| - `<|thought|>` | |
| - `<|reasoning|>` | |
| - `<|answer|>` | |
| ## Objective | |
| True causal next-token prediction. | |
| For an input sequence: | |
| `x[0], x[1], x[2], ...` | |
| the model learns: | |
| `x[0] -> x[1]` | |
| `x[1] -> x[2]` | |
| `x[2] -> x[3]` | |
| and so on. | |
| ## Context | |
| Every individual training example is strictly limited to: | |
| `1095` content tokens + EOS | |
| and padded to exactly `1096` positions. | |
| No oversized example is intentionally split across separate training examples. | |
| ## Example selection | |
| Starting from Session 3, new examples are selected via a deterministic | |
| shuffled scan of the dataset (seeded, reproducible across runs) rather than | |
| raw sequential order, to avoid category/source concentration within a | |
| session. Sessions 1-2 (40,000 examples) were selected sequentially before | |
| this correction and remain part of the trained corpus. | |
| ## Dataset | |
| `Plans11/Organized_PreTrain_1k_Context` | |
| Categories exposed by the dataset include: | |
| - Think | |
| - Thought | |
| - Reasoning | |
| - Instruct | |
| - Chat | |
| - Tool_Calling | |
| - Code_Instruct | |
| ## Resumability | |
| Training state is persisted to Hugging Face, including: | |
| - model.safetensors | |
| - training_state.pt | |
| - tokenizer files | |
| - config.json | |
| - progress.json | |
| - seen_examples.jsonl | |
| - training_metadata.json | |
| Examples are tracked using SHA-256 content hashes. | |
| The tokenizer becomes immutable after its initial creation. | |
| ## Current progress | |
| - Unique examples trained: 60,000 | |
| - Global optimizer steps: 1,441 | |
| - Last session: 20,000 | |
| This is an experimental small language model and is not guaranteed to | |
| produce factually or logically correct outputs. | |