File size: 1,997 Bytes
01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 5105147 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 2c2370d 01898e6 5105147 2c2370d 01898e6 2c2370d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | ---
license: mit
library_name: pytorch
tags:
- causal-lm
- decoder-only
- reasoning
- pretraining
- think
- thought
- reasoning
- code
---
# Nano.Deep.Reasoner.11m
A small decoder-only causal language model trained from scratch.
## Architecture
- Parameters: 11,229,120
- Context length: 1096
- Hidden size: 360
- Layers: 6
- Attention heads: 8
- Intermediate size: 1440
- Vocabulary size: 4,096
## Reasoning tokens
- `<|input|>`
- `<|think|>`
- `<|thought|>`
- `<|reasoning|>`
- `<|answer|>`
## Objective
True causal next-token prediction.
For an input sequence:
`x[0], x[1], x[2], ...`
the model learns:
`x[0] -> x[1]`
`x[1] -> x[2]`
`x[2] -> x[3]`
and so on.
## Context
Every individual training example is strictly limited to:
`1095` content tokens + EOS
and padded to exactly `1096` positions.
No oversized example is intentionally split across separate training examples.
## Example selection
Starting from Session 3, new examples are selected via a deterministic
shuffled scan of the dataset (seeded, reproducible across runs) rather than
raw sequential order, to avoid category/source concentration within a
session. Sessions 1-2 (40,000 examples) were selected sequentially before
this correction and remain part of the trained corpus.
## Dataset
`Plans11/Organized_PreTrain_1k_Context`
Categories exposed by the dataset include:
- Think
- Thought
- Reasoning
- Instruct
- Chat
- Tool_Calling
- Code_Instruct
## Resumability
Training state is persisted to Hugging Face, including:
- model.safetensors
- training_state.pt
- tokenizer files
- config.json
- progress.json
- seen_examples.jsonl
- training_metadata.json
Examples are tracked using SHA-256 content hashes.
The tokenizer becomes immutable after its initial creation.
## Current progress
- Unique examples trained: 60,000
- Global optimizer steps: 1,441
- Last session: 20,000
This is an experimental small language model and is not guaranteed to
produce factually or logically correct outputs.
|