File size: 1,997 Bytes
01898e6
 
 
 
 
 
 
2c2370d
01898e6
 
 
2c2370d
01898e6
 
 
 
2c2370d
01898e6
 
 
 
2c2370d
01898e6
 
 
 
2c2370d
01898e6
2c2370d
01898e6
 
 
 
 
 
 
2c2370d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
01898e6
2c2370d
01898e6
5105147
 
 
 
 
 
 
 
2c2370d
01898e6
 
 
2c2370d
01898e6
2c2370d
 
 
 
 
 
 
01898e6
2c2370d
01898e6
2c2370d
01898e6
2c2370d
 
 
 
 
 
 
01898e6
2c2370d
01898e6
2c2370d
01898e6
2c2370d
01898e6
5105147
 
2c2370d
01898e6
2c2370d
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
---
license: mit
library_name: pytorch
tags:
- causal-lm
- decoder-only
- reasoning
- pretraining
- think
- thought
- reasoning
- code
---

# Nano.Deep.Reasoner.11m

A small decoder-only causal language model trained from scratch.

## Architecture

- Parameters: 11,229,120
- Context length: 1096
- Hidden size: 360
- Layers: 6
- Attention heads: 8
- Intermediate size: 1440
- Vocabulary size: 4,096

## Reasoning tokens

- `<|input|>`
- `<|think|>`
- `<|thought|>`
- `<|reasoning|>`
- `<|answer|>`

## Objective

True causal next-token prediction.

For an input sequence:

`x[0], x[1], x[2], ...`

the model learns:

`x[0] -> x[1]`

`x[1] -> x[2]`

`x[2] -> x[3]`

and so on.

## Context

Every individual training example is strictly limited to:

`1095` content tokens + EOS

and padded to exactly `1096` positions.

No oversized example is intentionally split across separate training examples.

## Example selection

Starting from Session 3, new examples are selected via a deterministic
shuffled scan of the dataset (seeded, reproducible across runs) rather than
raw sequential order, to avoid category/source concentration within a
session. Sessions 1-2 (40,000 examples) were selected sequentially before
this correction and remain part of the trained corpus.

## Dataset

`Plans11/Organized_PreTrain_1k_Context`

Categories exposed by the dataset include:

- Think
- Thought
- Reasoning
- Instruct
- Chat
- Tool_Calling
- Code_Instruct

## Resumability

Training state is persisted to Hugging Face, including:

- model.safetensors
- training_state.pt
- tokenizer files
- config.json
- progress.json
- seen_examples.jsonl
- training_metadata.json

Examples are tracked using SHA-256 content hashes.

The tokenizer becomes immutable after its initial creation.

## Current progress

- Unique examples trained: 60,000
- Global optimizer steps: 1,441
- Last session: 20,000

This is an experimental small language model and is not guaranteed to
produce factually or logically correct outputs.