File size: 9,376 Bytes
0b911c0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5c05b35
 
 
 
 
 
 
0b911c0
 
 
 
5c05b35
0b911c0
 
 
 
 
 
 
 
5c05b35
 
 
0b911c0
 
 
 
 
 
 
 
 
 
5c05b35
 
 
 
 
0b911c0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92a01b3
0b911c0
 
 
 
 
 
 
 
 
6a1425d
 
0b911c0
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
---
license: apache-2.0
language:
  - en
  - fr
library_name: gguf
pipeline_tag: text-classification
tags:
  - gguf
  - decision-model
  - system-one
  - ternary
  - games
  - rust
  - wasm
  - skycmd
  - passed-gate
datasets:
  - maxgfr/skycmd-decisions
  - maxgfr/skycmd-glm-labels
  - maxgfr/skycmd-games
  - maxgfr/skycmd-numeric
  - maxgfr/skycmd-text
---

# s6-micro

> **Passed its pre-registered gate.** On a second look: this is the second version of s6-micro. The first
> version ([s6-micro-v1](https://huggingface.co/maxgfr/s6-micro-v1)) failed the ECE guard of the
> same gate at 0.0821 against 0.08. This version was designed after that reading (twice the
> decision-training minutes, everything else the same) and its gate was read on the same test
> records ([`docs/s6-gen-prereg.md`](https://github.com/maxgfr/skycmd/blob/main/docs/s6-gen-prereg.md),
> amendments 12, 16 and 17), so its pass is not a first look at these sets. It reached **0.553**
> on `typed_decisions_bench` (gate 0.45), an ECE of **0.0588** over the test sources (guard 0.08)
> and s3-pico's score in every game. On 11 of the bench's 20 tasks it gives one answer to at least
> 95% of the items; answering each question with its most common training answer scores 0.4815.
> Its decision training was restarted once from its checkpoint after a technical failure, its
> last minutes, its DAgger rounds and its evaluation ran on a loaded machine, and its trainer
> binary is not v1's.

**One 2.4 MB ternary model for four games and for typed questions in free text: the larger
sibling of [s6-nano](https://huggingface.co/maxgfr/s6-nano).** 5,587,201 parameters. It reads a
state as text and answers typed questions (`choice`, `score`, `noul`) with probabilities
calibrated on validation, one forward pass per question. It plays the four skycmd games at the oracle's
level, and passed a publication gate fixed before training, on its second version.

- **Demo:** [huggingface.co/spaces/maxgfr/s6-micro](https://huggingface.co/spaces/maxgfr/s6-micro)
  (the games, and typed questions you can edit)
- **Code:** [github.com/maxgfr/skycmd](https://github.com/maxgfr/skycmd) (100 % Rust)
- **All results, failures included:** [results page](https://maxgfr.github.io/skycmd/results/#s6-micro-v2-passed-on-a-second-look)
- **Technical report:** [maxgfr.github.io/skycmd/paper](https://maxgfr.github.io/skycmd/paper/)
- **Pre-registration:** [docs/s6-gen-prereg.md](https://github.com/maxgfr/skycmd/blob/main/docs/s6-gen-prereg.md) (gates in §6, this version in amendments 12, 16 and 17)
- **Data:** trained on [skycmd-decisions](https://huggingface.co/datasets/maxgfr/skycmd-decisions), [skycmd-glm-labels](https://huggingface.co/datasets/maxgfr/skycmd-glm-labels), [skycmd-games](https://huggingface.co/datasets/maxgfr/skycmd-games), [skycmd-numeric](https://huggingface.co/datasets/maxgfr/skycmd-numeric), [skycmd-text](https://huggingface.co/datasets/maxgfr/skycmd-text); tested on [skycmd-decisions](https://huggingface.co/datasets/maxgfr/skycmd-decisions), [skycmd-zero-shot](https://huggingface.co/datasets/maxgfr/skycmd-zero-shot)
- **Every demo, with its verdict:** [the skycmd Space](https://huggingface.co/spaces/maxgfr/skycmd)

## Results

Test sets, at most 2,000 questions per source, ternary GGUF of the last DAgger round, full depth
(τ = 1); the other columns are the same records:

| set | s6-micro | s6-micro-v1 (failed) | s6-nano | reference |
|---|---:|---:|---:|---|
| `typed_decisions_bench` | **0.553** | 0.547 | 0.480 | majority class 0.398; the most common training answer of each question 0.4815; s3-pico 0.382, s4-micro 0.352, s5-lite 0.696 |
| MASSIVE intents, English | 0.531 | 0.518 | 0.425 | s3-pico 0.187 (see the limitations) |
| MASSIVE intents, French | 0.314 | 0.275 | 0.239 | s3-pico 0.117 |
| When2Call (test split of a trained source) | 0.401 | 0.430 | 0.404 | s3-pico 0.368; its most common training answer 0.359 |
| NOTAM subject / NOTAM multi (never trained on) | 0.092 / 0.353 | 0.052 / 0.394 | 0.055 / 0.493 | s3-pico 0.163 / 0.319 |
| all sources | 0.380 | 0.378 | 0.360 | |
| **ECE over all these sources** | **0.0588** | 0.0821 | 0.0479 | **the gate allows 0.08** |

- **ECE by source:** bench 0.0498, MASSIVE en 0.0715, MASSIVE fr 0.0300, When2Call 0.2165, NOTAM
  subject 0.0185, NOTAM multi 0.1409. The 95% bootstrap interval of the pooled ECE is
  [0.0526, 0.0687] (v1: [0.0746, 0.0900]). On When2Call the model is over-confident (mean
  confidence 0.612 for an accuracy of 0.401): it answers `request_for_info` to 1,405 of the 2,000
  items and `tool_call` to 45, where `tool_call` is the gold answer of 696.
- **In distribution:** Banking77 intents 0.763 (it is in the training data).

Closed loop (`sky-arena`, 30 episodes per game, the same seeds as s3-pico, full depth):

| game | result |
|---|---|
| Flappy Drone | 200 / 200 pipes, 0 crashes |
| Rescue Run | 100 % |
| Obstacle Dodge | 87 % (= oracle), 20 reflex vetoes |
| Doom-like | 100 % of levels, kill rate 0.77 (= oracle), 2 vetoes |

Latency, measured while other jobs loaded the machine (not comparable with the other models):
median 8,970 µs per decision natively over the test sources; in WASM under Node, a median of
61 ms on the long JSON states of 200 bench questions. WASM and native agree to 8.9e-7 on those 200
questions, with the same answer on all 200.

## Full depth

The file answers at full depth: all 6 block passes, every time. Its three exit thresholds are set
to 1 in the GGUF (`skycmd-s2.exit_threshold.{choice,score,noul}`), as amendment 13 of the
pre-registration defines a publication at full depth; the weights are those of the evaluated
file (the last DAgger round), byte for byte. Conformal risk control (α = 0.02) chose earlier
thresholds, which kept the four arena scores but added one Dodge veto (21 against 20); the
pre-registered rule allows none, so they were rejected. You can still pass another threshold at
run time (`setExitThreshold` in the browser).

## Architecture

The s4-micro shape (`skycmd-s2` GGUF tag): question and options form a static stream encoded once
and cached; the state goes through 2 weight-shared blocks looped three times, with exits after 2,
4 and 6 block passes; options never attend to each other; ternary weights trained with QAT. d 256,
4 heads, vocabulary 4,096, windows of 384 state tokens and 128 question tokens. The pre-registered
ablation adopted none of the s6 additions (Parallax, Muon v2, XSA and others), so this model uses
softmax attention and AdamW.

## Limitations

- **A second look.** v1 failed the ECE guard and this version was designed after reading that
  result, on the same test records. Its pass would be stronger on test sets read for the first
  time.
- A modest generalist: 0.553 on the bench, against about 0.69 for 17M models (s5-lite 0.696,
  oscar-1-17m 0.6775). On 11 of the 20 bench tasks it answers one constant to at least 95% of the
  items.
- It cannot read NOTAMs (0.092 on the subject question), and on When2Call it stays at 0.401 and
  rarely chooses to call a tool.
- The English MASSIVE score is not strictly zero-shot: 206 of the 2,000 test utterances also
  appear in the training split, 201 of them in a source derived from MASSIVE. None of the 1,000
  French ones does.
- One run, one seed. The decision training died once on a file-access error and was resumed from
  its checkpoint (the steps after it were done again); the end of the run shared the machine with
  other jobs.
- The games are civil drone scenarios and a Doom-like shooter drawn with Freedoom art; the model
  only advises and coded rules always keep a veto. This is a demo and a tutorial, not flight
  software.

## Usage

Runs in the skycmd engine only (browser, Node, Python, CLI, server):

```sh
cargo install --git https://github.com/maxgfr/skycmd sky-serve
skycmd decide --model s6-micro.gguf --request req.json --pretty
```

Request format and bindings: [install guide](https://maxgfr.github.io/skycmd/install/).

## Training

From scratch in Rust (mlx-rs) on an Apple M5, in a braked mode, so the budgets are working
minutes: masked-language pretraining in float (180 minutes, 31,700 steps, 0.52B tokens, 93 per
parameter; the pretraining of v1, not run again), then decision training with ternary QAT after
a 10 % float warm-up (360 minutes, uncompiled, 65,471 steps of 64, learning rate ×0.5 chosen by a
pre-registered pilot) and three DAgger rounds on the games (20 minutes each: 3,822, 4,673 and
4,969 steps). Data: the typed-decision sets of Hugging Face, classic datasets converted into
typed questions, game states labelled by exact oracles and a numeric curriculum, the same as v1
and s6-nano. Pretraining text: SimpleStories, TinyStories, FineWeb and the non-web parts of
Dolma 3 and TxT360. Soft labels on 747,077 training questions come from GLM-5.3-flash (open
weights, MIT, called through the z.ai API), kept only where they agree with the gold answer; no
test question was sent. The teacher's weights are open (MIT); the labels and their
publication are described in
[DATA_LICENSES.md](https://github.com/maxgfr/skycmd/blob/main/DATA_LICENSES.md), with every
other license.

## Files

- `s6-micro.gguf`: ternary weights, tokenizer, temperatures, and exit thresholds set to 1 (full
  depth). 2,420,032 bytes, sha256 `f66d702121e85e678e9b6ea73e9c01433e968b3a3734e8f68ac58f4cee4340ff`.