{ "kind": "hub", "title": "skycmd", "tagline": "Every skycmd demo in one place. Pick a model to open its demo, which says first whether the model passed or failed its gate.", "notes": [ "20 models, each with the demo of its own Space: 5 passed their gate, 10 failed it and are published as results, and 5 were published without a gate.", "Each demo opens with its model's verdict, as in its own Space. The badges of this list are those of the site's Models page, which also mark a pass on a second look or on a generated bench only.", "Everything runs in your browser: a Rust inference engine compiled to WebAssembly reads the model files of this Space, and a model's files are downloaded only when its demo is opened. Nothing is sent to a server.", "Each demo links to its model's own Space. The project site, maxgfr.github.io/skycmd, has the same demos with the results behind them." ], "spaces": [ { "config": { "model": "s6-nano", "title": "s6-nano", "tagline": "A 715 KB ternary decision model that plays four games at the oracle's level and answers typed questions in free text, above the majority class of the typed-decisions bench.", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-nano", "notes": [ "Published at full depth: its conformal early-exit thresholds were rejected by the pre-registered arena check.", "0.480 on the typed-decisions bench, where the majority class is 0.398 and s5-lite reaches 0.696: expect mistakes on questions in free text.", "It passed its gate, but a constant answer per task passes it too: answering each question with its most common training answer scores 0.4815 on the bench, and on 17 of the 20 bench tasks s6-nano gives one answer to at least 95% of the items (docs/s6-nano2-study.md)." ], "game": null, "gate": "passed", "gate_kind": null, "gate_note": "0.480 on the typed-decisions bench, above the majority class (0.398) that its gate asked it to beat, an ECE of 0.0479 (guard 0.08) and no game more than 5% below s3-pico. A constant answer per task passes this gate too: answering each question with its most common training answer scores 0.4815", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#skycmd-s6-s6-nano-s6-micro-general-purpose-decisions-and-games-pre-registered" }, "models": [ "s6-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-nano" }, { "config": { "model": "s6-micro", "title": "s6-micro", "tagline": "A 2.4 MB ternary decision model that plays four games at the oracle's level and answers typed questions in free text: 0.553 on the typed-decisions bench. Its second version passed the gate that the first one failed, on the same test records.", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-micro", "notes": [ "A second look: the first version (s6-micro-v1) failed the ECE guard at 0.0821 against 0.08. This version was designed after that reading (twice the decision-training minutes) and judged on the same test records: 0.553 on the bench (gate 0.45), ECE 0.0588, every game at s3-pico's score (docs/s6-gen-prereg.md, amendment 17).", "Published at full depth: its conformal early-exit thresholds were rejected by the pre-registered arena check (one extra Dodge veto).", "Expect mistakes on questions in free text: 0.401 on When2Call, where it rarely chooses to call a tool, and 0.092 on NOTAM subjects. On 11 of the 20 bench tasks it gives one answer to at least 95% of the items." ], "game": null, "gate": "passed", "gate_kind": null, "gate_note": "Second version: 0.553 on the typed-decisions bench (gate 0.45) and ECE 0.0588 (guard 0.08). The first version failed the ECE guard at 0.0821, and this one was designed after that reading and judged on the same test records, so its pass is a second look", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#skycmd-s6-s6-nano-s6-micro-general-purpose-decisions-and-games-pre-registered" }, "models": [ "s6-micro" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-micro" }, { "config": { "model": "s6-secrets-pico", "title": "s6-secrets-pico", "tagline": "A 250 KB ternary secret detector that passes its pre-registered gate on a synthetic bench and fails on real files: published as a result, not as a scanner.", "kind": "secrets", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-secrets-pico", "notes": [ "On a synthetic bench of file types it never saw, it beats a gitleaks-style rule scanner: secret 0.9990 against 0.977, service 0.9795 against 0.787. The bench comes from our own generator.", "On real files it fails: on 719 source, configuration and documentation files with no secret, it flags 59.8% of the files at p ≥ 0.9 and 49.4% at p ≥ 0.999 (github.com/maxgfr/secretgate/pull/19). Do not use it to scan real files.", "The line answer is weak (0.6859 on the bench): do not rely on it.", "The presets are texts of the generator's bench, and their keys are fake, generated in each provider's public format with a random body. Everything runs in your browser: nothing you paste is sent anywhere." ], "game": null, "gate": "passed", "gate_kind": null, "gate_note": "On its synthetic bench, from our own generator, it beat a gitleaks-style rule scanner: secret 0.9990 against 0.977, service 0.9795 against 0.787. Measured after publication on 719 real files with no secret, it flags 59.8% of them at p ≥ 0.9: it is not a scanner, do not use it on real files", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-secrets-pico-api-key-and-secret-detection-pre-registered" }, "models": [ "s6-secrets-pico" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-secrets-pico" }, { "config": { "model": "s6-micro-v1", "title": "s6-micro-v1", "tagline": "Failed its pre-registered gate, published as a result: a 2.4 MB ternary model for four games and typed questions, 0.547 on the typed-decisions bench but ECE 0.0821 against a guard of 0.08.", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-micro-v1", "notes": [ "The gate (docs/s6-gen-prereg.md §6): at least 0.45 on the typed-decisions bench, ECE at most 0.08, no game more than 5% below s3-pico. It reached 0.547 and plays every game at s3-pico's score, but its ECE is 0.0821 at full depth: it failed, and is recommended for no use.", "Published at full depth (its three exit thresholds set to 1). Its four-times-smaller sibling s6-nano passed the same gate, and so did the second version, s6-micro, on a second look at the same test records.", "Expect mistakes on questions in free text: 0.430 on When2Call, 0.052 on NOTAM subjects." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It missed its calibration guard: ECE 0.0821 against 0.08 at full depth (0.547 on the typed-decisions bench, where the gate asked for at least 0.45)", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#skycmd-s6-s6-nano-s6-micro-general-purpose-decisions-and-games-pre-registered" }, "models": [ "s6-micro-v1" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-micro-v1" }, { "config": { "model": "s6-ear-nano", "title": "s6-ear-nano", "tagline": "Failed three of its four pre-registered gates, published as a result: a 899 KB ternary model that hears the last 10 s of audio and answers typed questions about it.", "kind": "ear", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-ear-nano", "notes": [ "End of turn 0.5681 against 0.9263 for Smart Turn v3.2 on the same clips (fail; Pipecat's report; the v1 harness read 0.8518); 12 Speech Commands keywords 0.8885 against 0.90 (fail); zero-shot keywords 0.2100 in English (pass, bar 0.20) and 0.1330 in French (fail).", "Its control, trained the same way without the Mimi pretraining, is more accurate on all four sets: the pretraining did not help in this run. The control is not published.", "The demo answers at full depth, where the gates were read; the file keeps the exit thresholds its training run stored (choice 0.5, score 0.9, noul 0.68)." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It missed three of its four gates: end of turn 0.5681 against 0.9263 for Smart Turn v3.2, keywords 0.8885 against 0.90, French zero-shot keywords 0.1330 against 0.20; its control model, trained without the Mimi pretraining, did better on all four sets", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-ear-nano-first-run-gates-failed" }, "models": [ "s6-ear-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-ear-nano" }, { "config": { "model": "s6-ear2-nano", "title": "s6-ear2-nano", "tagline": "A 632 KB ternary end-of-turn detector: from the last 4 s of audio it says whether the speaker has finished. It passed its pre-registered gate: 0.8993 on the 31,527 test clips of Smart Turn v3.2, where Smart Turn itself is at 0.9260.", "kind": "turn", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-ear2-nano", "notes": [ "The gate: accuracy 0.8993 (bars 0.85 and Smart Turn's 0.9260 minus 0.08); at its cautious threshold a false cut-off on 0.0847 of the not-finished clips (bar 0.20) with an END recall of 0.8627 (bar 0.80); 2.39 ms per decision in WASM (bar 5 ms) from 631,936 bytes.", "It is a student of Smart Turn v3.2 (Pipecat, BSD-2-Clause, 8.7 MB, about 32 ms per decision natively), trained on Smart Turn's public training set; it cannot be more right than the labels it imitates.", "On real conversations it is far weaker, like its teacher: on eot-bench its AUC is 0.7299 (Smart Turn 0.7838), and at its cautious threshold it catches about half of the real ends of turn (0.5253). Use it with a fallback timer, as this demo does.", "Weakest languages on the test: Vietnamese 0.7749, Marathi 0.7907, Bengali 0.7950. Most test clips are synthetic speech. The official test was partly in-sample for design choices (the 4 s window), never for the weights." ], "game": null, "gate": "passed", "gate_kind": null, "gate_note": "An end-of-turn detector (the last 4 s of audio): 0.8993 on the 31,527 test clips of Smart Turn v3.2 (bar 0.85; the teacher measured 0.9260 on the same clips), false cut-off 0.0847 and END recall 0.8627 at the validation threshold tau85, 2.39 ms per decision in WASM and 631,936 bytes. On real conversations (eot-bench) it and its teacher are far weaker, and at tau85 it answers \"finished\" on 0.5253 of the real ends of turn", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-ear2-nano-s6-ear-v2-an-end-of-turn-student-of-smart-turn-pre-registered-gate-passed" }, "models": [ "s6-ear2-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-ear2-nano" }, { "config": { "model": "s6-ear3-nano", "title": "s6-ear3-nano", "tagline": "Failed its pre-registered gate, published as a result: s6-ear2-nano's 632 KB end-of-turn graph trained again on real two-speaker conversations and on the silences this demo asks at (s6-ear v3). On eot-bench, real speech addressed to an agent, read 0.2 s into each pause, its AUC is 0.7197, where v2 reads 0.7448 and Smart Turn 0.7873. The detector to use stays s6-ear2-nano.", "kind": "turn", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-ear3-nano", "notes": [ "The gate (docs/s6-ear3-prereg.md §9): E1 asked for an AUC at least 0.02 above v2's on eot-bench, 0.2 s into each pause (16,230 cuts, 14 languages), with the lower bound of a paired bootstrap above 0: −0.0252 [−0.0336, −0.0167], so E1 fails. E2 (recall at a false cut-off of 0.10, −0.0563 [−0.0832, −0.0315]) and E3 (the 13 languages other than English, −0.0209 [−0.0308, −0.0111], a lower bound above −0.02 asked) fail too. A1 holds (0.8978 on Smart Turn's 31,527 test clips, bar 0.8893), and so do A2 (false cut-off 0.0881, END recall 0.8597 at τ85) and G3 (2.39 ms in WASM, 631,936 bytes, v2's tensor list).", "What it learned: on the 113 held-out sessions of its own corpus, human-to-human conversations (reported, not gated), its AUC is 0.6878 against v2's 0.6031. By the reading its pre-registration fixed before the run, the human-to-human signal was learned and does not transfer to speech addressed to an agent. In English, the language of its conversations, eot-bench reads 0.7375 against v2's 0.7737.", "Its labels come from the other speaker's channel (end of turn when the other takes the floor within 4 s, hold when the same speaker resumes first), never from a generative model; before training they agreed with a second detector, Silero VAD, on 0.9509 of the validation records (bar 0.85). The corpus is otoSpeech (gated, CC BY 4.0), credited on the model card; no record derived from it is published.", "Disclosure: v3 was designed after v2's eot-bench reading, which motivated the headline test, the bar outside English and the expected per-language effects; its label cross-check and caps were amended after a measurement on five training sessions, before any validation or test session was read. One training run, one seed." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It failed E1, E2 and E3: on eot-bench (real human-to-agent speech, 14 languages) read 0.2 s into each pause, its AUC is 0.7197 against v2's 0.7448 on the same cuts (−0.0252 [−0.0336, −0.0167], at least +0.02 with a lower bound above 0 asked), its recall at a false cut-off of 0.10 is 0.0563 below v2's, and outside English it loses 0.0209 [−0.0308, −0.0111] (a lower bound above −0.02 asked); Smart Turn reads 0.7873. A1 (0.8978 on Smart Turn's test clips, bar 0.8893), A2 and G3 hold. On held-out human-to-human conversations (reported) it reads 0.6878 against v2's 0.6031: the human-to-human signal does not transfer to agents. v2 (s6-ear2-nano) stays the line's file", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-ear3-nano-s6-ear-v3-an-end-of-turn-detector-trained-on-real-conversations-pre-registered-gate-failed" }, "models": [ "s6-ear3-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-ear3-nano" }, { "config": { "model": "s6-cast-pico", "title": "s6-cast-pico", "tagline": "Failed its pre-registered gate, published as a result: a 143 KB ternary forecaster (nine deciles) that scores nGMASE 0.9471 and nWQL 0.6646 on GIFT-Eval, where the gate asked for 0.838 and 0.587.", "kind": "forecast", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-cast-pico", "notes": [ "The gate was to beat the training-free FLAIR on GIFT-Eval (nGMASE ≤ 0.838 and nWQL ≤ 0.587), with the goal of beating TinyCast (0.774 / 0.545). It met neither, with or without flip (0.9099 / 0.6417 with flip).", "It beats the seasonal naive on WQL in 84 of 97 configurations, but on MASE only in 52; on MASE, medium and long horizons are worse than the seasonal naive.", "s6-cast-pico starts unticked: tick it to compare it with TinyCast and the seasonal naive on the same series." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "On GIFT-Eval it scored nGMASE 0.9471 and nWQL 0.6646, where the gate asked for 0.838 and 0.587", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-cast-pico-probabilistic-forecasting-pre-registered-gate-failed" }, "models": [ "s6-cast-pico" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-cast-pico" }, { "config": { "model": "s6-cast2-pico", "title": "s6-cast2-pico", "tagline": "Failed its pre-registered gate, published as a result: a 149 KB ternary forecaster (nine deciles) with a periodicity front end that scores nGMASE 0.8847 and nWQL 0.6190 on GIFT-Eval, where its gate asked for 0.838 and 0.587. Better than its first version, behind TinyCast.", "kind": "forecast", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-cast2-pico", "notes": [ "Failed its gate: G1 asked for nGMASE ≤ 0.838 and nWQL ≤ 0.587 on GIFT-Eval without flip, the bars of the training-free FLAIR; it scored 0.8847 and 0.6190 (0.8692 / 0.6071 with flip). G2 holds: its MASE ratio to the seasonal naive is under 1.00 at each term (short 0.833, medium 0.928, long 0.988). The Chronos-ZS gate was dropped before training (its harness did not reproduce the published TinyCast); there it reads 0.9014 / 0.7143, reported, against 0.8598 / 0.6928 for our TinyCast port without flip.", "It improves on s6-cast-pico (0.9471 / 0.6646) and stays behind our TinyCast port (0.7918 / 0.5622 without flip) on MASE in every term, band and domain. GIFT-Eval is partly in-sample for its design choices, never for its weights. By its pre-registration the s6-cast line stops here: TinyCast is the forecaster to use.", "Measured on one synthetic series of 2,048 points, one core of an Apple M5: 838 µs natively and 1,636 µs in WASM per 64-step forecast, against 21,531 µs and 39,998 µs for our TinyCast port; its file is 148,512 bytes, TinyCast's 595,840.", "s6-cast2-pico starts unticked: tick it to compare it with TinyCast and the seasonal naive on the same series." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "On GIFT-Eval it scored nGMASE 0.8847 and nWQL 0.6190, where its gate G1 asked for 0.838 and 0.587 (v1: 0.9471 and 0.6646; our TinyCast port without flip: 0.7918 and 0.5622); G2 holds, its MASE ratio to the seasonal naive is under 1.00 at each term (short 0.833, medium 0.928, long 0.988)", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-cast2-pico-s6-cast-v2-a-forecaster-with-a-periodicity-front-end-pre-registered-gate-failed" }, "models": [ "s6-cast2-pico" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-cast2-pico" }, { "config": { "model": "s6-secrets-verify-pico", "title": "s6-secrets-verify-pico", "tagline": "Failed its pre-registered gate G1 by 0.027, published as a result: a 250 KB ternary verifier that judges whether a value found by a secret-scanning rule is a real credential. Line F1 0.8561 on 221 real CredData test repositories, where the gate asked for 0.7332 + 0.15 = 0.8832; the six other gates held.", "kind": "secrets-verify", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-secrets-verify-pico", "notes": [ "The gate (docs/s6-secrets2-prereg.md §8): G1 asked for a line F1 at least 0.15 above the best rule scanner on the CredData test, on every repository and without the largest one. On every repository it reached 0.8561 against Betterleaks 0.7332 + 0.15 = 0.8832: it failed G1. Without the largest repository it holds (0.8114 against 0.6867), and the six other gates held.", "What it does well: above every rule scanner tested on CredData (Betterleaks 0.7332, Gitleaks 0.7037), calibrated on real candidates (ECE 0.0073), and an accepted candidate in 0.35% of 8,825 clean real files. Its weakest category is API keys (line F1 0.541). Since version 1.9.0, secretgate offers it as an experimental option, off by default, where it can only add redactions.", "It judges the candidates of secretgate's rule engine (frozen as sg-cand-1). This demo does not run those rules: it proposes with one simple rule (a value of at least 8 characters assigned to a name with a credential word) so that the model's judgement can be seen in a browser.", "The presets were written for this demo and every value in them is fake. Everything runs in your browser: nothing you paste is sent anywhere. Do not paste real credentials into any web page." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It missed G1 on its every-repository reading: line F1 0.8561 against 0.7332 + 0.15 for the best rule scanner (Betterleaks); the six other gates held", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-secrets-verify-pico-s6-secrets-v2-a-verifier-of-rule-candidates-pre-registered-gate-failed" }, "models": [ "s6-secrets-verify-pico" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-secrets-verify-pico" }, { "config": { "model": "s6-route-nano", "title": "s6-route-nano", "tagline": "Failed its pre-registered gate, published as a result: a 2.4 MB ternary tool router that tied BM25, the score it reads in its input (0.8395 against 0.8500 on 2,000 slates of tools never seen in training), at 38.9 ms per request in WASM against a bar of 2 ms. This Space runs the file alone; the pair that passed is the Space s6-route.", "kind": "router-v2", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-route-nano", "notes": [ "The gate (docs/s6-nano2-prereg.md §8 and amendment 1) asked it to beat BM25 on T-tools, 2,000 slates of tools that no training item contains: 0.8395 against 0.8500 (difference −0.0105, 95% interval [−0.0265, +0.0055]), so it failed G1. It also failed G4: 38.9 ms per request in WASM with the tool list cached, where the bar is 2 ms. G2 and G3 held (ECE 0.0244).", "It is above potion-base-2M with cosine + 0.03 (0.6975) and above both trained baselines of its size (0.6735 and 0.4500), but outside its training families it is below potion-cos on every other source.", "With every tool renamed tool_ it is at chance (0.1050), below the name-prior (0.1180), where BM25 keeps 0.8445 on the same slates.", "The demo shows this file alone, with no ask decision: the file stores its own ask threshold (τ_ask 0.72938, fixed on validation), which the browser engine does not expose. The Space also carries the embedding file of s6-route, because the engine's router needs both files; the pair, this file unchanged fused with potion-base-2M packed to 4 bits, passed its own gate on fresh slates." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It missed G1 against BM25, the lexical score it reads in its input: 0.8395 against 0.8500 on 2,000 slates of tools never seen in training (above potion-base-2M + 0.03 and both size-matched baselines), and G4: 38.9 ms per request in WASM against 2 ms", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-route-nano-s6-nano-v2-a-tool-router-pre-registered-gate-failed" }, "models": [ "s6-route-nano", "s6-route" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-route-nano" }, { "config": { "model": "s6-route", "title": "s6-route", "tagline": "A 3.7 MB tool router in two files: the s6-route-nano file fused with potion-base-2M packed to 4 bits. It passed its pre-registered gate on fresh slates, with a system defined after v2's test was read: 0.8868 on requests about tools never seen in training, against 0.8169 for BM25 and 0.9048 for all-MiniLM-L6-v2 (91.6 MB).", "kind": "router", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-route", "notes": [ "Disclosure: v3 was designed after v2's test was read, from an exploratory reading of v2's test dumps. Its parameters come from validation. Its gate is read on requests no system had scored; on T2-tools these requests are about the same held-out tools as v2's test. The latency bar is 30 times v2's and was set after v2 was measured. The system needs distinctive tool names.", "Every gate held: on 945 fresh requests, 0.8868 against 0.8169 for BM25 (+0.0698 [0.0455, 0.0952]) and 0.9048 for MiniLM (the bar was MiniLM − 0.03); ECE 0.0162; 30.9 ms per request in WASM with the tool list cached (bar 60 ms), 3,692,224 bytes for the two files.", "The advantage over BM25 leans on the three largest clusters of the test: without them +0.0202 [−0.0062, +0.0481], and a bootstrap over clusters gives +0.0698 [−0.0079, +0.1404]; both intervals hold zero.", "With every tool renamed tool_ it falls to 0.2497, where BM25 keeps 0.8138. MiniLM is more accurate on every test file, and outside the tool lists of its training families the router stays near its embedding part (below it on MetaTool, above it on CLINC150 and MASSIVE)." ], "game": null, "gate": "passed", "gate_kind": null, "gate_note": "Passed on fresh slates; the system was defined after v2's test was read, from an exploratory reading of v2's test dumps, with its parameters from validation. 0.8868 on 945 fresh requests about tools never seen in training, against 0.8169 for BM25 (+0.0698 [0.0455, 0.0952]) and 0.9048 for all-MiniLM-L6-v2 (91.6 MB); 30.9 ms per request in WASM (bar 60 ms, set after v2 was measured at 38.9 ms). These requests are about v2's held-out test tools. Without the three largest clusters the advantage over BM25 is +0.0202 [-0.0062, +0.0481], and with every tool renamed it falls to 0.2497: it needs distinctive tool names", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-route-s6-nano-v3-the-v2-router-fused-with-a-static-embedding-pre-registered-gate-passed" }, "models": [ "s6-route" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-route" }, { "config": { "model": "s6-route7-nano", "title": "s6-route7-nano", "tagline": "Failed its pre-registered gate, published as a result: the 3.7 MB tool router with its model part retrained on data rebuilt against the cues that give the answer without the request (s6-route v7, the last version of the line). On 2,000 fresh requests about MCP tools of a new family it reads 0.5610, where BM25 reads 0.5675; its trained part is at chance with the request hidden, the pair is not. This Space runs the pair.", "kind": "router-v7", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-route7-nano", "notes": [ "The gate (docs/s6-route7-prereg.md, with v6's §8): G1 asked the pair to beat BM25 on T5-tools, 2,000 fresh requests about the tools of 262 MCP servers (Toucan) that no training slate holds: 0.5610 against 0.5675, −0.0065 [−0.0235, +0.0120], so G1 fails. G2 fails on its part (c): with the names masked it answers none on 0.3345 of the requests, where none is right on 0.20. G3 fails: ECE 0.1845 against 0.05. G4 holds (13.9 ms per request in WASM, 3,692,288 bytes) and so does G5 (0.8815 on T2-tools against 0.8668).", "G6, the request hidden (replaced by ?): the trained model part picks the right tool on 0.1406 of the tool-right requests, at chance (1/7 = 0.1429), so G6 (a) holds; the pair picks it on 0.1969, above the bar of 0.1929 (server upper bound 0.2350 against 0.2229), so G6 (b) fails. The static embedding part alone, which no version of the line trained, already reads 0.1931 there.", "What it shows: the pair rarely follows a rewritten description (0.0494), and masking the names costs it 0.0170, where it costs the published s6-route 0.2720 on the same requests. What it does not: it falls from 0.8802 on validation to 0.5610 on the new tools, and why is not measured. On these requests all-MiniLM-L6-v2 (91.6 MB) reads 0.6630.", "The route line is closed with this result, and the router to use is s6-route. Disclosure: v7 was designed after v4's, v5's and v6's data tables were read, its training slates without a tool to anchor them are drawn around the tool nearest to the request, and the teacher rewrote the test tools' descriptions for the probe copies, never seeing a test request. The presets are those of the Router page, written for it." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It failed G1, G2, G3 and G6: on 2,000 fresh requests about the tools of MCP servers, a family that no training slate holds, the pair reads 0.5610 against BM25 0.5675 (−0.0065 [−0.0235, +0.0120], a lower bound above 0 asked); under masked names it answers none on 0.3345 of the slates (0.10 to 0.30 asked); its ECE is 0.1845 (at most 0.05 asked); with the request hidden its model part is at chance (0.1406) and the pair is not (0.1969, at most 0.1929 asked). G4 (13.9 ms per request in WASM, 3,692,288 bytes) and G5 (0.8815 on T2-tools) hold. The route line is closed and the published router stays s6-route", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-route7-nano-s6-route-v7-the-router-retrained-without-its-shortcut-pre-registered-gate-failed" }, "models": [ "s6-route7-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-route7-nano" }, { "config": { "model": "s6-sleep3-nano", "title": "s6-sleep3-nano", "tagline": "Failed its pre-registered gate, published as a result: a 752 KB ternary decision model with a fast-weight memory. The memory is used (0.4892 on the recall test against 0.2440 without one) and fades with the delay; sleep passes do not help (+0.0030), and it plays two of four games far worse than the same training without a memory. This Space plays recall episodes with it, with its memory zeroed, and with its control.", "kind": "memory", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s6-sleep3-nano", "notes": [ "The gate (docs/s6-sleep-prereg.md §7 and §8, docs/s6-sleep2-prereg.md §7, docs/s6-sleep3-prereg.md): H0, the memory is used, holds: recall 0.4892 at N = 1 against 0.2440 for the control without memory, +0.2452 (95% interval +0.2370 to +0.2532; at least 0.10 asked). H1, sleep helps, is not shown: N = 4 minus N = 1 = +0.0030 (interval −0.0010 to +0.0068). H2, the games do not lose, is not shown: Flappy 15.87 against 71.05, Dodge 5.89 against 7.94. H3 holds (typed_decisions_bench 0.4897 against 0.4808). The exported file fails the s6-nano gate on Flappy (2.10 against s3-pico's 200.00) and Dodge (6.90 against 9.47).", "The memory fades with the delay: at N = 1, 0.6147 for questions 1 to 4 frames after the show, 0.5590 at 5 to 8, 0.4530 at 9 to 16 and 0.3302 at 17 to 32, where the control is at about 0.24 everywhere. At the pico size the pilot of the same recipe had reached 1.0000 at every delay on development episodes.", "The episodes are the evaluation's generator and scoring (sky_games recall, in the RecallProbe class of sky-web), played by the exported files in the browser engine. The test read 1,000 other episodes with the training code; the 200 episodes here are validation-split seeds, so their shares differ from the test's by sampling noise.", "\"Memory zeroed\" resets the memory before every frame (the engine's own switch); the control is a second file, the same training with no memory. A research result: do not use this model as a part." ], "game": null, "gate": "failed", "gate_kind": null, "gate_note": "It failed H1 and H2 and the s6-nano gate: four sleep passes add +0.0030 on the recall test (95% interval −0.0010 to +0.0068, above 0 asked), and it plays two games far worse than the control without memory (flappy 15.87 against 71.05, dodge 5.89 against 7.94); H0 holds: with its memory it is right on 0.4892 of the recall asks against 0.2440 for the control (+0.2452)", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#s6-sleep3-nano-s6-sleep-v3-a-fast-weight-memory-pre-registered-gate-failed" }, "models": [ "s6-sleep3-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s6-sleep3-nano" }, { "config": { "model": "s5-lite", "title": "s5-lite", "tagline": "A 17M-parameter decision model whose options cannot see each other: reordering the options or renaming their keys cannot change an answer.", "kind": "decision", "proofs": true, "model_repo": "https://huggingface.co/maxgfr/s5-lite", "notes": [ "This checkpoint is a longer run (8,000 steps) added for the demo outside the pre-registered bets; see the results page for what failed and why.", "It answers 21 of the 30 English demo cases: expect mistakes, especially on incidents and severity." ], "game": null, "gate": null, "gate_kind": null, "gate_note": null, "gate_url": null }, "models": [ "s5-lite", "s4-micro" ], "space_url": "https://huggingface.co/spaces/maxgfr/s5-lite" }, { "config": { "model": "s4-micro", "title": "s4-micro", "tagline": "An experimental 5.6M ternary decision model with free-text options, distilled from Qwen3.5-4B in one night. It failed its gate, fixed before training: 0.343 on the typed-decisions bench, where the gate asked for more than 0.55.", "kind": "decision", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s4-micro", "notes": [ "Published as a documented negative result: it scores below the majority class on the typed-decisions bench." ], "game": null, "gate": "failed", "gate_kind": "fixed-before-training", "gate_note": "It scored 0.343 on the typed-decisions bench, where its gate asked for more than 0.55, and stayed below the majority class (0.398)", "gate_url": "https://github.com/maxgfr/skycmd/blob/main/BENCH.md#skycmd-s4-s4-micro-generic-decisions-distilled-from-qwen35-experimental" }, "models": [ "s4-micro" ], "space_url": "https://huggingface.co/spaces/maxgfr/s4-micro" }, { "config": { "model": "s3-pico", "title": "s3-pico", "tagline": "A 251 KB ternary decision model that plays four games in your browser, with a looped state block and early exits.", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s3-pico", "notes": [], "game": null, "gate": null, "gate_kind": null, "gate_note": null, "gate_url": null }, "models": [ "s3-pico" ], "space_url": "https://huggingface.co/spaces/maxgfr/s3-pico" }, { "config": { "model": "s1-pico", "title": "s1-pico", "tagline": "A 545 KB ModernBERT decision model in the System One format, the same file that runs in llama.cpp and wllama.", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s1-pico", "notes": [], "game": null, "gate": null, "gate_kind": null, "gate_note": null, "gate_url": null }, "models": [ "s1-pico" ], "space_url": "https://huggingface.co/spaces/maxgfr/s1-pico" }, { "config": { "model": "s2-nano", "title": "s2-nano", "tagline": "A 702 KB ternary decision model with a cached question stream and early exits.", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s2-nano", "notes": [], "game": null, "gate": null, "gate_kind": null, "gate_note": null, "gate_url": null }, "models": [ "s2-nano" ], "space_url": "https://huggingface.co/spaces/maxgfr/s2-nano" }, { "config": { "model": "s2-nano-doom", "title": "s2-nano-doom", "tagline": "A 702 KB ternary model that plays a Doom-like shooter from 14 numbers (Freedoom art).", "kind": "game", "proofs": false, "model_repo": "https://huggingface.co/maxgfr/s2-nano-doom", "notes": [], "game": "doom", "gate": null, "gate_kind": null, "gate_note": null, "gate_url": null }, "models": [ "s2-nano-doom" ], "space_url": "https://huggingface.co/spaces/maxgfr/s2-nano-doom" } ] }