| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| base_model: distilbert/distilbert-base-uncased |
| datasets: |
| - DavidHatley/system-one-mini-data |
| tags: |
| - multi-task-classification |
| - synthetic-data |
| - research |
| - custom-code |
| --- |
| |
| # System One Mini |
|
|
| System One Mini is an independent research prototype for five fixed, typed decisions over synthetic controlled software-diagnosis summaries. It is a 69.3M-parameter DistilBERT encoder with five depth-2 classification heads. |
|
|
| **This is not a reproduction of Jev or RLCD.** It is not a chat model, does not generate text, and cannot answer arbitrary questions. |
|
|
| ## Intended Use |
|
|
| Use this model to study fixed-schema multi-task classification on complete summaries generated under the documented controlled-intervention policy. It supports only: |
|
|
| | Decision | Options | |
| |---|---| |
| | `task_complete` | `false`, `true` | |
| | `should_retry` | `false`, `true` | |
| | `failure_source` | `code`, `environment`, `test`, `dependency`, `unknown` | |
| | `next_action` | `inspect_code`, `rerun_tests`, `inspect_environment`, `revert_change`, `escalate` | |
| | `evidence_sufficient` | `false`, `true` | |
|
|
| Do not use it for arbitrary logs, general coding assistance, natural production incidents, causal claims, safety-critical decisions, or autonomous agent control. |
|
|
| ## Loading |
|
|
| This repository contains reviewed custom Transformers code. Pin a repository revision in production because `trust_remote_code=True` executes that code. |
|
|
| ```python |
| from transformers import AutoModel, AutoTokenizer |
| |
| repo = "DavidHatley/system-one-mini" |
| tokenizer = AutoTokenizer.from_pretrained(repo) |
| model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval() |
| |
| state = "Latest verification: checks failed. Acceptance: not confirmed. ..." |
| tokens = tokenizer(state, truncation=True, max_length=model.config.max_length, return_tensors="pt") |
| outputs = model(**tokens) |
| print([logits.softmax(-1) for logits in outputs.logits]) |
| ``` |
|
|
| For named, temperature-scaled outputs, download `inference.py` from this repository and use its `load_model` and `predict` functions. |
|
|
| ## Architecture And Training |
|
|
| - Base: `distilbert/distilbert-base-uncased` at revision `12040accade4e8a0f71eabdb258fecc2e7e948be` |
| - Shared encoder followed by five independent `Linear -> GELU -> Dropout -> Linear` heads |
| - Maximum input length: 256 tokens |
| - Full fine-tuning for three epochs in BF16 on one NVIDIA RTX 3060 |
| - 20,000 deterministic synthetic training states; seed 1729 |
| - No reinforcement learning, external teacher model, paid API, or real user traces |
| - Checkpoint selected by validation mean NLL |
|
|
| ## Evaluation |
|
|
| The post-selection final set uses one deterministic synthetic renderer written and frozen after candidate selection. It was evaluated once. This tests wording transfer inside the same synthetic policy, not real-world diagnosis. |
|
|
| | Final variant | Accuracy | NLL | Brier | ECE | Mean confidence | |
| |---|---:|---:|---:|---:|---:| |
| | Raw | 86.25% | 0.8479 | 0.2630 | 0.1273 | 98.89% | |
| | Temperature scaled | 86.25% | 0.7133 | 0.3940 | 0.2673 | 63.19% | |
|
|
| Raw per-head results on the final renderer: |
|
|
| | Head | Accuracy | NLL | ECE | Mean confidence | |
| |---|---:|---:|---:|---:| |
| | `task_complete` | 100.00% | 0.0004 | 0.0004 | 99.96% | |
| | `should_retry` | 82.65% | 1.1987 | 0.1677 | 99.24% | |
| | `failure_source` | 92.95% | 0.2982 | 0.0451 | 97.46% | |
| | `next_action` | 62.80% | 2.4050 | 0.3641 | 99.17% | |
| | `evidence_sufficient` | 92.85% | 0.3373 | 0.0593 | 98.63% | |
|
|
| The earlier renderer-generalization split was used to compare model candidates and is therefore a development result, not an unbiased final test: |
|
|
| | Development variant | Accuracy | NLL | Brier | ECE | Mean confidence | |
| |---|---:|---:|---:|---:|---:| |
| | Raw | 59.27% | 2.4363 | 0.7666 | 0.3740 | 96.58% | |
| | Temperature scaled | 59.27% | 0.7875 | 0.4331 | 0.1476 | 62.39% | |
|
|
| ## Calibration |
|
|
| Per-head scalar temperatures were fitted only on the 2,000-example calibration split. On the final renderer, scaling improved NLL from 0.8479 to 0.7133, but worsened ECE from 0.1273 to 0.2673. Maximum softmax probability must not be interpreted as proven certainty. |
|
|
| ## Limitations |
|
|
| - Training and evaluation are entirely synthetic. |
| - The model learns a fixed schema and exact option sets, not arbitrary typed questions. |
| - Results are from one training seed and one post-selection final renderer. |
| - Performance is sensitive to wording and representation format. |
| - Raw predictions are strongly overconfident under distribution shift. |
| - Temperature scaling did not consistently improve every calibration metric across renderers. |
| - Inputs longer than 256 tokens are truncated. |
|
|
| ## Provenance |
|
|
| The model is derived from Apache-2.0-licensed DistilBERT. All added training examples are deterministic project-authored synthetic data. Checkpoint, dataset, calibration, and evaluation hashes are included under `provenance/`. |
|
|
