onlyaady commited on
Commit
66ba833
·
verified ·
1 Parent(s): fc43746

Update Readme.md

Browse files
Files changed (1) hide show
  1. README.md +205 -46
README.md CHANGED
@@ -2,63 +2,222 @@
2
  library_name: transformers
3
  base_model:
4
  - Qwen/Qwen3.5-4B
5
- - everettjf/ezjev-4b-s2
6
- tags: [decision-model, read-twice, custom-code, peft]
 
 
 
 
 
 
 
 
 
 
 
7
  license: apache-2.0
8
  ---
9
-
10
  # BaseDecision-Dyno
11
 
12
- This v2 candidate continues the real-data read-twice `lr1e6` checkpoint at step 11207.
13
- Read twice is applied inside `decide()` by default. Single read requires explicit
14
- `mode="flash"`. The limit is 16,384 rendered tokens per question, including repetition.
15
- There is no truncation and no automatic flash fallback. Text only.
16
-
17
- The source adapter scored **54.80 on the full public Decision Index 0.3**,
18
- at temperature 1.0, versus 54.30 for the preceding development baseline. This is
19
- a public development score, not a private/full Decision Index score or JevBench placement.
20
- `scores.json` records that adapter evaluation. Standalone export is checked separately
21
- on sampled prompts; the complete public evaluation has not been rerun on this package.
22
- Latency fields in that development score file are amortized batched throughput,
23
- not single-request or HTTP latency, and must not be used as serving-speed claims.
24
-
25
- **This checkpoint has not been calibrated after continuation.** Both mode temperatures
26
- are 1.0. Previous release temperatures are not reused. Flash and score-type quality
27
- have not received a full evaluation for this candidate. No latency target is claimed.
28
-
29
- All backbone and adapter parameters are included. PEFT LoRA modules are retained
30
- inside the standalone checkpoint to preserve the evaluated base-plus-adapter tensors;
31
- no second model download or external adapter stacking is required. Backbone BF16;
32
- LoRA parameters and the small decision head FP32. Full attention is noncausal;
33
- the linear-attention recurrence remains causal. There is no text generation.
34
-
35
- Use the existing validated environment: Python 3.12, PyTorch 2.14.0+cu130,
36
- Transformers 5.17.0, PEFT 0.20.0, safetensors and accelerate. Review the included
37
- custom code before setting `trust_remote_code=True`.
 
 
38
 
39
  ```python
40
  import torch
41
  from transformers import AutoModel
 
 
42
  torch.backends.cuda.enable_cudnn_sdp(False)
43
  model = AutoModel.from_pretrained(
44
- "onlyaady/BaseDecision-Dyno", trust_remote_code=True,
45
- dtype=torch.bfloat16, device_map={"": "cuda:0"},
 
 
46
  ).eval()
47
- result = model.decide({"state": "The invoice has a duplicated charge.",
48
- "questions": {"team": {"type": "choice", "criteria": {
49
- "billing": "Invoices and refunds", "technical": "Integration problems"}}}})
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
  assert result["read_twice"] is True
51
- print(result["answers"]["team"])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
  ```
53
 
54
- Native answers expose `choice` and `probabilities`. The TypeSafe HTTP server maps
55
- choice probabilities directly, noul to the probability under `true`, and score to
56
- an ordered-level distribution with its expected level. See `SERVING.md`.
57
- Questions run independently, one forward pass per question; state prefill is not shared.
58
-
59
- This model builds on Qwen and Everett's ezjev-s2 work. Upstream model, code and
60
- training-data terms continue to apply; this update grants no additional rights.
61
- Training used permitted real training splits plus replay with overlap filtering;
62
- historical replay is retained from prior runs. The runtime contains no benchmark lookup,
63
- external teacher, task-name routing or generated reasoning. Dataset provenance and
64
- decontamination do not establish absence of every possible upstream exposure.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  library_name: transformers
3
  base_model:
4
  - Qwen/Qwen3.5-4B
5
+ base_model_relation: finetune
6
+ language:
7
+ - en
8
+ tags:
9
+ - decision-model
10
+ - system-one
11
+ - decision-index
12
+ - text-classification
13
+ - multiple-choice
14
+ - calibration
15
+ - read-twice
16
+ - custom-code
17
+ - lora-merged
18
  license: apache-2.0
19
  ---
 
20
  # BaseDecision-Dyno
21
 
22
+ **A 4B-class decision model with calibrated option probabilities.**
23
+
24
+ Give BaseDecision-Dyno a document, conversation, policy, or structured record—and questions with caller-defined options. It returns the selected option and a probability for every option. It scores decisions directly, without generating explanations or decoding an answer sequence.
25
+
26
+ The calibrated BF16 checkpoint scores **53.83 on the full public Decision Index 0.3 suite**, using the official evaluation kit.
27
+
28
+ ## At a glance
29
+
30
+ | Property | Release |
31
+ |---|---|
32
+ | Model family | Qwen3.5-4B |
33
+ | Modality | Text only |
34
+ | Weights | Standalone merged checkpoint; no adapter stacking required |
35
+ | Runtime | Custom Hugging Face Transformers `AutoModel` |
36
+ | Default inference | One scoring pass per question |
37
+ | Optional inference | Explicit `mode="flash"`, single read |
38
+ | Output | Option key and normalized option probabilities |
39
+ | Choice options | 2–255 through the evaluated Decision Index adapter |
40
+ | Context | 16,384 **rendered tokens per question**, including prompt overhead |
41
+ | Precision | BF16 backbone; FP32 decision readout and softmax |
42
+ | Attention | Bidirectional full-attention layers; native recurrence in linear-attention layers |
43
+ | Generation / decoding cache | No autoregressive generation; decoding KV cache disabled |
44
+
45
+ ## Quickstart
46
+
47
+ The release includes custom model code. Review it before enabling `trust_remote_code=True`. Use `AutoModel` and the structured `decide()` interface to preserve the trained prompt, attention, option mapping, and calibration.
48
+
49
+ The validated environment used **Python 3.12.15, PyTorch 2.14.0+cu130, and Transformers 5.17.0**. Package loading and sampled protocol checks also passed on an NVIDIA A100-SXM4-80GB. The examples below require a compatible NVIDIA CUDA GPU.
50
 
51
  ```python
52
  import torch
53
  from transformers import AutoModel
54
+ MODEL_PATH = "onlyaady/BaseDecision-Dyno"
55
+ # Match the backend setting used by the evaluated engine.
56
  torch.backends.cuda.enable_cudnn_sdp(False)
57
  model = AutoModel.from_pretrained(
58
+ MODEL_PATH,
59
+ trust_remote_code=True,
60
+ dtype=torch.bfloat16,
61
+ device_map={"": "cuda:0"},
62
  ).eval()
63
+ request = {
64
+ "state": "The customer's invoice contains the same charge twice.",
65
+ "questions": {
66
+ "team": {
67
+ "type": "choice",
68
+ "instructions": "Which team should handle this request?",
69
+ "criteria": {
70
+ "billing": "Charges, invoices, and refunds",
71
+ "technical": "Technical integration problems",
72
+ "sales": "Questions about buying the product",
73
+ },
74
+ },
75
+ "duplicate": {
76
+ "type": "noul",
77
+ "instructions": "The customer was charged twice for the same item.",
78
+ },
79
+ },
80
+ }
81
+ # Read twice is the default. Calibration is applied internally.
82
+ result = model.decide(request)
83
  assert result["read_twice"] is True
84
+ print(result["answers"]["team"]["choice"])
85
+ print(result["answers"]["team"]["probabilities"])
86
+ print(result["answers"]["duplicate"]["probabilities"]["true"])
87
+ # Explicit single-read mode, with its own calibration applied internally.
88
+ flash_result = model.decide(request, mode="flash")
89
+ assert flash_result["read_twice"] is False
90
+ ```
91
+
92
+ For a single question, `{"state": ..., "question": ...}` is also accepted; its answer is returned under `answers["q"]`. Multiple questions are processed independently, each in its own sequence and forward pass. They do not share a single prefill pass.
93
+
94
+ The response includes `answers`, `mode`, `read_twice`, `sequence_tokens`, `calibrated`, and `release_id`. For both decision types, the native API returns `choice` and `probabilities`. A `noul` answer uses the probability keys `"true"` and `"false"`; the Decision Index adapter converts this distribution into the benchmark’s scalar `noul` response.
95
+
96
+ ### Persistent inference
97
+
98
+ Download the model and its included JSONL runner:
99
+
100
+ ```bash
101
+ hf download onlyaady/BaseDecision-Dyno --local-dir BaseDecision-Dyno
102
+ cd BaseDecision-Dyno
103
+ # Default: read twice, with calibration applied internally.
104
+ python inference.py --input requests.jsonl
105
+ # Explicit single-read mode, with its own calibration.
106
+ python inference.py --mode flash --input requests.jsonl
107
  ```
108
 
109
+ Each input line is one structured JSON request; each output line is one JSON response. Create `requests.jsonl` with your requests before running these commands.
110
+
111
+ The runner loads the model once and keeps it resident across successive requests. Omit `--input` to read requests from stdin until EOF. Use the validated CUDA environment. No compilation, quantization, or cross-request batching is enabled by this runner.
112
+
113
+ ## Read twice by default
114
+
115
+ Read twice repeats the state and question/options content within one rendered sequence, followed by **one scoring pass per question**. It is not two independent predictions, voting, or generated reasoning.
116
+
117
+ - `model.decide(request)`, `model(request)`, and `model.prepare_request(request)` default to read twice.
118
+ - `mode="flash"` explicitly selects a single read.
119
+ - The 16,384-token limit applies **after rendering and repetition**. Default mode therefore accepts less original content than flash mode.
120
+ - Oversized requests raise a length error. Inputs are not silently truncated, and default mode never falls back automatically to flash.
121
+ - Keep state and questions structured; do not manually repeat the input before calling `decide()`.
122
+
123
+ The default is implemented in the supplied model code rather than left to the caller's prompt. Someone who modifies the code or directly uses lower-level weights can change that behavior.
124
+
125
+ ## Decision Index results
126
+
127
+ **Completed self-run public evaluation with the official kit, edition 0.3.** The exact calibrated model and strict default protocol above were used. These are public results, not a maintainer-run private or Full leaderboard score.
128
+
129
+ | Metric | Result |
130
+ |---|---:|
131
+ | Public Decision Index | **53.83** |
132
+ | Raw index | 65.19 |
133
+ | Breadth skill score | 52.16 |
134
+ | Requests processed before official exclusions | 140,620 |
135
+ | Requests after official exclusions | 140,178 |
136
+ | Successful scored requests | 140,016 |
137
+ | Unsupported scored requests | 162 |
138
+ | Errors | 0 |
139
+
140
+ Index values are **scores, not accuracy percentages**. The public composite includes 37 panel benchmarks. The official report also contains additional diagnostic benchmark entries.
141
+
142
+ | Area | Skill score (0–100) | Raw score (0–100) |
143
+ |---|---:|---:|
144
+ | Knowledge & Reasoning | 35.96 | 50.26 |
145
+ | Language Understanding | 62.37 | 73.23 |
146
+ | Retrieval & Classification | 59.15 | 70.28 |
147
+ | Tools & Automation | 71.66 | 74.19 |
148
+ | Arts & Human Taste | 34.70 | 56.36 |
149
+
150
+ Raw scores combine benchmark-specific metrics; they should not all be read as plain accuracy. See the accompanying [`scores.json`](scores.json) for individual benchmarks, coverage, and scoring details.
151
+
152
+ Unsupported requests were retained in the official scoring path: POP909-CL (158), ToolRet (2), API-Bank (1), and HLE (1). The summary reports alone do not establish the individual refusal reason for every row.
153
+
154
+ ## Calibration
155
+
156
+ Mode-specific scalar temperatures are applied **once to raw logits**, followed by a softmax over the current question's options:
157
+
158
+ ```text
159
+ probabilities = softmax(raw_logits / temperature)
160
+ ```
161
+
162
+ | Mode | Temperature |
163
+ |---|---:|
164
+ | Default / read twice | 2.0561549881751153 |
165
+ | Flash / single read | 1.9163401529278457 |
166
+
167
+ `decide()` already applies the appropriate temperature. Do not apply another temperature to its returned probabilities. These values replace the earlier 1.26 setting.
168
+
169
+ Temperatures were fitted on 10,000 decisions and audited on 10,000 separate decisions: 8,000 held out by group and 2,000 from families absent from temperature fitting. Those families were not necessarily absent from backbone training.
170
+
171
+ | Mode | Audit metric ↓ | Earlier T = 1.26 | Calibrated |
172
+ |---|---|---:|---:|
173
+ | Default | Negative log-likelihood | 0.897542 | **0.817370** |
174
+ | Default | Brier score, summed over options | 0.390352 | **0.365954** |
175
+ | Default | 15-bin hard-label ECE | 0.107526 | **0.028381** |
176
+ | Flash | Negative log-likelihood | 0.876103 | **0.823770** |
177
+ | Flash | Brier score, summed over options | 0.385906 | **0.368202** |
178
+ | Flash | 15-bin hard-label ECE | 0.091027 | **0.030229** |
179
+
180
+ Accuracy on the 9,266 hard-labeled audit decisions was unchanged by calibration: 70.26% in default mode and 69.78% in flash mode. Positive scalar temperature scaling preserves the selected option. These audit accuracies are separate from the Decision Index score.
181
+
182
+ Calibration improved on this audit distribution; it is not guaranteed on every domain. Training, quantization, or a material backend change requires renewed evaluation and may require recalibration.
183
+
184
+
185
+ ## Performance and deployment
186
+
187
+ The completed public run measured the following request latencies on shared **NVIDIA RTX PRO 6000 Blackwell** resources:
188
+
189
+ | Default read-twice latency | Time |
190
+ |---|---:|
191
+ | Median | 53.3 ms |
192
+
193
+ These are CUDA-synchronized individual-request timings from the public evaluation. Shared physical GPU contention and the benchmark request mix affect the measurements. They are not the maintainers' dedicated-hardware latency result, and they exclude model loading and external network/queueing overhead.
194
+
195
+ On an earlier short-input sample (60 distinct requests per mode, each measured three times; up to 512 original single-read rendered tokens and at most 32 options), the uncompiled runtime measured:
196
+
197
+ | Mode | Median | p95 |
198
+ |---|---:|---:|
199
+ | Default / read twice | 29.58 ms | 63.43 ms |
200
+ | Flash / single read | 26.15 ms | 36.32 ms |
201
+
202
+ The short-input results do not establish a universal 150 ms default or 100 ms flash bound. Long contexts and many-question requests can take substantially longer.
203
+
204
+ Keep the model loaded across requests. The supplied runner does this without changing the scoring path. Compilation and quantized inference are not qualified for this release. A 16GB laptop deployment remains a target; this BF16 GPU release does not establish laptop memory fit or latency.
205
+
206
+ ## Limitations
207
+
208
+ - Text-only release; image, audio, and video inference are not supported.
209
+ - The evaluated adapter supports `choice` and `noul`; a dedicated ordinal `score` response interface is not qualified by this release.
210
+ - Default repetition consumes context and compute. The context limit is per rendered question, not 16k original tokens plus a second copy.
211
+ - Option probabilities are normalized over the supplied options. They do not prove that the option set contains a correct answer.
212
+ - English-focused evaluation does not establish comparable multilingual performance.
213
+ - Factual, numerical, ambiguous, and out-of-domain decisions can be wrong, including when confidence is high.
214
+ - Adversarial robustness and resistance to instructions planted in the state have not been established.
215
+ - Owners can modify the supplied runtime; read-twice enforcement is an API behavior, not a cryptographic restriction on weights.
216
+
217
+ ## Licensing and acknowledgements
218
+
219
+ The **BaseDecision-Dyno software repository is licensed under Apache-2.0**.
220
+
221
+ This release builds on [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B); Qwen3.5-4B's base license does not by itself settle the licensing of this derived release or all training sources. Preserve applicable upstream notices and dataset terms.
222
+
223
+ Special Thanks to everettjf/ezjev-4b-s2 & Maintainer of tasksource.