| # deg contract v0.1 |
|
|
| Status: contract and schema compiler implemented (`src/deg/schema.py`, `response.py`, |
| `request.py`, `parsers.py`); model, trainer and in-process runtime implemented |
| (`model.py`, `train.py`, `infer.py`, `evaluate.py`, design in §12); HTTP server and Python |
| SDK implemented (`api.py`, `server.py`, `client.py`, §9). |
|
|
| ## 1. Model |
|
|
| - Base model: **Qwen3.5-0.8B** (default; LFM2.5-1.2B-Instruct is the alternative |
| backbone, §12.8), with three kinds of heads on one trunk: |
| - decision heads: `choice`, `score`, `noul` (as in jev v1); |
| - a span pointer (start/end over context tokens, plus a null outcome); |
| - the LM head, used only for `x-generate` fields. |
| - Model id: `deg-qwen3.5-0.8b-0.1` (alias `deg-latest`). |
|
|
| ## 2. The one flag (and two annotations) |
|
|
| `x-generate: true` is the only extension that changes a field's mode. |
|
|
| - It is allowed only on `type: string` or on `type: array` whose `items` is a plain |
| `type: string` (no `enum`, no `format`). On an array, it applies to every item. |
| - `x-generate: false` means the same as leaving it out. |
|
|
| Two annotations refine a field without changing its mode: |
|
|
| - `x-max-tokens: N` (integer >= 1, only where `x-generate` applies; on an array it applies |
| to every item): that field's own decode budget, overriding `options.max_new_tokens`, so |
| one request can mix budgets. |
| - `x-levels: {"<level>": "label", ...}` (only on an integer score field, keys among its |
| admissible levels, labels non-empty strings): each labeled level's option line becomes |
| `<level>: <label>` (the layout the score head was trained on from jev level criteria); |
| unlabeled levels keep the bare number. Use it instead of listing level meanings in the |
| description. |
|
|
| Any other `x-*` keyword is an error, so a typo like `x-genrate` fails loudly. |
|
|
| There is **no router head**. A field's mode follows only from its JSON type and keywords |
| (§3). The model never chooses between deciding, extracting and generating. |
|
|
| ## 3. Type → mechanism |
|
|
| | Schema (no `x-generate`) | Mode | Output | |
| |---|---|---| |
| | `enum` (any scalar types, 2..255 values; `null` allowed as a value) | `decide.choice` | the value, its probability, the full distribution, confidence | |
| | `oneOf`/`anyOf` whose branches are all `const`/`enum` | `decide.choice` | the same; branch `description`s become option descriptions | |
| | `boolean` | `decide.noul` | `p(true)`; the value is `p >= 0.5` | |
| | `["boolean","null"]` | `decide.choice` over `[true, false, null]` | as choice | |
| | `integer` whose bounds admit 2..11 values (after `multipleOf`) | `decide.score` | the most probable level, the distribution, the expected level | |
| | every other number: any `type: number`, or an `integer` with a missing bound or more than 11 admissible values | `extract.number` | a span plus a deterministic number parser; bounds only validate | |
| | `string` with `format: date` / `date-time` | `extract.date` | a span plus a deterministic date parser, giving an ISO 8601 / RFC 3339 value | |
| | `string` with `format: email` / `uri` / `uuid` | `extract.span` | a verbatim span that must pass the format check | |
| | `string` | `extract.span` | a verbatim span | |
| | `array` | per item | the length is a runtime count; each item uses the item schema's mode | |
| | `object` | recursive | the properties, in schema order | |
|
|
| | Schema with `x-generate: true` | Mode | |
| |---|---| |
| | `string` (optional `minLength`, `maxLength`, `pattern`) | `generate` | |
| | `array` of `string` | `generate` per item | |
|
|
| Rules of thumb that follow from the table: |
|
|
| - **Number rule.** Only a small integer scale is a judgment: an `integer` whose bounds give |
| at most 11 possible values (`1..5`, `0..10`, `0..100` step 25) is decided. Every other |
| number is a quantity copied from the context. Bounds (`minimum`, `maximum`, exclusive |
| bounds, `multipleOf`) never turn a quantity into a decision; they only validate the |
| extracted value. There is no expected-value path for numbers: deg never computes a |
| number that is not written in the context. To get a judgment on a finer or non-integer |
| scale, use an integer scale or an `enum`. |
| - Extracted values are always copied from the context. Parsers only normalize them; they |
| never invent digits or dates. |
| - **x-generate fields are the only source of new tokens.** |
|
|
| ## 4. Null, optional and constraints |
|
|
| - Nullable means `type: [X, "null"]`, `anyOf`/`oneOf: [X, {type: null}]`, or an `enum` |
| that lists `null`. Nullable objects and arrays are not supported. |
| - Decide fields are always filled, whether optional or not. Nullable score fields get an |
| extra null outcome, and nullable booleans become a 3-way choice. |
| - When an extract field finds nothing, or its span fails the parser or a constraint, the |
| plan's `on_null` decides what happens: |
| - `null`: the field is nullable, so the value is `null`; |
| - `omit`: the field is optional and not nullable, so the key is left out; |
| - `forbid`: the field is required and not nullable, so the extractor must return its |
| best valid span. If it can't, that is a runtime error (`FillError`, HTTP 500), never a |
| made-up value. This is deliberate: authors who want a fallback mark the field |
| nullable (`type: [X, "null"]`) and get `null` with a `null_reason` instead. |
| **Exception: inside an array item** (anchored or index-addressed, §12.6) a `forbid` |
| leaf with no valid value means the item does not exist: the item is dropped (not an |
| error), the array's meta says `"dropped": n`, and a `minItems` the drops break is |
| handled as in §12.6 (drop the enclosing item, else a `FillError` at the top level). |
| - `meta.null_reason` is one of `not_found`, `unparsable`, or the name of the violated |
| keyword. |
| - Constraints are enforced as follows: |
| - Spans are filtered by `minLength`, `maxLength`, `pattern` (searched, as in JSON |
| Schema) and `format`. |
| - Numbers are filtered by `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum` |
| and `multipleOf`. |
| - Generated text must satisfy `minLength`, `maxLength` and `pattern`. Output that |
| violates them is a runtime error (`FillError`), never a silent truncation or repair. |
| The generator uses **constrained decoding** (a length budget plus a regex/grammar |
| automaton for `pattern`; EOS masked until `minLength`), so a valid output exists by |
| construction and this error should not occur in practice; it is a guard, not a code |
| path. |
| - Arrays: the length is bounded by `minItems` and `maxItems`. When `maxItems` is absent, |
| the cap is `options.max_items` (default `null` = no cap: an anchored array has as many |
| items as the context has, §12.6; an index-addressed array stops at 32, the count head's |
| trained range). `uniqueItems` is checked when the response is validated. |
| - Objects: `additionalProperties` may be absent, `true` or `false`. deg never emits |
| undeclared keys. A schema-valued `additionalProperties` is an error, because the key |
| names would be unknown. |
| |
| ## 5. Validation errors (HTTP 422, every problem listed as `{loc, msg}`) |
|
|
| - `x-generate` on anything other than `string` or `array` of `string`. This includes |
| `enum`, `boolean`, `integer`, `number`, `object`, arrays of non-strings, and strings |
| that have `enum` or `format`. |
| - **Unsupported keywords:** `allOf`, `not`, `if`/`then`/`else`, `const` (a single value |
| has nothing to decide), `dependentSchemas`, `dependentRequired`, `dependencies`, |
| `patternProperties`, `propertyNames`, `unevaluatedProperties`, `minProperties`, |
| `maxProperties`, `unevaluatedItems`, `prefixItems`, `additionalItems`, tuple `items: |
| [...]`, `contains`, `minContains`, `maxContains`, `$dynamicRef`, `$dynamicAnchor`, |
| `$recursiveRef`, `$recursiveAnchor`, `contentEncoding`, `contentMediaType`, |
| `contentSchema`. |
| - **Limited keywords:** |
| - `anyOf`/`oneOf` are allowed only in the nullable form or the const/enum form. |
| - `$ref` must be local (`#` or `#/...`) and not recursive. Its siblings may only be |
| annotations, which override the target's. |
| - Union types are allowed only as `[X, "null"]`. |
| - `format` must be one of `date`, `date-time`, `email`, `uri` or `uuid`. |
| - Also errors: |
| - unknown keywords, and keywords that don't apply to the type (e.g. `maxLength` on |
| `integer`); |
| - an `enum` with a duplicate value, a value of the wrong type, or fewer than 2 or more |
| than 255 values; |
| - integer bounds that admit 0 or 1 values; |
| - number bounds that admit no value (e.g. `minimum` > `maximum`); |
| - an invalid `pattern`; |
| - a `required` name that isn't in `properties`; |
| - a root that isn't `type: object`; |
| - any document that isn't a valid JSON Schema (draft 2020-12 by default, or the draft |
| named in `$schema`). |
| - Allowed annotations: `title`, `description`, `default`, `examples`, `$comment`, |
| `$schema`, `$id`, `$anchor`, `$defs`, `definitions`, `readOnly`, `writeOnly`, |
| `deprecated`. A leaf's `title` and `description` are its instructions to the model. |
| |
| ## 6. Fill plan |
|
|
| `compile_schema(schema) -> FillPlan` is a pure function. The plan holds: |
|
|
| - `fields`: the leaves in schema order (depth-first). Each leaf has: |
| - its path: property names, with `*` meaning each item; |
| - its mode and JSON type; |
| - `options`/`option_descriptions` (choice) or `levels` (score); |
| - `null_outcome`, `format`, `constraints`; |
| - `required`, `nullable`, `on_null`, `title`, `description`. |
| - `arrays`: the array nodes with `min_items`, `max_items` and `unique_items`. |
| - `FieldPlan.fingerprint`: a sha256 of the leaf's canonical spec. It is identical wherever |
| that leaf appears, whatever its siblings or their order. |
| - `FillPlan.fingerprint`: a hash of the **set** of parallel-phase fingerprints plus the |
| **ordered list** of generate fingerprints. |
| - Reordering decide/extract properties keeps the hash. |
| - Reordering generate fields changes it. |
|
|
| ## 7. Execution order |
|
|
| 1. **Read once.** The context is rendered once and encoded once: |
| - strings are used verbatim; |
| - JSON is serialized with `", "`/`": "` separators, the given key order and |
| `ensure_ascii=false`. |
| 2. **Parallel phase: all decide.\* and extract.\* fields.** Each field's outcome is a |
| function of exactly the context encoding, the field's key path, the field's own |
| annotations, and its options/levels/constraints. It is never a function of other fields |
| or their order. |
| - So outcomes are **bitwise-reproducible** when fields are reordered, added or removed, |
| or when the schema is split into several requests or merged into a bigger one. |
| - Array items are first enumerated as item anchors (the count). Each item's leaves are |
| then conditioned on the context plus that item's anchor. |
| 3. **Generate phase.** The `generate` fields run after the parallel phase. Each one is |
| conditioned on the context and all decided/extracted values (one shared KNOWN VALUES |
| block), never on other generate fields: generate fields are independent of each other |
| and decode together. |
| - Decoding is greedy by default. |
| - `options.temperature > 0` requires `options.seed`; each field samples from its own |
| stream seeded by `(seed, pointer)`. |
| - Decoding is bounded by `maxLength`, `pattern` and the field's `x-max-tokens`, else |
| `options.max_new_tokens` (default 256 per field). |
| - Not implemented: generate-to-generate dependencies. If added, they must be declared |
| (e.g. `"x-generate": {"after": "/summary"}`) and the dependent field must continue |
| from the parent row's final state after its generated tokens (forked), never |
| re-prefill the parent's text; unannotated schemas keep the independence above. |
| |
| ## 8. Response |
|
|
| ```json |
| { |
| "model": "deg-qwen3.5-0.8b-0.1", |
| "value": {"department": "billing", "urgent": true, "reply": "..."}, |
| "meta": { |
| "/department": {"mode": "decide.choice", "probability": 0.8, "confidence": 0.7, |
| "probabilities": [0.8, 0.15, 0.05]}, |
| "/urgent": {"mode": "decide.noul", "probability": 0.91}, |
| "/anger": {"mode": "decide.score", "probability": 0.6, "confidence": 0.5, |
| "probabilities": [0, 0.1, 0.6, 0.3, 0], "expected": 2.2}, |
| "/order_id": {"mode": "extract.span", "confidence": 0.88, "span": [32, 39]}, |
| "/due": {"mode": "extract.date", "confidence": 0.4, "span": null, |
| "null_reason": "not_found"}, |
| "/line_items": {"mode": "array", "items": 2}, |
| "/reply": {"mode": "generate", "generated_tokens": 11} |
| }, |
| "usage": {"prompt_tokens": 42, "generated_tokens": 11}, |
| "plan": "<FillPlan.fingerprint>" |
| } |
| ``` |
|
|
| - `value` is always valid against the input schema. `assemble` checks this with |
| `jsonschema` and deg's own format checks, and raises `FillError` (HTTP 500) otherwise. |
| - `meta` is keyed by concrete RFC 6901 pointer, with integer array indices. An array |
| that dropped items whose required values were not in the context (§12.6) has |
| `"dropped": n` beside `"items"`. |
| - `probabilities` follow the schema's option or level order; for score, the null outcome |
| comes last. |
| - `span` gives `[start, end)` code-point offsets into the rendered context. |
| `assumed_offset: true` marks a date-time whose offset came from |
| `options.default_offset`. |
| - `confidence = (count * peak - 1) / (count - 1)` is the jev v1 formula. Ties go to the |
| first maximum. |
| - `usage.generated_tokens` is the sum of the per-field `generated_tokens`. It is 0 for a |
| schema with no `x-generate`. |
| - On a multimodal context an extract field whose pointer lands in a media item has |
| `"span": null` and says where instead: `"media": "m0"` plus `"box": [x0, y0, x1, y1]` |
| (pixels of the source image, `x1`/`y1` exclusive) or `"time": [s, e]` (seconds of the |
| source audio), `"tokens": [i, j]` (the item's soft tokens the pointer chose) and |
| `"read": "crop"` (the value was read from that region, §12.10). No schema keyword asks |
| for it, as none asks for `span`: the value and the schema are unchanged, and the |
| location is meta. A located value that cannot be read is null with |
| `null_reason: "media_unread"`. |
|
|
| ## 9. API |
|
|
| `POST /v1/fill` |
|
|
| ```json |
| {"context": "string | object | array", |
| "schema": {"type": "object", "...": "..."}, |
| "model": "deg-latest", |
| "options": {"max_new_tokens": 256, "max_items": null, "date_order": null, |
| "default_offset": "Z", "temperature": 0, "seed": null, |
| "array_decode": "count_peaks", "null_margin": null}} |
| ``` |
|
|
| - `array_decode` (`"threshold"` | `"count_peaks"` | `"hybrid"`) picks how anchored |
| arrays find their items (§12.6); also accepted in the decision APIs' `options`. |
| - `null_margin` (finite number, nats, or `null`) biases nullable/optional extract leaves |
| toward null: such a leaf is null when its best valid span's log-prob is below the null |
| log-prob + `null_margin` (§12.1). `null` = the server's `--null-margin` (default 0 = |
| plain argmax). > 0 trades made-up answers for omissions (deg-2b held-out choice: 2.6, |
| deg-0.8b: 1.4; `deg.bench.null_margin`, results/deg/null-margin-v1). It touches no |
| decide field, no `forbid` (required non-nullable) leaf and no count/anchor decision; it |
| applies to leaves inside index-addressed and anchored items the same way (a nullable |
| item leaf becomes null, an optional one is omitted). Also accepted in the decision APIs' |
| `options`. |
|
|
| - `date_order` is `"MDY"`, `"DMY"` or `null`. With `null`, a numeric date like |
| `03/05/2026` parses only if exactly one reading is a valid date; otherwise the field is |
| null or omitted. |
| - Unknown request fields and unknown options are 422 errors. |
|
|
| `GET /v1/models` lists the accepted model names; `GET /health` reports the batcher. |
|
|
| ### 9.1 Decision APIs (`src/deg/api.py`, `server.py`) |
|
|
| `python -m deg.server --ckpt DIR --port 8790` serves three routes on one model. Each |
| request becomes **one** fill: every question is a property of one schema (key = question |
| name / id), so the input is read once, decisions/extractions are independent fields, and |
| `generate` questions run last, conditioned on the other answers. |
|
|
| | Question (OpenAI / Jev) | Field schema | Mode | |
| |---|---|---| |
| | `predicate` / `noul` | `boolean`; Jev `criteria.true/false` appended to the description as `Yes means:` / `No means:` lines (the jev training format) | decide.noul | |
| | `choice` | `oneOf` of `const` + `description` (or `enum` when no option has a description); OpenAI values are typed (`"true"` != `true`) | decide.choice | |
| | `score` | `integer`, `minimum 0`, `maximum L-1`, description = the instructions; level labels (and descriptions) as `x-levels: {"i": "label[: description]"}`, so each is the option line `i: label` the score head was trained on | decide.score | |
| | `extract` (deg) | `[string\|number\|integer, null]` + `format` from `value_type`, `pattern`, `minLength`, `maxLength` | extract.* (nullable: a miss is `null`) | |
| | `generate` (deg) | `string`, `x-generate: true`, `minLength`, `maxLength`, `pattern`; `max_tokens` -> that field's `x-max-tokens` (questions in one request may use different budgets) | generate | |
|
|
| A one-option choice or one-level score (accepted by live Jev) is answered with probability 1 |
| without the model. |
|
|
| - `POST /v1/decisions` mirrors the OpenAI Decisions API (openai-python 3.26.1 |
| `types/decision.py`, `types/decision_create_params.py`; guide |
| https://developers.openai.com/api/docs/guides/decisions): request `model`, `input` |
| (string or user messages of `input_text` parts; `input_image` is refused with 400 because |
| this model reads text only), `questions`, `safety_identifier`; response `{answers, model, |
| usage}` with answers in question order: `predicate {type, name, probability}`, `choice |
| {type, name, choice, probabilities: [{value, probability}], confidence}`, `score {type, |
| name, score, probabilities: [{value, label, probability}], confidence}` (`score` = the |
| expected level index), usage `{input_tokens, input_tokens_details{cache_write_tokens, |
| cached_tokens}, output_tokens, output_tokens_details{reasoning_tokens}, total_tokens}`. |
| `name` is optional (echoed, `null` when absent) and must be unique. Errors: 400 |
| `{"error": {message, type: "invalid_request_error", param, code}}`. deg never returns a |
| `refusal` answer. |
| - `POST /v1/systemone` mirrors live TypeSafe Jev (`jev-1.13.0`, probed 2026-10-09; |
| `tests/fixtures/jev_systemone_live.json`): answers keyed by id in request order, `noul |
| {type, noul}`, `choice {type, choice, confidence, probabilities{option: p}}`, `score {type, |
| score, confidence, legend{"i": level}, probabilities{"i": p}}`, usage `{input_tokens, |
| output_tokens}`. Limits and errors as live Jev: 1..255 options (`{"detail": "Too many |
| choices. Must have at most 255 choices."}`), 1..10 levels, unknown type / unknown |
| top-level field -> 400 `{"detail": {"error_type": "api_usage_error", "message": "Invalid |
| request."}}`, unknown model -> 400 `Unknown model: X`, missing field / empty questions -> |
| 422 `{"detail": [{type, loc, msg, input}]}`, missing key -> 403. Unknown question keys are |
| ignored, as live Jev does. Unlike live Jev, probabilities are not rounded to 2 decimals. |
| - Extensions on both: `extract` -> `{type, value, span, confidence[, null_reason]}`, |
| `generate` -> `{type, text, output_tokens}`; request `options` (`temperature`, `seed`, |
| `date_order`, `default_offset`). Usage output tokens are the generated tokens (0 without |
| `generate`). |
| - Model names: deg aliases, plus `gpt-6-luna` (decisions) and `jev-latest`, |
| `jev-preview`, `clef-flash` (systemone) so existing clients work unchanged; responses |
| carry deg's model id. |
| - Batching: one GPU worker; requests arriving within the batch window (default 5 ms) are |
| taken together (default 16) and handed to `deg.infer.fill_many(model, jobs)` when the |
| runtime defines it, else run back to back. A failing job fails only its own request; |
| more than `--max-pending` queued jobs -> 429. |
|
|
| ## 10. Determinism guarantees |
|
|
| - **Plan:** the same schema always gives the same plan and fingerprints. Compilation never |
| mutates the input. |
| - **Parallel phase:** greedy and sampled requests behave identically here, since decide |
| and extract involve no sampling. On the same build and hardware, outcomes are bitwise |
| identical across: |
| - repeated runs; |
| - field reordering; |
| - adding or removing other fields; |
| - splitting or merging schemas. |
|
|
| - batching with other requests (`fill_many`). |
|
|
| The runtime batches per-field queries with batch-invariant kernels so that a field's |
| numerics don't depend on the batch composition (§12.2). |
| - **Generate phase:** given the same parallel outcomes, a generate field's text is |
| bitwise reproducible (greedy, or sampled with `seed`) and independent of the other |
| generate fields in the schema and of other requests decoding with it. |
| - **Post-processing** (argmax, tie-breaks, parsing, assembly) is pure Python over |
| the head outputs, with no clock, locale or randomness. |
|
|
| ## 11. What is measured against the constrained-decoding baseline |
|
|
| The baseline is the same base model (Qwen3.5-0.8B, at the same quantization) emitting the |
| whole JSON object under grammar-constrained decoding (a JSON-Schema-derived grammar), |
| greedy, from the same prompt. |
|
|
| The same eval set of (context, schema, gold JSON) is measured on both: |
|
|
| - **Tokens:** generated tokens per response, prompt tokens, and their total. |
| - **Latency:** end-to-end on one pinned CPU core (the deploy target), plus GPU throughput. |
| - **Accuracy per mode:** |
| - choice: exact match; |
| - score: MAE; |
| - noul: accuracy; |
| - span: exact match and token F1; |
| - number and date: exact match after parsing; |
| - null: precision and recall. |
| - **Calibration:** the ECE of decide probabilities and of extract confidences. |
| - **Validity:** the rate of schema-valid responses. deg must hit 100%, by construction. |
| - **Determinism:** bitwise equality across 10 repeats, 5 field permutations, and |
| split-vs-merged schemas. |
| - **jev gates:** the jev v1 eval gates on `/v1/systemone`. |
|
|
| ## 12. Model (v1 implementation) |
|
|
| Code: `src/deg/model.py` (trunk, forks, heads), `infer.py` (`fill`), `train.py`, |
| `evaluate.py`. Tests: `tests/test_model.py`. |
|
|
| ### 12.1 Encoding: one trunk, one fork per question |
|
|
| - **Trunk** = a fixed system/user prefix, the rendered context, `"\n"`. It contains the |
| context only, never the schema, so it is shared by every field and every request on |
| the same context. |
| - **Fork** = one continuation of a *copy* of the trunk state. Qwen3.5 is a hybrid: Gated |
| DeltaNet layers carry a recurrent state plus a 3-token causal-conv tail, attention layers |
| carry K/V. A fork starts from exactly that state, so it reads the whole context and |
| nothing else. Fork kinds: |
| - decide/extract leaf: `FIELD <concrete pointer>`, `ITEM i of <array>` per enclosing |
| array (index mode; anchored items use `ITEM <array> AT: "..."`, §12.6), `TITLE`, `DESCRIPTION`, `KIND`, extract |
| `CONSTRAINTS`, then `OPTIONS:` with one `- <option>` line per outcome, then `ANSWER:`. |
| Outcome text follows the plan order (choice: value + option description; score: |
| level, plus a null line for nullable scales; noul: no option lines). |
| - array count: `ARRAY <pointer>`, item anchors of enclosing arrays, the array's |
| title/description and item-field pointers, `ANSWER:`. |
| - generate: `KNOWN VALUES:` (every decided/extracted value, one `pointer = json` line |
| each, in leaf order; generate values are never included), `WRITE <pointer>`, |
| annotations, `LIMITS`, `VALUE:`, then the text and `<|im_end|>`. At inference the |
| KNOWN VALUES block is prefilled ONCE on the trunk and every generate field forks that |
| state with only its own header (no per-field restatement of the values). |
| - **Why forks and not an attention mask:** DeltaNet layers are recurrent, so a mask cannot |
| hide one field's tokens from the next; separate continuations can. The backbone is a |
| functional re-implementation of the HF Qwen3.5 stack over explicit state |
| (`Backbone.forward(ids, lengths, state)`); it matches the HF forward (fp32 relative error |
| < 1e-3 on GPU, `assert_close` on CPU) and lets gradients flow from forks into the trunk. |
| - **Padding is exact**: padded positions get `beta = 0` and `g = 0` in the delta rule (no |
| write, no decay), their K/V are masked, and the conv tail is gathered at each row's real |
| length, so a right-padded row gives the same state as an unpadded one. |
| - **Long contexts stream; there is no context cap.** Inputs up to `DENSE_PREFILL_MAX` |
| (8192) tokens are read in one shot (dense causal mask). Longer inputs prefill in |
| `PREFILL_CHUNK` (4096) token chunks, carrying each DeltaNet layer's recurrent state and |
| conv tail and appending each attention layer's K/V; on CUDA the chunk attends the K/V |
| buffer through the flash-style fork kernel (no mask tensor at all), elsewhere through |
| SDPA with a `[B, chunk, P+chunk]` mask. Nothing is O(L^2): peak memory is the K/V |
| (6 attention layers x 2 KV heads x 256 x bf16 = 12 KiB/token) + hidden states |
| (2 KiB/token) + one chunk's work (~0.46 GiB at 4096, independent of L). Span-pointer |
| keys are projected in 8192-row blocks (bitwise equal to one `bi_linear`), and span |
| candidates come from an `[T, MAX_SPAN_TOKENS]` band, never a T x T grid. Chunked |
| outputs equal the one-shot read within bf16 noise; at most 8192 tokens they are bitwise |
| the one-shot path. A context whose K/V does not fit the device fails with a CUDA OOM |
| (reported, never truncated); serving those needs K/V offload to host memory. |
| - **Nothing scales with fields x context on the device.** Fork batches are sized from the |
| memory free right now (`fork_token_budget`: `FORK_MEM_FRACTION` of free + cached-unused |
| memory at `FORK_BYTES_PER_TOKEN` ~128 KiB per padded fork token, capped at |
| `FORK_TOKENS`; OOM split stays as a backstop); after a fork pass only the rows the heads |
| read (answer + option tokens) are kept. Span logits for a long context or a big schema |
| are computed in answer-row micro-batches of at most `HEAD_LOGIT_BYTES` and context-row |
| blocks of `HEAD_ROWS` (keys re-projected per micro-batch, no `[T, 256]` copy), then wait |
| in host memory until decoding (bitwise the one-shot values). After field decoding the |
| raw head outputs and the trunk hidden states are released, and the KNOWN VALUES block |
| extends the context state in place (`forward(consume=True)`), so a context's K/V is |
| never held twice. Peak device memory = weights + context K/V + hidden + one fork batch. |
| |
| ### 12.2 Independence (bitwise) and the one-pass runtime |
|
|
| Inference runs every query of a request (array counts, array items speculated up to 8 |
| per array, every decide/extract field) as fork rows of ONE batch |
| (`DegModel.answer_many`; chunked only above 16k padded tokens, for memory), and every |
| x-generate field of a request as rows of one decode batch. `fill_many` puts the rows of |
| several requests in the same batches. A row's numerics never depend on the other rows |
| because every op in the fork path is batch-invariant (`src/deg/kernels.py`): |
|
|
| - **Prefixes by reference.** A fork row reads its prefix (context trunk, or trunk + |
| KNOWN VALUES) in place: attention reads the prefix K/V at per-row offsets, the |
| recurrent kernel starts from the prefix DeltaNet state by index. No per-row copy of |
| the context exists; per-row state is the fixed-size recurrent state, conv tail and the |
| row's own K/V. |
| - **GEMMs**: cuBLAS picks kernels (tiles, split-K) by shape, so a row's result changes |
| with the number of rows (measured on the L4: rows differ for many M). `bi_linear` is a |
| Triton GEMM with one fixed tile config and no split-K; `bi_linear_small` (decode steps |
| and the LM head) uses 16-row tiles and a split-K factor fixed by the weight shape, with |
| partials summed in a fixed order. Each call site always uses the same config, so a |
| row's output depends only on that row. Projections sharing an input are fused into one |
| GEMM (weights concatenated once; parameters become views). |
| - **Attention** (`fork_attention`): one program per (row, head, query block), prefix |
| keys then own causal keys in a fixed block order; fully masked blocks are exact no-ops |
| of the online softmax. |
| - **DeltaNet** (`fork_delta_rule`): FLA's fused recurrent gated delta rule, one program |
| per (row, head, value block), tokens in order; padded positions have `beta = g = 0`. |
| - **Norms**: fixed-config Triton row kernels; elementwise ops run eagerly. |
| `torch.compile` is not used in the fork path: it specializes size-1 dims, so one row |
| alone and the same row in a batch would run different generated kernels. |
| - **Heads** run batched in fp32 through `bi_linear` (IEEE fp32 dot). |
|
|
| `tests/test_model.py` checks with `torch.equal` that every field's raw head outputs are |
| identical alone, reordered, inside bigger schemas and in split vs merged schemas; that a |
| generate field's text is identical alone and next to another generate field; and that |
| `fill_many` equals `fill` per request, on CPU (tiny random Qwen3.5; the CPU reference runs |
| each fork row alone at a 64-token width grid because torch's CPU SIMD/scalar-tail split |
| depends on tensor size) and on GPU (the real model). Caveat: the trunk prefill (batch 1) |
| still uses the FLA chunk kernel and torch.compile'd ops with autotuning, so equality is |
| guaranteed within one process; across processes it holds when the autotune cache |
| matches (same build, hardware and cache). |
|
|
| **Decode step.** Each generate decode step (all rows x 1 token, LM head, argmax) is |
| captured once as a CUDA graph over static buffers (`DecodeGraph`: prefixes up to a |
| power-of-2 token capacity, rows bucketed 1/2/4/..., own-token capacity) and replayed. |
| Greedy takes the argmax when it is admissible and only otherwise walks the top 256 in |
| logit order (the same rule as the constraint walk). Measured on one L4 (bf16): ~7.3-7.9 ms |
| per step at batch 1 vs 36 ms before; the floor is weight bandwidth (~1.5 GB/step incl. the |
| LM head). |
|
|
| ### 12.3 Heads (fp32) |
|
|
| `a` = hidden state at `ANSWER:`, `o_i` = hidden state at the end of option line `i`, |
| `c_t` = trunk hidden state at context token `t`. |
|
|
| - `choice`: `q(a) · k(o_i) / sqrt(256) + u · o_i`, softmax over options (the jev v1 |
| pointer head). The option is encoded as text in the fork, so any enum works zero-shot. |
| Training shuffles option order (target permuted) with p = 0.5. |
| - `score`: a separate pointer head of the same form over the level lines. |
| - `noul`: `w · a + b` → logits `[l, 0]`, i.e. `[p_true, p_false]`. |
| - `extract`: start and end pointers `qs(a) · ks(c_t)`, `qe(a) · ke(c_t)` over the context |
| tokens only (prefix, suffix and query tokens are not addressable), plus a null logit from |
| `a` as one extra outcome. Decoding: |
| - numbers, dates, and email/uri/uuid strings: every candidate span the deterministic |
| grammar finds in the context (the same candidate regexes datagen used to locate gold |
| spans) is scored by `log p_start + log p_end` of its covering tokens; the best one |
| that parses and meets the constraints wins, unless null scores higher. |
| - plain strings: the top 32 token spans (`i <= j < i + 128`), mapped to code points |
| and trimmed of whitespace and boundary punctuation a token carried in (`trim_span`; |
| 98.7% of gold spans are recovered exactly by trimming the covering tokens). |
| - `options.null_margin` m (default 0) shifts that comparison: null wins when |
| `log p(best valid span) < log p(null) + m`. Decoding stays deterministic and |
| batch-invariant (the margin is per request, applied after the shared head pass). |
| - `forbid` fields ignore null and take the best valid span; none at all is a |
| `FillError` (inside an array item: the item is dropped, §12.6). |
| - confidence = `exp(log p_start + log p_end)` of the chosen outcome. |
| - `count`: linear over 0..64 items, masked to `[minItems, min(maxItems or |
| options.max_items or 32, 64)]`. Index-addressed arrays only (§12.6). Arrays resolve |
| level by level (a nested array's count fork needs its parent's item anchor). |
| - `anchor`: per context token `t`, an independent logit `qa(a) · ka(c_t) / sqrt(256) + |
| ua · c_t + b` that an item of the array starts at `t` (sigmoid; `b` starts at -6), |
| masked to the array's region (§12.6). |
| - anchored `extract`: inside an anchored item, the start/end logits of context token |
| `t` get a learned bias `B[start|end, inside, bucket(t - anchor)]`, where `inside` means |
| `anchor <= t < region end` and the buckets edge at -256, -64, -16, -4, 0, 1, 2, 4, ..., |
| 256 tokens (zero-initialized). |
| - `generate`: the base LM head (tied embeddings, frozen in training), greedy by default, |
| seeded multinomial (one stream per field) when `temperature > 0`. Constrained decoding: candidates are tried in |
| logit (or sampled) order among the top 256; a token is admissible if the decoded text |
| stays within `maxLength` and is a partial match of `(?s:.*?)(?:pattern)(?s:.*)` |
| (`regex` partial matching, i.e. JSON Schema search semantics); EOS is admissible only |
| when `minLength` and the pattern are met. `generated_tokens` counts EOS. |
| - Calibration: one temperature per head kind (`choice, score, noul, extract, count`), fit |
| by NLL on datagen `val` only (`deg.evaluate calibrate --source datagen-val`, default; |
| jev dev is reported separately with `--report-jev`, or pooled with `--source |
| datagen-val+jev-dev`), stored in `heads.safetensors`; `count` is not fit. Decide heads |
| fit the one-hot filled value (`--target hard`, default), the label accuracy and ECE |
| are judged against; the soft training targets (`--target soft`) are less confident |
| than the gold and over-soften (results/deg/bench-v2/calibration-v2.md). |
| |
| ### 12.4 Training (`deg.train`) |
|
|
| - Data: datagen records (`split == "train"`; val/test, including the held-out test |
| domains, are never trained on) plus jev v1 `train.jsonl` decisions (noul/choice/score |
| with teacher soft labels), mapped to one-field forks (`FIELD /answer`, the instructions |
| and criteria as text, score levels as `i: criterion` options). A micro-batch draws |
| records at random (jev with probability `--jev-frac`, default 0.3) up to a token budget. |
| `--reload-every N` re-reads the growing datagen files. |
| - Sources: `--records PATH[:WEIGHT]` is repeatable (default: the v2 `full-off` file, |
| weight 1). Each source reads its own `split == "train"` records (its own val/test labels |
| are kept out) into its own pool, and every reload re-reads all sources. Each deg |
| micro-batch picks one source with probability proportional to its weight among the |
| **active** sources, those with at least `--source-min-train` (default 50, summed over |
| ranks) train records, so a small, slowly growing source (e.g. |
| `/data/deg/datagen/v3-long/records.jsonl:0.25` next to v2 at 1.0 → 20%) gets its fixed |
| share as soon as it reaches the threshold. The jev mix stays per record inside every |
| micro-batch. End-of-run calibration fits on the val records of every source (contexts |
| within `--max-context-tokens`); the end-of-run eval uses the first source. |
| - Long contexts: a record whose context exceeds `--max-context-tokens` (default 6000) |
| is kept whole and trains a **span-aware window** at every draw (inference is unchanged: |
| it streams the whole context). The window is `max-context-tokens` minus the template |
| tokens, placed uniformly among the positions that contain one gold span picked at |
| random (spans and alternatives), so windows differ across epochs and cover the gold. |
| A cropped window trains only what its text shows: extract fields with a gold span (or |
| alternative) inside the window, and anchored arrays with at least one gold item start |
| inside it (the items starting outside are left out with their subtrees). Absent-gold |
| (null) extract fields, decide and generate fields (their evidence has no span) and |
| count heads train only on a whole context; nothing is ever trained as null or empty |
| because of cropping. Train logs report `windows`, `window_targets`, `window_dropped`. |
| - Decide cap per field (`--decide-forks-per-field K`, default 0 = off; `cap_decide_forks`, |
| applied first in every draw): at most K decide forks (choice/score/noul) per schema field |
| (array indices folded to `*`), a fresh random subset each draw. v3-long filings carry |
| ~135 choice forks per record, 99.8% one repeated in-array enum per table cell (`unit`, |
| `scale`, `statement_type`; 68% of their booleans are false); uncapped they are most of |
| the decide forks of every step that draws one. run-2b, which trained them uncapped, lost |
| decide accuracy on short inputs (datagen val top-level choice .939 -> .873 at step 4500; |
| JevBench choice 94 -> 74-92 of 139 across steps 500-6000, noisy; results/deg/regress-2b). |
| - Fork cap per draw (`cap_forks`): every drawn example (whole context, window, filler) |
| trains a fresh random subset of its forks when they exceed its room: `--record-fork-tokens` |
| when set, else the token budget beside its trunk and at least half the budget however |
| long the context. Whole-array forks (anchor, and anchored arrays' count) are always |
| kept; the other forks are taken in random order while they fit, the first one whatever |
| its length (so a generate fork longer than the room still trains). A record therefore |
| costs one trunk pass plus about the room in fork tokens per step, whatever its field |
| count: unsubsampled, a long filing (1,153 field rows, 176 generate forks each repeating |
| its ~16k-token known-value block) was 2.9M fork tokens and 260 micro-batches, a 20+ min |
| step. Loss terms stay means over the forks drawn (a subsample; forks longer than the |
| room are drawn less often). Train logs report `subsampled_records`, `subsampled_forks`. |
| - Training trunks have no 8192 cap: a trunk longer than `DENSE_PREFILL_MAX` is read in |
| `PREFILL_CHUNK` chunks with every layer of every chunk gradient-checkpointed |
| (`Backbone._forward_chunked_ckpt`; the mask is built inside the checkpoint, earlier |
| chunks' K/V are passed in as they are), so nothing O(L^2) is saved and one layer's |
| recompute is O(chunk x L). It equals the dense checkpointed read in hidden states, |
| state, fork outputs and gradients (`tests/test_train_long.py`). The memory planner |
| models the chunked trunk separately (per-chunk recurrent state, the concatenated K/V |
| copy, no saved mask, chunk-sized recompute). |
| - Row chunks share one trunk pass: a record too big for one micro-batch is split by the |
| memory planner into field-row chunks (`MemPlanner.split_rows`); `run_step` runs them on |
| ONE trunk forward (`run_chunks`): the trunk outputs (final hidden, recurrent state, conv |
| tails, K/V) are detached into leaves, each chunk's forks + heads backpropagate into the |
| leaves, and the trunk backpropagates once from their accumulated gradient (equal to the |
| unsplit record's gradient, dense and chunked trunks alike: `tests/test_train_mem.py`). |
| An out-of-memory error in such a run drops its gradients and reruns the chunks one by |
| one with their own trunk passes (`--no-share-trunk` always does). Logs: `trunk_shared`. |
| - Losses, each a mean over its items: |
| - `decide`: KL(t ‖ q). `t` is the record's `decide_probs` (Clef probabilities where |
| Clef's argmax matched the filled value), else one-hot of the filled value; jev uses |
| `probs`, else `hard`. |
| - `extract`: start and end CE where every `span_alternatives` entry is a positive: |
| with `span_probs` the target is that soft distribution; otherwise the loss is the |
| marginal `-log Σ_pos p`. A field absent from the filled value targets null. |
| - `count`: CE of the array length (index-addressed arrays). |
| - `anchor`: per anchored array instance, BCE of every in-region token against the gold |
| item starts, summed and divided by 1 + the number of items (§12.6). |
| - `lm`: next-token CE on x-generate text + EOS, teacher-forced on gold values; the |
| prompt is the inference layout (KNOWN VALUES = decided/extracted values only). |
| - `anchor_count` (`--anchor-count-weight W`, default 0 = off): with W > 0 every |
| anchored array of a whole context (never a cropped window: a count needs the whole |
| array) also gets the index-mode count fork, CE of its gold item count over |
| `[minItems, min(maxItems | MAX_COUNT, MAX_COUNT)]` with the target capped at the top |
| class (which then means "at least that many"), so `count_peaks` decoding has a |
| trained count. |
| - total = `decide + extract + 0.5 count + anchor + W anchor_count + L lm` |
| (`--anchor-weight`, `--anchor-count-weight`, `--lm-weight L`, default 1). |
| - Filler augmentation (`--filler-frac P`, `--filler-corpus`, default |
| `/data/deg/external/v3/docs.jsonl`, `--filler-max-tokens N`, default 16384): the |
| anchor head under-fires on long documents when it has only seen short ones. With |
| probability P a drawn short datagen record is rebuilt with its document at a random |
| offset inside unrelated text (`pre + "\n\n" + doc + "\n\n" + post`), total length |
| log-uniform from its own length + 64 to `min(N, --max-context-tokens)`; every gold |
| span shifts with it and every target is the plain example's (the filler is unrelated |
| text). A `--filler-negative-frac` share (default 0.25) is filler only: the record's |
| top-level anchored arrays with minItems 0 train their anchor fork with no item start |
| (every token negative) and count 0; nothing else trains on it. Filler comes only from |
| v3 **train-split** documents (`datagen.v3.split_of`), never the record's own document |
| and never ExtractBench (refused by path). Draws are placeholders sorted at their planned |
| length and materialized when the packing reaches them, like long-document windows, so |
| the packing and the memory planner see real examples; a rebuilt example that would |
| exceed `--max-context-tokens` falls back to the plain one. Logs: `filler`, |
| `filler_negative`, `filler_tokens`. |
| - Optimization: bf16 weights with fp32 master copies (AdamW, lr 2e-5 trunk / 1e-3 heads, |
| warmup then cosine to 10%), embeddings and the tied LM head frozen, per-layer gradient |
| checkpointing, elementwise ops fused with `torch.compile`. Forks are length-grouped |
| (≤25% padding). Multi-GPU: `torchrun`, one data shard per rank, explicit gradient |
| all-reduce. Logs: losses, tokens/s (real and padded), GPU utilization and memory. |
| |
| ### 12.5 Smoke run and full-run launch |
|
|
| Smoke (2026-10-08, one L4, `/data/deg/runs/smoke-1`, 300 steps, budget 14k tokens, |
| ≤32 records/step, jev 20k-record subset at 30%): total loss 13.0 → ~1.5 (decide 1.54 → |
| 0.15, extract 8.2 → ~0.5, lm 1.83 → ~0.5, count 2.7 → ~0.8); ~4.0k real tokens/s |
| (4.7k padded), 85-92% GPU utilization, 14.9 GiB peak. Eval through the full `fill` path |
| after calibration: |
|
|
| | split | records | valid | decide acc / ECE | extract EM | count acc | generated / full-JSON tokens | |
| |---|---|---|---|---|---|---| |
| | val | 29 | 100% | 0.93 / 0.097 | 0.79 | 0.73 | 0.33 | |
| | test (held-out domains) | 60 | 100% | 0.90 / 0.071 | 0.85 | 0.70 | 0.29 | |
| | jev dev (1312 questions) | | | 0.81 / 0.024 | | | | |
|
|
| Full run (4 GPUs; not launched by default — the orchestrator allocates GPUs): |
|
|
| ```bash |
| systemd-run --user --unit deg-train-full --property=MemoryMax=80G \ |
| --setenv=CUDA_VISIBLE_DEVICES=0,1,2,3 --setenv=PYTHONUNBUFFERED=1 \ |
| --setenv=PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ |
| --working-directory=/shared/rwkv/packages/deg \ |
| /bin/bash -c 'exec /shared/rwkv/.venv/bin/torchrun --standalone --nproc-per-node 4 \ |
| -m deg.train --out /data/deg/runs/full-1 --steps 20000 --budget 14000 \ |
| --max-records 32 --jev-frac 0.3 --reload-every 500 --save-every 1000 \ |
| --eval-records 200 --eval-jev 1000 > /data/deg/runs/full-1.log 2>&1' |
| ``` |
|
|
| ### 12.6 Position-anchored array items (`array_mode = "anchor"`, run-2 default) |
| |
| Run-1 addressed array item `i` by index ("ITEM i of /arr") and sized arrays with a count |
| head over 0..64. On real documents that failed: array-item leaves were 10% correct vs 45% |
| for scalars, forcing the gold counts did not help, and nested counts copied the parent's |
| index (results/deg/extractbench-v1/DIAGNOSIS.md, cause d). Anchor mode addresses an |
| item by **where it is**, not by its ordinal: |
| |
| - **Which arrays.** An array whose item subtree has an extract leaf is anchored. Arrays of |
| only decide/generate leaves (no span can say where an item is) keep the count head and |
| index addressing. The mode is static per schema and checkpoint |
| (`model.meta["array_mode"]`; a checkpoint without anchor heads, i.e. run-1, loads as |
| `index` and behaves exactly as before; `deg.train --array-mode anchor|index`, default |
| `anchor`). |
| - **Routing by document length** (`model.meta["anchor_min_tokens"]`, default 0 = off; |
| `deg.train --anchor-min-tokens N`, saved in `deg.json`). An anchor-mode model fills a |
| document of fewer than N context tokens exactly as index mode (count fork + `ITEM i` |
| forks), and trains its whole-context records the same way; documents of N or more |
| tokens, filler-augmented records that reach N, and every cropped window (its document is |
| longer than any window) anchor. Why: on short datagen records run-2b step-4500 in index |
| mode beats its own anchor mode (test array-item EM .844 vs .769 count_peaks, extract EM |
| .896 vs .866; run-1 index .891 / .922), while anchors win on long documents |
| (ExtractBench; results/deg/regress-2b). |
| - **Anchor fork.** One fork per array instance: `ARRAY <ptr>`, the enclosing items' |
| locators, title/description, `ITEM FIELDS`, `KIND: mark where each item starts in the |
| context`, `ANSWER:`. The anchor head scores every context token of the array's |
| **region**; tokens with p > 0.5 are marked, each run of adjacent marked tokens is one |
| item (its best token). Fewer than `minItems`: the best remaining non-adjacent in-region |
| tokens are added (none left: `FillError`); more than `maxItems` (or `options.max_items` |
| when given): the highest are kept. **No other cap**: the count is the number of |
| anchors, and item forks run in the usual memory-bounded fork micro-batches. |
| - **Decode modes** (`options.array_decode`, default `count_peaks`). `threshold`: as above. |
| `count_peaks`: the array also runs its count fork (the index-mode count query, trained |
| on anchored arrays by `anchor_count`, §12.4); N = its argmax over `[minItems, |
| min(maxItems | max_items | MAX_COUNT, MAX_COUNT)]` (at the saturated top class, the |
| threshold count when larger), capped by maxItems; the items are the N best local |
| maxima of the anchor logits with non-maximum suppression (`PEAK_RADIUS` = 4 tokens: |
| one peak per item), then the best remaining tokens outside every radius if too few |
| peaks exist. `hybrid`: threshold, unless no token passes p > 0.5 and the count says |
| N > 0, then count_peaks. The count fork is one more independent fork and peak picking |
| is a pure function of the logits, so every mode is deterministic and batch-invariant. |
| Default chosen on datagen val only (`results/deg/array-decode-v1`, run-2b step-3000, |
| 131 records / 153 anchored arrays, paired): aligned item-leaf EM threshold .609, |
| count_peaks .553, hybrid .659 (count exact .575 / .784 / .641); with 4k tokens of |
| filler before the record: .428 / .482 / .572 (count .342 / .704 / .513). |
| `PEAK_RADIUS` 2/4/8 changes count_peaks/hybrid leaf EM by at most .03. |
| Default re-chosen at run-2b step-4500 (count head trained by `anchor_count` since step |
| 4000; `results/deg/regress-2b`): count_peaks vs hybrid vs threshold, datagen val count |
| .846 / .577 / .558, extract EM .828 / .818 / .816; datagen test (200) count .850 / .740 / |
| .690, extract EM .866 / .847 / .842; ExtractBench normalized 239 vs 221 (hybrid), |
| aligned 339 vs 338, count_ok 132 vs 111, valid docs 28 vs 27. |
| - **Regions.** A top-level array's region is the whole context. Item `k`'s region runs |
| from its anchor to the next sibling anchor (the last item's to its parent's region |
| end). A nested array searches anchors only inside its parent item's region, so a child |
| count cannot copy the parent's count or index. |
| - **Item forks.** Every fork inside an anchored item renders anchored indices as `*` |
| (`FIELD /results/*/time`) and one locator line per enclosing anchored item: |
| `ITEM /results AT: "<context text from the anchor, at most 24 tokens, cut at the region |
| end>"`. Extract forks also get the anchored span bias (§12.3) from the innermost |
| anchor and region. There is no ordinal anywhere in an anchored item's forks. |
| - **Items that do not exist.** An anchor only says where an item would start; the item |
| exists only if every required non-nullable (`on_null: forbid`) extract leaf in it has a |
| valid value. When one has none (`forbid` already ignores null and searches every valid |
| candidate in the context, so this means no span parses/meets the constraints at all — |
| e.g. a `format: email` string item anchored in a résumé with no address), the |
| innermost anchored item holding that leaf is **dropped**: never a `FillError`, never a |
| made-up value. Survivors keep their anchors, regions and fork outputs (no fork sees an |
| anchored item's index, so nothing is re-run and their values are bitwise unchanged) |
| and are renumbered in document order; the array's meta gets `"dropped": n`; generate |
| fields of a dropped item are not decoded and its values are not in KNOWN VALUES. If the |
| drops leave an array below `minItems`, its enclosing anchored item cannot be |
| represented and is dropped in turn; a top-level array left below `minItems` is a |
| `FillError` (`minItems N: only k items have their required values`), like an anchor |
| fork with too few positions. There is no "replacement anchor" fallback: for typed |
| leaves (numbers, dates, email/uri/uuid) the valid candidate set is the whole context |
| and does not depend on the anchor, so another anchor cannot succeed where this one |
| failed. Dropping is a pure function of each item's own fork outputs, so it is |
| deterministic and batch-invariant. It does not depend on `array_decode` (which only |
| picks the anchors). **Index-addressed items** (index-mode models, documents routed to |
| index mode below `anchor_min_tokens`, arrays without extract leaves) follow the same |
| rule: the innermost enclosing item — anchored or index-addressed — is dropped and the |
| survivors renumbered with their fork outputs unchanged (their `ITEM i of` forks are |
| not re-run); generate forks of survivors render the new index. A required |
| non-nullable value outside every array item keeps the §4 `FillError`. |
| - **Rounds.** Anchored arrays add one fork round per nesting level (anchors first, then |
| the items' forks and nested anchor forks); index arrays keep the speculative one-pass |
| layout. |
| - **Order.** Items come out in document order (anchor order). `deg.evaluate` reports |
| positional `extract_em` and order-insensitive `extract_em_aligned`; |
| `deg.extractbench.score_record(..., align=True)` and the ExtractBench eval report |
| `normalized_aligned_rate` next to the positional rates. |
| - **Independence.** An anchor fork depends only on the context and its own query (array |
| pointer, enclosing locators, annotations); an item's forks depend only on the context, |
| their own query, and the item's anchor and region. Both are bitwise identical alone, |
| inside a bigger schema and reordered (`tests/test_anchors.py`). |
| - **Training gold.** From the datagen spans (`spans`, else the first `span_alternatives` |
| entry): an item's anchor is the first token of its earliest span that no sibling item |
| shares (a shared span such as a table column header does not locate an item), else of |
| its earliest span. An anchored array instance whose items do not all get distinct |
| anchors inside the region is left out of training (counted as |
| `anchor_skipped_arrays` in the data log); an empty gold array trains an anchor fork with |
| no positives (the hard negative for nested arrays). |
|
|
| ### 12.7 Who writes x-generate text: hybrid and LoRA (deg-suite v1 follow-up) |
|
|
| deg-suite v1 found the untrained base Qwen3.5-0.8B writes better x-generate text than |
| every fine-tuned checkpoint (judge faithful 0.94 vs 0.27 on DialogSum, 1.59 vs 0.43 on |
| mixed, out of 2; lengths similar): full fine-tuning made generated text unfaithful. |
| Three ways to keep the base model's generation, all deterministic and batch-invariant: |
|
|
| * **Two-copy hybrid** (`load_model(..., generate="base")`, `--generate base` on the server |
| and the suite): a frozen base body sits beside the fine-tuned one (token embeddings = |
| tied LM head shared when equal, +~1 GB bf16). Decide/extract are unchanged; for |
| x-generate the base body re-reads trunk + KNOWN VALUES in one pass (one more context |
| read) and decodes the header of :func:`deg.infer.base_generate_header`, which closes |
| the user turn with a plain instruction and opens the assistant turn (thinking off). |
| * **LoRA** (`deg.train --lora-rank R`): the backbone stays the base; named low-rank |
| adapters on every attention / linear-attention / MLP projection (`lora_A` [n*r, in], |
| `lora_B` [out, n*r], B = 0 at start) train with the heads. A per-row selector picks |
| each fork row's adapter (`--adapters shared` or `decide,extract`: by fork kind; choice, |
| score, noul -> decide; extract, count, anchor -> extract), so rows with different |
| adapters share one batch and stay bitwise independent (the delta goes through the same |
| bi_linear GEMMs). x-generate rows run adapter-off = the base model, never trained. |
| `--lora-context base` (default): the context is read ONCE by the base weights and |
| every fork (adapted decide/extract rows, adapter-off generate rows) continues it, so the |
| one-pass structure holds and the trunk has no gradient in training. |
| `--lora-context <adapter>`: that adapter also reads the context; generation re-reads it |
| adapter-off (2x prefill for requests with x-generate fields). |
| Checkpoints keep the base `backbone.safetensors` and add `adapters.safetensors` + |
| `deg.json["lora"]`. |
| * **GENERATE LoRA on the hybrid's base copy** (`--generate base+lora:<dir>`; |
| `load_model(..., generate="base+lora:<dir>")`): the two-copy hybrid, with one low-rank |
| adapter trained by `deg.gentrain` merged into the frozen base copy at load |
| (`W += alpha/r * B @ A`, fp32 on the CPU, cast to the body dtype; deterministic). |
| Deg's own backbone, heads and the one-pass flow are unchanged; generate rows stay |
| batch-invariant (the merged copy is an ordinary frozen body); a zero adapter is the |
| plain hybrid. A generator dir holds `generator.json` (`format` |
| `deg-generator-lora-v1`, rank, alpha, training meta) + `lora.safetensors` |
| (`<module>.lora_A` [r, in], `<module>.lora_B` [out, r], named as in the HF |
| `Qwen3_5ForCausalLM`). Training (`python -m deg.gentrain`): SFT on x-generate fields |
| in exactly the hybrid's layout (context + KNOWN VALUES + `base_generate_header`, target |
| = value + EOS; `deg_example(..., base_layout=True)`), the adapter on every |
| `LORA_TARGETS` projection reading the whole sequence; data = gen-v1 `train` (suite |
| guard; open-weight writer + judge only, `check_open_teacher`; fields the judge kept at |
| faithful 2 / responsive 2; targets over the field's decode budget dropped, never cut); |
| `val` token NLL selects the checkpoint (early stopping); `--kl-base W` adds |
| W * KL(base || adapted) on the target tokens. |
| * **Anti-forgetting for full fine-tunes** (`--lm-kl-base W`): KL(frozen base || deg) on |
| the x-generate target tokens, computed by the frozen base on the identical trunk + fork |
| tokens (no grad). |
|
|
| ### 12.8 Pluggable backbones (`--backbone`, `deg.json["backbone"]`) |
|
|
| The trunk/fork runtime, the heads, training and the memory planner do not depend on the |
| architecture beyond one `Family` of layer functions (`model.py`): every supported base is |
| a hybrid of **state layers** and **attention layers**. `State.lin[i]` = (recurrent state, |
| conv tail) per state layer, `State.kv[i]` = K/V per attention layer; `ForkState` reads the |
| prefix K/V by reference and copies only the fixed-size per-row state, so prefill once, |
| fork-by-reference, chunked prefill (inference and checkpointed training), the CUDA-graph |
| decode step and the two-copy hybrid are shared code. A family supplies: the trunk/training |
| state layer (`rec_layer`), attention projections and output (`attn_qkv`, `attn_out`), the |
| batch-invariant fork versions of both, per-row state shapes, the final norm, the rotary |
| call and the LoRA targets. |
|
|
| | `--backbone` | layers | per state layer | per attention layer | tokens | |
| |---|---|---|---|---| |
| | `qwen3.5-0.8b` (default) | 18 Gated DeltaNet + 6 attention, hidden 1024 | recurrent [16,128,128] fp32 + 3-token conv tail | K/V 2 heads x 256 | ChatML, `<think></think>` gen turn | |
| | `qwen3.5-2b` (`--base-model Qwen/Qwen3.5-2B`) | same 18 + 6 layout, hidden 2048, MLP 6144 | as the 0.8B | as the 0.8B | as the 0.8B | |
| | `lfm2.5-1.2b` (LiquidAI LFM2.5-1.2B-Instruct) | 10 gated short-conv + 6 GQA attention (32 q / 8 kv heads x 64), hidden 2048, MLP 8192, tied embeddings, vocab 65,536 | none (empty `[B,0]`) + 2-token tail of `B*x` | K/V 8 heads x 64 | `<\|startoftext\|>` BOS + ChatML, pad `<\|pad\|>`, gen turn without a thinking block | |
|
|
| LFM2 conv layer: `B, C, x = in_proj(norm(h))`, `y = C * conv3(B * x)` (depthwise, no |
| activation), `h += out_proj(y)`; attention has per-head q/k RMSNorm and plain 1-D RoPE; |
| norms are `w * norm(x)`. Both families have K/V = 12 KiB/token (6 layers x 2 KiB). |
|
|
| - **Determinism (LFM).** The fork path uses the same kernels as Qwen3.5: `bi_linear` / |
| `bi_linear_small` for every projection (fused q/k/v and w1/w3), `fork_attention` (head |
| dim 64, GQA group 4), a fixed-config Triton row kernel for the LFM norm |
| (`kernels.rms_rows_w`), and an eager elementwise short conv accumulated in fp32 in a |
| fixed tap order (per element, so batch-invariant; no conv kernel is needed). No fallback |
| path was required. `tests/test_lfm_backbone.py` checks (tiny random LFM2 + the real |
| tokenizer, CPU) the functional stack against HF `Lfm2ForCausalLM`, every fork row |
| against an independent full forward of context + its query and bitwise alone vs |
| batched, keep/decode continuation, chunked prefill = one-shot (inference, padded rows, |
| prefix state, checkpointed training incl. gradients), and runs the `test_model` / |
| `test_hybrid_generate` independence and hybrid-generate contracts against the LFM |
| fixtures; on CUDA (real model) bitwise alone / reordered / bigger schema / fill_many, |
| generate rows next to other generate rows, HF match, chunked prefill match. |
| - **Checkpoints** record `backbone`; a checkpoint loads its own base (pre-§12.8 |
| checkpoints are Qwen3.5-0.8B) and refuses another `--backbone`; `--generate base` |
| attaches that backbone's frozen base. Token budgets (`--max-context-tokens`, |
| `--anchor-min-tokens`, `--budget`) count the backbone's own tokens. |
| - **Memory.** `MemCoef.for_backbone` gives analytic coefficients per backbone (calibrated |
| online by the measured scale as before). LFM2.5-1.2B full fine-tuning keeps ~15.7 GiB |
| resident (bf16 weights 2.2 GiB + fp32 master/AdamW 12 B and bf16 grads 2 B per trainable |
| parameter, 1.04B trainable with the embeddings frozen) vs ~6.8 GiB for Qwen3.5-0.8B, so |
| it needs a higher `--mem-target-gb` on a 24 GB L4. |
| - **Qwen3.5 is size-generic.** `--backbone` / `--base-model` accept `qwen3.5-<size>` or |
| the HF id (`Qwen/Qwen3.5-2B`); every Qwen3.5 size is the one `QWEN35` family with the |
| same tokenizer, chat layout and gen turn, and the backbone and heads read layer types, |
| hidden/MLP widths and head counts from the snapshot's config. `qwen3.5-2b` has its own |
| memory coefficients (2x saved hidden per token, ~1.9x wider layer recompute; same |
| recurrent state, K/V and vocabulary). `tests/test_qwen2b_backbone.py` runs the model, |
| hybrid-generate and long-read contracts on a tiny 2B-shaped Qwen3.5 (hidden 2x, MLP |
| 3x hidden), checks the real 2B config sizes backbone and heads (meta device), that a |
| 2B checkpoint refuses other backbones and `--generate base` loads the frozen 2B, and on |
| CUDA the real 2B's bitwise independence and HF match. |
|
|
| ### 12.9 Data-parallel training: ZeRO-2 over several GPUs (`torchrun`, `deg.zero`) |
|
|
| `torchrun --nproc-per-node N -m deg.train ...` trains one model on N GPUs. Each rank |
| reads its own uid shard of every source (`zlib.crc32(uid) % N`), draws its own |
| micro-batches with the per-rank `--budget` and memory plan, and the step's gradient is the |
| gradient of the **global** step loss: each loss term's item counts are all-reduced before |
| the forward (`global_counts`), each rank's term is its share of the global mean, and the |
| ranks' gradients are **summed** (not averaged; a rank with more decide items carries more |
| of the decide gradient, exactly as one batch of all ranks' records). One global batch is N |
| per-rank micro-batch sets (N x `--budget` tokens). Gradient clipping uses the global norm. |
| A rank that cannot finish its step (minimal unit OOM) makes all ranks redraw |
| (`all_ranks_ok`), so every optimizer step is complete on every rank. |
|
|
| **ZeRO-2** (`deg.zero.ZeroMaster`, default with N > 1; `--no-zero` = whole masters per |
| rank). bf16 compute weights stay whole on every rank (the functional backbone calls its |
| layers' weights directly). The fp32 master weights, both AdamW moments and the fp32 |
| gradient are sharded: the trainable parameters form one flat vector cut into 32M-element |
| buckets; rank r owns the r-th of N pieces of each bucket. A step's bf16 gradients stay on |
| the parameters until the step's last part (several-part steps sum them in a pinned host |
| fp32 vector, so the device never holds two gradient copies), then a bucketed fp32 |
| reduce-scatter (SUM) gives each rank its shard's gradient; AdamW steps the shard (one flat |
| parameter: element-wise the same arithmetic and decay as the unsharded trainer); each |
| rank rounds its shard to bf16 and an all-gather refreshes every rank's compute weights |
| (bitwise `p.copy_(master)`). Per trainable parameter a rank keeps 2 B (bf16 grad) + |
| 16/N B instead of 18 B: Qwen3.5-2B (1.37B trainable, embeddings frozen) on 3 L4s holds |
| ~3.8 GiB weights + 2.6 GiB grads + 6.8 GiB shard state = ~13 GiB static vs ~26 GiB |
| unsharded. The memory planner uses this static (`sharded_static_bytes`), then the |
| measured resident memory as before. |
|
|
| - **Start/resume.** Heads (fresh, per-rank seeds) and backbone are broadcast from rank 0 |
| at start. Every rank keeps the whole bf16 weights, so rank 0 saves an ordinary deg |
| checkpoint (`step-N/`, same format as one GPU; no sharded files, no conversion). |
| `--init <run>/step-N --start-step N` resumes: weights from the checkpoint, the cosine |
| schedule continues at step N, optimizer moments start fresh with a `--warmup` re-warm |
| (as on one GPU), the data order is re-drawn (the rng mixes in N). The process group |
| timeout is 4 h (rank 0 alone calibrates, evaluates and saves at the end). |
| - **Smoke on one GPU.** `--zero-sim-world N` shards as rank 0 of N ranks with no |
| collectives: the real per-rank memory and compute time on one GPU (other shards are not |
| updated; never a real run). |
| - `tests/test_zero.py` (CPU, gloo, 3 processes): sharded masters + AdamW + global clip |
| equal one process stepping whole fp32 masters on the summed gradient, for one-part |
| (device bf16 grads) and two-part (host fp32 sum) steps, replicas bitwise equal; the |
| flat layout covers every element once; global counts make two ranks' gradients and |
| losses sum to the one-batch ones. |
|
|
|
|
| ### 12.10 Grounded extraction on media (`deg.ground`) |
|
|
| A text extract field copies a span and `meta.span` says where it came from. On a media |
| context the same extract field may point into an image or audio item; then meta gives a |
| box or a time window (§8). There is no annotation: the mode follows from the field's type |
| (§3), and text vs media follows from where the pointer lands. |
|
|
| - **Pointer.** The extract fork and span head are unchanged; they already score every |
| context token, the media **soft tokens** included. `landing`: when the start pointer's |
| probability mass on the soft tokens beats both the text tokens' and null's, the field |
| is located in media (`decode_ground`: the best pair inside one item, logits |
| renormalized over soft tokens + null); otherwise the text path runs exactly as before |
| with the soft tokens masked. No parameters are added and a context without media never |
| takes this path, so text checkpoints load unchanged and text fills are bitwise |
| identical. |
| - **Image geometry.** Gemma 4 resizes an image (aspect-preserving) to multiples of 48 px |
| with at most `max_soft_tokens * 9` 16-px patches and average-pools 3x3 patches: soft |
| token `k` is cell `(k mod gw, k div gw)` of a `gw x gh` grid in raster order |
| (`ground.image_grid`, tested against HF's resize and the tower's position ids). |
| `start` = the box's top-left cell, `end` = its bottom-right cell: `start <= end` in |
| raster order, the span head's own order. Valid pairs have `col(end) >= col(start)` and |
| `row(end) >= row(start)`; cells map back to source pixels (`cells_to_box`). |
| - **Audio geometry.** One soft token per 40 ms (`audio_ms_per_token`): `start`/`end` are |
| the window's first and last token, `[start * 0.04, (end + 1) * 0.04]` s, clipped. |
| - **Value from the region: crop re-read.** A media item has no text to copy, so the |
| located region (padded 15 % per side, at least 48 px; audio: the window +-0.5 s) is cut |
| from the source and filled as the only media item of a one-field x-generate schema |
| (`ground.read_schema`: the leaf's key, title/description, maxLength); the text is then |
| parsed by the leaf's mode like a copied span (`response.extract`: number/date parsing, |
| constraints). All reads of a `fill_many` batch run as one nested batch, before the |
| KNOWN VALUES block needs them. Chosen over a region-masked attention bias because (1) at |
| a receipt's 280-token budget one soft token covers ~48x48 resized px, often more than a |
| printed word; a crop at the same budget reads the region at several times the |
| resolution, which masking cannot add; (2) it reuses the trunk's media path unchanged: no |
| change to the fork attention kernels, the CUDA decode graphs or the text path; (3) its |
| training data are ordinary media records. Media given as embeddings only (`Media` parts, |
| no source file) can be located but not read (`null_reason: "media_unread"`). |
| - **Training.** A media record's `grounds` (concrete pointer -> `{"media", "box"}` / |
| `{"media", "time"}`, or null = not in the media) gives an extract leaf's pointer target: |
| the gold corner soft tokens (`ground_target`; null -> the null slot), in the normal |
| `extract` term (`extract_media` logs its share). A present value with neither a text |
| span nor a ground entry trains no pointer (never null). Crop-read records |
| (`records.read.jsonl`, family `mm-read-v1`) are the gold region cropped exactly as |
| inference crops (same `crop_sha` key, so the media cache serves both) with the leaf's |
| read schema and gold value. |
| - **Gold.** `python -m deg.datagen.mm_ground --set cord-v2`: CORD-v2 word boxes |
| (`ocr_boxes`, with their line category) matched to the gold values (items |
| name/quantity/price, totals); a matched field turns from x-generate into a plain string |
| (an extract field) in `records.ground.jsonl` and the `.ground` eval files. A value with |
| no exact same-category word run gets no entry. invoices-donut (no boxes), |
| RICO-Screen2Words (no view hierarchy, no localized field), MINDS-14 and AMI (no word |
| timings in these sources) stay value-only. |
| - **Eval.** `python -m deg.bench.mm_eval run --ground` fills `<set>.<split>.ground.jsonl` |
| and reports, beside (never in) the set score: box IoU, IoU >= .5 rate, containment |
| (share of the gold box inside the predicted one), gold-centre hit, time IoU and |
| containment, and null accuracy for gold-null entries. A field not located scores 0. |
| - `tests/test_ground.py` (CPU, tiny Gemma 4 hub): geometry against HF and the towers, |
| corner decoding, landing and null, CORD matching and crop-read records, a training step |
| with image and audio pointer targets, fills that locate and read values (and the text |
| path when the pointer lands in text), and the metrics. |
|
|
