source/SPEC.md

deg contract v0.1

Status: contract and schema compiler implemented (src/deg/schema.py, response.py, request.py, parsers.py); model, trainer and in-process runtime implemented (model.py, train.py, infer.py, evaluate.py, design in §12); HTTP server and Python SDK implemented (api.py, server.py, client.py, §9).

1. Model

• Base model: Qwen3.5-0.8B (default; LFM2.5-1.2B-Instruct is the alternative backbone, §12.8), with three kinds of heads on one trunk:
  • decision heads: choice, score, noul (as in jev v1);
  • a span pointer (start/end over context tokens, plus a null outcome);
  • the LM head, used only for x-generate fields.
• Model id: deg-qwen3.5-0.8b-0.1 (alias deg-latest).

2. The one flag (and two annotations)

x-generate: true is the only extension that changes a field's mode.

• It is allowed only on type: string or on type: array whose items is a plain type: string (no enum, no format). On an array, it applies to every item.
• x-generate: false means the same as leaving it out.

Two annotations refine a field without changing its mode:

• x-max-tokens: N (integer >= 1, only where x-generate applies; on an array it applies to every item): that field's own decode budget, overriding options.max_new_tokens, so one request can mix budgets.
• x-levels: {"<level>": "label", ...} (only on an integer score field, keys among its admissible levels, labels non-empty strings): each labeled level's option line becomes <level>: <label> (the layout the score head was trained on from jev level criteria); unlabeled levels keep the bare number. Use it instead of listing level meanings in the description.

Any other x-* keyword is an error, so a typo like x-genrate fails loudly.

There is no router head. A field's mode follows only from its JSON type and keywords (§3). The model never chooses between deciding, extracting and generating.

3. Type → mechanism

Schema (no x-generate)	Mode	Output
enum (any scalar types, 2..255 values; null allowed as a value)	decide.choice	the value, its probability, the full distribution, confidence
oneOf/anyOf whose branches are all const/enum	decide.choice	the same; branch descriptions become option descriptions
boolean	decide.noul	p(true); the value is p >= 0.5
["boolean","null"]	decide.choice over [true, false, null]	as choice
integer whose bounds admit 2..11 values (after multipleOf)	decide.score	the most probable level, the distribution, the expected level
every other number: any type: number, or an integer with a missing bound or more than 11 admissible values	extract.number	a span plus a deterministic number parser; bounds only validate
string with format: date / date-time	extract.date	a span plus a deterministic date parser, giving an ISO 8601 / RFC 3339 value
string with format: email / uri / uuid	extract.span	a verbatim span that must pass the format check
string	extract.span	a verbatim span
array	per item	the length is a runtime count; each item uses the item schema's mode
object	recursive	the properties, in schema order

Schema with x-generate: true	Mode
string (optional minLength, maxLength, pattern)	generate
array of string	generate per item

Rules of thumb that follow from the table:

• Number rule. Only a small integer scale is a judgment: an integer whose bounds give at most 11 possible values (1..5, 0..10, 0..100 step 25) is decided. Every other number is a quantity copied from the context. Bounds (minimum, maximum, exclusive bounds, multipleOf) never turn a quantity into a decision; they only validate the extracted value. There is no expected-value path for numbers: deg never computes a number that is not written in the context. To get a judgment on a finer or non-integer scale, use an integer scale or an enum.
• Extracted values are always copied from the context. Parsers only normalize them; they never invent digits or dates.
• x-generate fields are the only source of new tokens.

4. Null, optional and constraints

• Nullable means type: [X, "null"], anyOf/oneOf: [X, {type: null}], or an enum that lists null. Nullable objects and arrays are not supported.
• Decide fields are always filled, whether optional or not. Nullable score fields get an extra null outcome, and nullable booleans become a 3-way choice.
• When an extract field finds nothing, or its span fails the parser or a constraint, the plan's on_null decides what happens:
  • null: the field is nullable, so the value is null;
  • omit: the field is optional and not nullable, so the key is left out;
  • forbid: the field is required and not nullable, so the extractor must return its best valid span. If it can't, that is a runtime error (FillError, HTTP 500), never a made-up value. This is deliberate: authors who want a fallback mark the field nullable (type: [X, "null"]) and get null with a null_reason instead. Exception: inside an array item (anchored or index-addressed, §12.6) a forbid leaf with no valid value means the item does not exist: the item is dropped (not an error), the array's meta says "dropped": n, and a minItems the drops break is handled as in §12.6 (drop the enclosing item, else a FillError at the top level).
• meta.null_reason is one of not_found, unparsable, or the name of the violated keyword.
• Constraints are enforced as follows:
  • Spans are filtered by minLength, maxLength, pattern (searched, as in JSON Schema) and format.
  • Numbers are filtered by minimum, maximum, exclusiveMinimum, exclusiveMaximum and multipleOf.
  • Generated text must satisfy minLength, maxLength and pattern. Output that violates them is a runtime error (FillError), never a silent truncation or repair. The generator uses constrained decoding (a length budget plus a regex/grammar automaton for pattern; EOS masked until minLength), so a valid output exists by construction and this error should not occur in practice; it is a guard, not a code path.
• Arrays: the length is bounded by minItems and maxItems. When maxItems is absent, the cap is options.max_items (default null = no cap: an anchored array has as many items as the context has, §12.6; an index-addressed array stops at 32, the count head's trained range). uniqueItems is checked when the response is validated.
• Objects: additionalProperties may be absent, true or false. deg never emits undeclared keys. A schema-valued additionalProperties is an error, because the key names would be unknown.

5. Validation errors (HTTP 422, every problem listed as {loc, msg})

• x-generate on anything other than string or array of string. This includes enum, boolean, integer, number, object, arrays of non-strings, and strings that have enum or format.
• Unsupported keywords: allOf, not, if/then/else, const (a single value has nothing to decide), dependentSchemas, dependentRequired, dependencies, patternProperties, propertyNames, unevaluatedProperties, minProperties, maxProperties, unevaluatedItems, prefixItems, additionalItems, tuple items: [...], contains, minContains, maxContains, $dynamicRef, $dynamicAnchor, $recursiveRef, $recursiveAnchor, contentEncoding, contentMediaType, contentSchema.
• Limited keywords:
  • anyOf/oneOf are allowed only in the nullable form or the const/enum form.
  • $ref must be local (# or #/...) and not recursive. Its siblings may only be annotations, which override the target's.
  • Union types are allowed only as [X, "null"].
  • format must be one of date, date-time, email, uri or uuid.
• Also errors:
  • unknown keywords, and keywords that don't apply to the type (e.g. maxLength on integer);
  • an enum with a duplicate value, a value of the wrong type, or fewer than 2 or more than 255 values;
  • integer bounds that admit 0 or 1 values;
  • number bounds that admit no value (e.g. minimum > maximum);
  • an invalid pattern;
  • a required name that isn't in properties;
  • a root that isn't type: object;
  • any document that isn't a valid JSON Schema (draft 2020-12 by default, or the draft named in $schema).
• Allowed annotations: title, description, default, examples, $comment, $schema, $id, $anchor, $defs, definitions, readOnly, writeOnly, deprecated. A leaf's title and description are its instructions to the model.

6. Fill plan

compile_schema(schema) -> FillPlan is a pure function. The plan holds:

• fields: the leaves in schema order (depth-first). Each leaf has:
  • its path: property names, with * meaning each item;
  • its mode and JSON type;
  • options/option_descriptions (choice) or levels (score);
  • null_outcome, format, constraints;
  • required, nullable, on_null, title, description.
• arrays: the array nodes with min_items, max_items and unique_items.
• FieldPlan.fingerprint: a sha256 of the leaf's canonical spec. It is identical wherever that leaf appears, whatever its siblings or their order.
• FillPlan.fingerprint: a hash of the set of parallel-phase fingerprints plus the ordered list of generate fingerprints.
  • Reordering decide/extract properties keeps the hash.
  • Reordering generate fields changes it.

7. Execution order

1. Read once. The context is rendered once and encoded once:
   • strings are used verbatim;
   • JSON is serialized with ", "/": " separators, the given key order and ensure_ascii=false.
2. Parallel phase: all decide.* and extract.* fields. Each field's outcome is a function of exactly the context encoding, the field's key path, the field's own annotations, and its options/levels/constraints. It is never a function of other fields or their order.
   • So outcomes are bitwise-reproducible when fields are reordered, added or removed, or when the schema is split into several requests or merged into a bigger one.
   • Array items are first enumerated as item anchors (the count). Each item's leaves are then conditioned on the context plus that item's anchor.
3. Generate phase. The generate fields run after the parallel phase. Each one is conditioned on the context and all decided/extracted values (one shared KNOWN VALUES block), never on other generate fields: generate fields are independent of each other and decode together.
   • Decoding is greedy by default.
   • options.temperature > 0 requires options.seed; each field samples from its own stream seeded by (seed, pointer).
   • Decoding is bounded by maxLength, pattern and the field's x-max-tokens, else options.max_new_tokens (default 256 per field).
   • Not implemented: generate-to-generate dependencies. If added, they must be declared (e.g. "x-generate": {"after": "/summary"}) and the dependent field must continue from the parent row's final state after its generated tokens (forked), never re-prefill the parent's text; unannotated schemas keep the independence above.

8. Response

{
  "model": "deg-qwen3.5-0.8b-0.1",
  "value": {"department": "billing", "urgent": true, "reply": "..."},
  "meta": {
    "/department": {"mode": "decide.choice", "probability": 0.8, "confidence": 0.7,
                     "probabilities": [0.8, 0.15, 0.05]},
    "/urgent":     {"mode": "decide.noul", "probability": 0.91},
    "/anger":      {"mode": "decide.score", "probability": 0.6, "confidence": 0.5,
                     "probabilities": [0, 0.1, 0.6, 0.3, 0], "expected": 2.2},
    "/order_id":   {"mode": "extract.span", "confidence": 0.88, "span": [32, 39]},
    "/due":        {"mode": "extract.date", "confidence": 0.4, "span": null,
                     "null_reason": "not_found"},
    "/line_items": {"mode": "array", "items": 2},
    "/reply":      {"mode": "generate", "generated_tokens": 11}
  },
  "usage": {"prompt_tokens": 42, "generated_tokens": 11},
  "plan": "<FillPlan.fingerprint>"
}

• value is always valid against the input schema. assemble checks this with jsonschema and deg's own format checks, and raises FillError (HTTP 500) otherwise.
• meta is keyed by concrete RFC 6901 pointer, with integer array indices. An array that dropped items whose required values were not in the context (§12.6) has "dropped": n beside "items".
• probabilities follow the schema's option or level order; for score, the null outcome comes last.
• span gives [start, end) code-point offsets into the rendered context. assumed_offset: true marks a date-time whose offset came from options.default_offset.
• confidence = (count * peak - 1) / (count - 1) is the jev v1 formula. Ties go to the first maximum.
• usage.generated_tokens is the sum of the per-field generated_tokens. It is 0 for a schema with no x-generate.

9. API

POST /v1/fill

{"context": "string | object | array",
 "schema": {"type": "object", "...": "..."},
 "model": "deg-latest",
 "options": {"max_new_tokens": 256, "max_items": null, "date_order": null,
             "default_offset": "Z", "temperature": 0, "seed": null,
             "array_decode": "count_peaks", "null_margin": null}}

• array_decode ("threshold" | "count_peaks" | "hybrid") picks how anchored arrays find their items (§12.6); also accepted in the decision APIs' options.

• null_margin (finite number, nats, or null) biases nullable/optional extract leaves toward null: such a leaf is null when its best valid span's log-prob is below the null log-prob + null_margin (§12.1). null = the server's --null-margin (default 0 = plain argmax). > 0 trades made-up answers for omissions (deg-2b held-out choice: 2.6, deg-0.8b: 1.4; deg.bench.null_margin, results/deg/null-margin-v1). It touches no decide field, no forbid (required non-nullable) leaf and no count/anchor decision; it applies to leaves inside index-addressed and anchored items the same way (a nullable item leaf becomes null, an optional one is omitted). Also accepted in the decision APIs' options.

• date_order is "MDY", "DMY" or null. With null, a numeric date like 03/05/2026 parses only if exactly one reading is a valid date; otherwise the field is null or omitted.

• Unknown request fields and unknown options are 422 errors.

GET /v1/models lists the accepted model names; GET /health reports the batcher.

9.1 Decision APIs (src/deg/api.py, server.py)

python -m deg.server --ckpt DIR --port 8790 serves three routes on one model. Each request becomes one fill: every question is a property of one schema (key = question name / id), so the input is read once, decisions/extractions are independent fields, and generate questions run last, conditioned on the other answers.

Question (OpenAI / Jev)	Field schema	Mode
predicate / noul	boolean; Jev criteria.true/false appended to the description as Yes means: / No means: lines (the jev training format)	decide.noul
choice	oneOf of const + description (or enum when no option has a description); OpenAI values are typed ("true" != true)	decide.choice
score	integer, minimum 0, maximum L-1, description = the instructions; level labels (and descriptions) as x-levels: {"i": "label[: description]"}, so each is the option line i: label the score head was trained on	decide.score
extract (deg)	[string|number|integer, null] + format from value_type, pattern, minLength, maxLength	extract.* (nullable: a miss is null)
generate (deg)	string, x-generate: true, minLength, maxLength, pattern; max_tokens -> that field's x-max-tokens (questions in one request may use different budgets)	generate

A one-option choice or one-level score (accepted by live Jev) is answered with probability 1 without the model.

• POST /v1/decisions mirrors the OpenAI Decisions API (openai-python 3.26.1 types/decision.py, types/decision_create_params.py; guide https://developers.openai.com/api/docs/guides/decisions): request model, input (string or user messages of input_text parts; input_image is refused with 400 because this model reads text only), questions, safety_identifier; response {answers, model, usage} with answers in question order: predicate {type, name, probability}, choice {type, name, choice, probabilities: [{value, probability}], confidence}, score {type, name, score, probabilities: [{value, label, probability}], confidence} (score = the expected level index), usage {input_tokens, input_tokens_details{cache_write_tokens, cached_tokens}, output_tokens, output_tokens_details{reasoning_tokens}, total_tokens}. name is optional (echoed, null when absent) and must be unique. Errors: 400 {"error": {message, type: "invalid_request_error", param, code}}. deg never returns a refusal answer.
• POST /v1/systemone mirrors live TypeSafe Jev (jev-1.13.0, probed 2026-10-09; tests/fixtures/jev_systemone_live.json): answers keyed by id in request order, noul {type, noul}, choice {type, choice, confidence, probabilities{option: p}}, score {type, score, confidence, legend{"i": level}, probabilities{"i": p}}, usage {input_tokens, output_tokens}. Limits and errors as live Jev: 1..255 options ({"detail": "Too many choices. Must have at most 255 choices."}), 1..10 levels, unknown type / unknown top-level field -> 400 {"detail": {"error_type": "api_usage_error", "message": "Invalid request."}}, unknown model -> 400 Unknown model: X, missing field / empty questions -> 422 {"detail": [{type, loc, msg, input}]}, missing key -> 403. Unknown question keys are ignored, as live Jev does. Unlike live Jev, probabilities are not rounded to 2 decimals.
• Extensions on both: extract -> {type, value, span, confidence[, null_reason]}, generate -> {type, text, output_tokens}; request options (temperature, seed, date_order, default_offset). Usage output tokens are the generated tokens (0 without generate).
• Model names: deg aliases, plus gpt-6-luna (decisions) and jev-latest, jev-preview, clef-flash (systemone) so existing clients work unchanged; responses carry deg's model id.
• Batching: one GPU worker; requests arriving within the batch window (default 5 ms) are taken together (default 16) and handed to deg.infer.fill_many(model, jobs) when the runtime defines it, else run back to back. A failing job fails only its own request; more than --max-pending queued jobs -> 429.

10. Determinism guarantees

• Plan: the same schema always gives the same plan and fingerprints. Compilation never mutates the input.

• Parallel phase: greedy and sampled requests behave identically here, since decide and extract involve no sampling. On the same build and hardware, outcomes are bitwise identical across:

  • repeated runs;

  • field reordering;

  • adding or removing other fields;

  • splitting or merging schemas.

  • batching with other requests (fill_many).

  The runtime batches per-field queries with batch-invariant kernels so that a field's numerics don't depend on the batch composition (§12.2).

• Generate phase: given the same parallel outcomes, a generate field's text is bitwise reproducible (greedy, or sampled with seed) and independent of the other generate fields in the schema and of other requests decoding with it.

• Post-processing (argmax, tie-breaks, parsing, assembly) is pure Python over the head outputs, with no clock, locale or randomness.

11. What is measured against the constrained-decoding baseline

The baseline is the same base model (Qwen3.5-0.8B, at the same quantization) emitting the whole JSON object under grammar-constrained decoding (a JSON-Schema-derived grammar), greedy, from the same prompt.

The same eval set of (context, schema, gold JSON) is measured on both:

• Tokens: generated tokens per response, prompt tokens, and their total.
• Latency: end-to-end on one pinned CPU core (the deploy target), plus GPU throughput.
• Accuracy per mode:
  • choice: exact match;
  • score: MAE;
  • noul: accuracy;
  • span: exact match and token F1;
  • number and date: exact match after parsing;
  • null: precision and recall.
• Calibration: the ECE of decide probabilities and of extract confidences.
• Validity: the rate of schema-valid responses. deg must hit 100%, by construction.
• Determinism: bitwise equality across 10 repeats, 5 field permutations, and split-vs-merged schemas.
• jev gates: the jev v1 eval gates on /v1/systemone.

12. Model (v1 implementation)

Code: src/deg/model.py (trunk, forks, heads), infer.py (fill), train.py, evaluate.py. Tests: tests/test_model.py.

12.1 Encoding: one trunk, one fork per question

• Trunk = a fixed system/user prefix, the rendered context, "\n". It contains the context only, never the schema, so it is shared by every field and every request on the same context.
• Fork = one continuation of a copy of the trunk state. Qwen3.5 is a hybrid: Gated DeltaNet layers carry a recurrent state plus a 3-token causal-conv tail, attention layers carry K/V. A fork starts from exactly that state, so it reads the whole context and nothing else. Fork kinds:
  • decide/extract leaf: FIELD <concrete pointer>, ITEM i of <array> per enclosing array (index mode; anchored items use ITEM <array> AT: "...", §12.6), TITLE, DESCRIPTION, KIND, extract CONSTRAINTS, then OPTIONS: with one - <option> line per outcome, then ANSWER:. Outcome text follows the plan order (choice: value + option description; score: level, plus a null line for nullable scales; noul: no option lines).
  • array count: ARRAY <pointer>, item anchors of enclosing arrays, the array's title/description and item-field pointers, ANSWER:.
  • generate: KNOWN VALUES: (every decided/extracted value, one pointer = json line each, in leaf order; generate values are never included), WRITE <pointer>, annotations, LIMITS, VALUE:, then the text and <|im_end|>. At inference the KNOWN VALUES block is prefilled ONCE on the trunk and every generate field forks that state with only its own header (no per-field restatement of the values).
• Why forks and not an attention mask: DeltaNet layers are recurrent, so a mask cannot hide one field's tokens from the next; separate continuations can. The backbone is a functional re-implementation of the HF Qwen3.5 stack over explicit state (Backbone.forward(ids, lengths, state)); it matches the HF forward (fp32 relative error < 1e-3 on GPU, assert_close on CPU) and lets gradients flow from forks into the trunk.
• Padding is exact: padded positions get beta = 0 and g = 0 in the delta rule (no write, no decay), their K/V are masked, and the conv tail is gathered at each row's real length, so a right-padded row gives the same state as an unpadded one.
• Long contexts stream; there is no context cap. Inputs up to DENSE_PREFILL_MAX (8192) tokens are read in one shot (dense causal mask). Longer inputs prefill in PREFILL_CHUNK (4096) token chunks, carrying each DeltaNet layer's recurrent state and conv tail and appending each attention layer's K/V; on CUDA the chunk attends the K/V buffer through the flash-style fork kernel (no mask tensor at all), elsewhere through SDPA with a [B, chunk, P+chunk] mask. Nothing is O(L^2): peak memory is the K/V (6 attention layers x 2 KV heads x 256 x bf16 = 12 KiB/token) + hidden states (2 KiB/token) + one chunk's work (~0.46 GiB at 4096, independent of L). Span-pointer keys are projected in 8192-row blocks (bitwise equal to one bi_linear), and span candidates come from an [T, MAX_SPAN_TOKENS] band, never a T x T grid. Chunked outputs equal the one-shot read within bf16 noise; at most 8192 tokens they are bitwise the one-shot path. A context whose K/V does not fit the device fails with a CUDA OOM (reported, never truncated); serving those needs K/V offload to host memory.
• Nothing scales with fields x context on the device. Fork batches are sized from the memory free right now (fork_token_budget: FORK_MEM_FRACTION of free + cached-unused memory at FORK_BYTES_PER_TOKEN ~128 KiB per padded fork token, capped at FORK_TOKENS; OOM split stays as a backstop); after a fork pass only the rows the heads read (answer + option tokens) are kept. Span logits for a long context or a big schema are computed in answer-row micro-batches of at most HEAD_LOGIT_BYTES and context-row blocks of HEAD_ROWS (keys re-projected per micro-batch, no [T, 256] copy), then wait in host memory until decoding (bitwise the one-shot values). After field decoding the raw head outputs and the trunk hidden states are released, and the KNOWN VALUES block extends the context state in place (forward(consume=True)), so a context's K/V is never held twice. Peak device memory = weights + context K/V + hidden + one fork batch.

12.2 Independence (bitwise) and the one-pass runtime

Inference runs every query of a request (array counts, array items speculated up to 8 per array, every decide/extract field) as fork rows of ONE batch (DegModel.answer_many; chunked only above 16k padded tokens, for memory), and every x-generate field of a request as rows of one decode batch. fill_many puts the rows of several requests in the same batches. A row's numerics never depend on the other rows because every op in the fork path is batch-invariant (src/deg/kernels.py):

• Prefixes by reference. A fork row reads its prefix (context trunk, or trunk + KNOWN VALUES) in place: attention reads the prefix K/V at per-row offsets, the recurrent kernel starts from the prefix DeltaNet state by index. No per-row copy of the context exists; per-row state is the fixed-size recurrent state, conv tail and the row's own K/V.
• GEMMs: cuBLAS picks kernels (tiles, split-K) by shape, so a row's result changes with the number of rows (measured on the L4: rows differ for many M). bi_linear is a Triton GEMM with one fixed tile config and no split-K; bi_linear_small (decode steps and the LM head) uses 16-row tiles and a split-K factor fixed by the weight shape, with partials summed in a fixed order. Each call site always uses the same config, so a row's output depends only on that row. Projections sharing an input are fused into one GEMM (weights concatenated once; parameters become views).
• Attention (fork_attention): one program per (row, head, query block), prefix keys then own causal keys in a fixed block order; fully masked blocks are exact no-ops of the online softmax.
• DeltaNet (fork_delta_rule): FLA's fused recurrent gated delta rule, one program per (row, head, value block), tokens in order; padded positions have beta = g = 0.
• Norms: fixed-config Triton row kernels; elementwise ops run eagerly. torch.compile is not used in the fork path: it specializes size-1 dims, so one row alone and the same row in a batch would run different generated kernels.
• Heads run batched in fp32 through bi_linear (IEEE fp32 dot).

tests/test_model.py checks with torch.equal that every field's raw head outputs are identical alone, reordered, inside bigger schemas and in split vs merged schemas; that a generate field's text is identical alone and next to another generate field; and that fill_many equals fill per request, on CPU (tiny random Qwen3.5; the CPU reference runs each fork row alone at a 64-token width grid because torch's CPU SIMD/scalar-tail split depends on tensor size) and on GPU (the real model). Caveat: the trunk prefill (batch 1) still uses the FLA chunk kernel and torch.compile'd ops with autotuning, so equality is guaranteed within one process; across processes it holds when the autotune cache matches (same build, hardware and cache).

Decode step. Each generate decode step (all rows x 1 token, LM head, argmax) is captured once as a CUDA graph over static buffers (DecodeGraph: prefixes up to a power-of-2 token capacity, rows bucketed 1/2/4/..., own-token capacity) and replayed. Greedy takes the argmax when it is admissible and only otherwise walks the top 256 in logit order (the same rule as the constraint walk). Measured on one L4 (bf16): 7.3-7.9 ms per step at batch 1 vs 36 ms before; the floor is weight bandwidth (1.5 GB/step incl. the LM head).

12.3 Heads (fp32)

a = hidden state at ANSWER:, o_i = hidden state at the end of option line i, c_t = trunk hidden state at context token t.

• choice: q(a) · k(o_i) / sqrt(256) + u · o_i, softmax over options (the jev v1 pointer head). The option is encoded as text in the fork, so any enum works zero-shot. Training shuffles option order (target permuted) with p = 0.5.
• score: a separate pointer head of the same form over the level lines.
• noul: w · a + b → logits [l, 0], i.e. [p_true, p_false].
• extract: start and end pointers qs(a) · ks(c_t), qe(a) · ke(c_t) over the context tokens only (prefix, suffix and query tokens are not addressable), plus a null logit from a as one extra outcome. Decoding:
  • numbers, dates, and email/uri/uuid strings: every candidate span the deterministic grammar finds in the context (the same candidate regexes datagen used to locate gold spans) is scored by log p_start + log p_end of its covering tokens; the best one that parses and meets the constraints wins, unless null scores higher.
  • plain strings: the top 32 token spans (i <= j < i + 128), mapped to code points and trimmed of whitespace and boundary punctuation a token carried in (trim_span; 98.7% of gold spans are recovered exactly by trimming the covering tokens).
  • options.null_margin m (default 0) shifts that comparison: null wins when log p(best valid span) < log p(null) + m. Decoding stays deterministic and batch-invariant (the margin is per request, applied after the shared head pass).
  • forbid fields ignore null and take the best valid span; none at all is a FillError (inside an array item: the item is dropped, §12.6).
  • confidence = exp(log p_start + log p_end) of the chosen outcome.
• count: linear over 0..64 items, masked to [minItems, min(maxItems or options.max_items or 32, 64)]. Index-addressed arrays only (§12.6). Arrays resolve level by level (a nested array's count fork needs its parent's item anchor).
• anchor: per context token t, an independent logit qa(a) · ka(c_t) / sqrt(256) + ua · c_t + b that an item of the array starts at t (sigmoid; b starts at -6), masked to the array's region (§12.6).
• anchored extract: inside an anchored item, the start/end logits of context token t get a learned bias B[start|end, inside, bucket(t - anchor)], where inside means anchor <= t < region end and the buckets edge at -256, -64, -16, -4, 0, 1, 2, 4, ..., 256 tokens (zero-initialized).
• generate: the base LM head (tied embeddings, frozen in training), greedy by default, seeded multinomial (one stream per field) when temperature > 0. Constrained decoding: candidates are tried in logit (or sampled) order among the top 256; a token is admissible if the decoded text stays within maxLength and is a partial match of (?s:.*?)(?:pattern)(?s:.*) (regex partial matching, i.e. JSON Schema search semantics); EOS is admissible only when minLength and the pattern are met. generated_tokens counts EOS.
• Calibration: one temperature per head kind (choice, score, noul, extract, count), fit by NLL on datagen val only (deg.evaluate calibrate --source datagen-val, default; jev dev is reported separately with --report-jev, or pooled with --source datagen-val+jev-dev), stored in heads.safetensors; count is not fit. Decide heads fit the one-hot filled value (--target hard, default), the label accuracy and ECE are judged against; the soft training targets (--target soft) are less confident than the gold and over-soften (results/deg/bench-v2/calibration-v2.md).

12.4 Training (deg.train)

• Data: datagen records (split == "train"; val/test, including the held-out test domains, are never trained on) plus jev v1 train.jsonl decisions (noul/choice/score with teacher soft labels), mapped to one-field forks (FIELD /answer, the instructions and criteria as text, score levels as i: criterion options). A micro-batch draws records at random (jev with probability --jev-frac, default 0.3) up to a token budget. --reload-every N re-reads the growing datagen files.
• Sources: --records PATH[:WEIGHT] is repeatable (default: the v2 full-off file, weight 1). Each source reads its own split == "train" records (its own val/test labels are kept out) into its own pool, and every reload re-reads all sources. Each deg micro-batch picks one source with probability proportional to its weight among the active sources, those with at least --source-min-train (default 50, summed over ranks) train records, so a small, slowly growing source (e.g. /data/deg/datagen/v3-long/records.jsonl:0.25 next to v2 at 1.0 → 20%) gets its fixed share as soon as it reaches the threshold. The jev mix stays per record inside every micro-batch. End-of-run calibration fits on the val records of every source (contexts within --max-context-tokens); the end-of-run eval uses the first source.
• Long contexts: a record whose context exceeds --max-context-tokens (default 6000) is kept whole and trains a span-aware window at every draw (inference is unchanged: it streams the whole context). The window is max-context-tokens minus the template tokens, placed uniformly among the positions that contain one gold span picked at random (spans and alternatives), so windows differ across epochs and cover the gold. A cropped window trains only what its text shows: extract fields with a gold span (or alternative) inside the window, and anchored arrays with at least one gold item start inside it (the items starting outside are left out with their subtrees). Absent-gold (null) extract fields, decide and generate fields (their evidence has no span) and count heads train only on a whole context; nothing is ever trained as null or empty because of cropping. Train logs report windows, window_targets, window_dropped.
• Decide cap per field (--decide-forks-per-field K, default 0 = off; cap_decide_forks, applied first in every draw): at most K decide forks (choice/score/noul) per schema field (array indices folded to *), a fresh random subset each draw. v3-long filings carry ~135 choice forks per record, 99.8% one repeated in-array enum per table cell (unit, scale, statement_type; 68% of their booleans are false); uncapped they are most of the decide forks of every step that draws one. run-2b, which trained them uncapped, lost decide accuracy on short inputs (datagen val top-level choice .939 -> .873 at step 4500; JevBench choice 94 -> 74-92 of 139 across steps 500-6000, noisy; results/deg/regress-2b).
• Fork cap per draw (cap_forks): every drawn example (whole context, window, filler) trains a fresh random subset of its forks when they exceed its room: --record-fork-tokens when set, else the token budget beside its trunk and at least half the budget however long the context. Whole-array forks (anchor, and anchored arrays' count) are always kept; the other forks are taken in random order while they fit, the first one whatever its length (so a generate fork longer than the room still trains). A record therefore costs one trunk pass plus about the room in fork tokens per step, whatever its field count: unsubsampled, a long filing (1,153 field rows, 176 generate forks each repeating its ~16k-token known-value block) was 2.9M fork tokens and 260 micro-batches, a 20+ min step. Loss terms stay means over the forks drawn (a subsample; forks longer than the room are drawn less often). Train logs report subsampled_records, subsampled_forks.
• Training trunks have no 8192 cap: a trunk longer than DENSE_PREFILL_MAX is read in PREFILL_CHUNK chunks with every layer of every chunk gradient-checkpointed (Backbone._forward_chunked_ckpt; the mask is built inside the checkpoint, earlier chunks' K/V are passed in as they are), so nothing O(L^2) is saved and one layer's recompute is O(chunk x L). It equals the dense checkpointed read in hidden states, state, fork outputs and gradients (tests/test_train_long.py). The memory planner models the chunked trunk separately (per-chunk recurrent state, the concatenated K/V copy, no saved mask, chunk-sized recompute).
• Row chunks share one trunk pass: a record too big for one micro-batch is split by the memory planner into field-row chunks (MemPlanner.split_rows); run_step runs them on ONE trunk forward (run_chunks): the trunk outputs (final hidden, recurrent state, conv tails, K/V) are detached into leaves, each chunk's forks + heads backpropagate into the leaves, and the trunk backpropagates once from their accumulated gradient (equal to the unsplit record's gradient, dense and chunked trunks alike: tests/test_train_mem.py). An out-of-memory error in such a run drops its gradients and reruns the chunks one by one with their own trunk passes (--no-share-trunk always does). Logs: trunk_shared.
• Losses, each a mean over its items:
  • decide: KL(t ‖ q). t is the record's decide_probs (Clef probabilities where Clef's argmax matched the filled value), else one-hot of the filled value; jev uses probs, else hard.
  • extract: start and end CE where every span_alternatives entry is a positive: with span_probs the target is that soft distribution; otherwise the loss is the marginal -log Σ_pos p. A field absent from the filled value targets null.
  • count: CE of the array length (index-addressed arrays).
  • anchor: per anchored array instance, BCE of every in-region token against the gold item starts, summed and divided by 1 + the number of items (§12.6).
  • lm: next-token CE on x-generate text + EOS, teacher-forced on gold values; the prompt is the inference layout (KNOWN VALUES = decided/extracted values only).
  • anchor_count (--anchor-count-weight W, default 0 = off): with W > 0 every anchored array of a whole context (never a cropped window: a count needs the whole array) also gets the index-mode count fork, CE of its gold item count over [minItems, min(maxItems | MAX_COUNT, MAX_COUNT)] with the target capped at the top class (which then means "at least that many"), so count_peaks decoding has a trained count.
  • total = decide + extract + 0.5 count + anchor + W anchor_count + L lm (--anchor-weight, --anchor-count-weight, --lm-weight L, default 1).
• Filler augmentation (--filler-frac P, --filler-corpus, default /data/deg/external/v3/docs.jsonl, --filler-max-tokens N, default 16384): the anchor head under-fires on long documents when it has only seen short ones. With probability P a drawn short datagen record is rebuilt with its document at a random offset inside unrelated text (pre + "\n\n" + doc + "\n\n" + post), total length log-uniform from its own length + 64 to min(N, --max-context-tokens); every gold span shifts with it and every target is the plain example's (the filler is unrelated text). A --filler-negative-frac share (default 0.25) is filler only: the record's top-level anchored arrays with minItems 0 train their anchor fork with no item start (every token negative) and count 0; nothing else trains on it. Filler comes only from v3 train-split documents (datagen.v3.split_of), never the record's own document and never ExtractBench (refused by path). Draws are placeholders sorted at their planned length and materialized when the packing reaches them, like long-document windows, so the packing and the memory planner see real examples; a rebuilt example that would exceed --max-context-tokens falls back to the plain one. Logs: filler, filler_negative, filler_tokens.
• Optimization: bf16 weights with fp32 master copies (AdamW, lr 2e-5 trunk / 1e-3 heads, warmup then cosine to 10%), embeddings and the tied LM head frozen, per-layer gradient checkpointing, elementwise ops fused with torch.compile. Forks are length-grouped (≤25% padding). Multi-GPU: torchrun, one data shard per rank, explicit gradient all-reduce. Logs: losses, tokens/s (real and padded), GPU utilization and memory.

12.5 Smoke run and full-run launch

Smoke (2026-10-08, one L4, /data/deg/runs/smoke-1, 300 steps, budget 14k tokens, ≤32 records/step, jev 20k-record subset at 30%): total loss 13.0 → ~1.5 (decide 1.54 → 0.15, extract 8.2 → ~0.5, lm 1.83 → ~0.5, count 2.7 → ~0.8); ~4.0k real tokens/s (4.7k padded), 85-92% GPU utilization, 14.9 GiB peak. Eval through the full fill path after calibration:

split	records	valid	decide acc / ECE	extract EM	count acc	generated / full-JSON tokens
val	29	100%	0.93 / 0.097	0.79	0.73	0.33
test (held-out domains)	60	100%	0.90 / 0.071	0.85	0.70	0.29
jev dev (1312 questions)			0.81 / 0.024

Full run (4 GPUs; not launched by default — the orchestrator allocates GPUs):

systemd-run --user --unit deg-train-full --property=MemoryMax=80G \
  --setenv=CUDA_VISIBLE_DEVICES=0,1,2,3 --setenv=PYTHONUNBUFFERED=1 \
  --setenv=PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --working-directory=/shared/rwkv/packages/deg \
  /bin/bash -c 'exec /shared/rwkv/.venv/bin/torchrun --standalone --nproc-per-node 4 \
    -m deg.train --out /data/deg/runs/full-1 --steps 20000 --budget 14000 \
    --max-records 32 --jev-frac 0.3 --reload-every 500 --save-every 1000 \
    --eval-records 200 --eval-jev 1000 > /data/deg/runs/full-1.log 2>&1'

12.6 Position-anchored array items (array_mode = "anchor", run-2 default)

Run-1 addressed array item i by index ("ITEM i of /arr") and sized arrays with a count head over 0..64. On real documents that failed: array-item leaves were 10% correct vs 45% for scalars, forcing the gold counts did not help, and nested counts copied the parent's index (results/deg/extractbench-v1/DIAGNOSIS.md, cause d). Anchor mode addresses an item by where it is, not by its ordinal:

• Which arrays. An array whose item subtree has an extract leaf is anchored. Arrays of only decide/generate leaves (no span can say where an item is) keep the count head and index addressing. The mode is static per schema and checkpoint (model.meta["array_mode"]; a checkpoint without anchor heads, i.e. run-1, loads as index and behaves exactly as before; deg.train --array-mode anchor|index, default anchor).
• Routing by document length (model.meta["anchor_min_tokens"], default 0 = off; deg.train --anchor-min-tokens N, saved in deg.json). An anchor-mode model fills a document of fewer than N context tokens exactly as index mode (count fork + ITEM i forks), and trains its whole-context records the same way; documents of N or more tokens, filler-augmented records that reach N, and every cropped window (its document is longer than any window) anchor. Why: on short datagen records run-2b step-4500 in index mode beats its own anchor mode (test array-item EM .844 vs .769 count_peaks, extract EM .896 vs .866; run-1 index .891 / .922), while anchors win on long documents (ExtractBench; results/deg/regress-2b).
• Anchor fork. One fork per array instance: ARRAY <ptr>, the enclosing items' locators, title/description, ITEM FIELDS, KIND: mark where each item starts in the context, ANSWER:. The anchor head scores every context token of the array's region; tokens with p > 0.5 are marked, each run of adjacent marked tokens is one item (its best token). Fewer than minItems: the best remaining non-adjacent in-region tokens are added (none left: FillError); more than maxItems (or options.max_items when given): the highest are kept. No other cap: the count is the number of anchors, and item forks run in the usual memory-bounded fork micro-batches.
• Decode modes (options.array_decode, default count_peaks). threshold: as above. count_peaks: the array also runs its count fork (the index-mode count query, trained on anchored arrays by anchor_count, §12.4); N = its argmax over [minItems, min(maxItems | max_items | MAX_COUNT, MAX_COUNT)] (at the saturated top class, the threshold count when larger), capped by maxItems; the items are the N best local maxima of the anchor logits with non-maximum suppression (PEAK_RADIUS = 4 tokens: one peak per item), then the best remaining tokens outside every radius if too few peaks exist. hybrid: threshold, unless no token passes p > 0.5 and the count says N > 0, then count_peaks. The count fork is one more independent fork and peak picking is a pure function of the logits, so every mode is deterministic and batch-invariant. Default chosen on datagen val only (results/deg/array-decode-v1, run-2b step-3000, 131 records / 153 anchored arrays, paired): aligned item-leaf EM threshold .609, count_peaks .553, hybrid .659 (count exact .575 / .784 / .641); with 4k tokens of filler before the record: .428 / .482 / .572 (count .342 / .704 / .513). PEAK_RADIUS 2/4/8 changes count_peaks/hybrid leaf EM by at most .03. Default re-chosen at run-2b step-4500 (count head trained by anchor_count since step 4000; results/deg/regress-2b): count_peaks vs hybrid vs threshold, datagen val count .846 / .577 / .558, extract EM .828 / .818 / .816; datagen test (200) count .850 / .740 / .690, extract EM .866 / .847 / .842; ExtractBench normalized 239 vs 221 (hybrid), aligned 339 vs 338, count_ok 132 vs 111, valid docs 28 vs 27.
• Regions. A top-level array's region is the whole context. Item k's region runs from its anchor to the next sibling anchor (the last item's to its parent's region end). A nested array searches anchors only inside its parent item's region, so a child count cannot copy the parent's count or index.
• Item forks. Every fork inside an anchored item renders anchored indices as * (FIELD /results/*/time) and one locator line per enclosing anchored item: ITEM /results AT: "<context text from the anchor, at most 24 tokens, cut at the region end>". Extract forks also get the anchored span bias (§12.3) from the innermost anchor and region. There is no ordinal anywhere in an anchored item's forks.
• Items that do not exist. An anchor only says where an item would start; the item exists only if every required non-nullable (on_null: forbid) extract leaf in it has a valid value. When one has none (forbid already ignores null and searches every valid candidate in the context, so this means no span parses/meets the constraints at all — e.g. a format: email string item anchored in a résumé with no address), the innermost anchored item holding that leaf is dropped: never a FillError, never a made-up value. Survivors keep their anchors, regions and fork outputs (no fork sees an anchored item's index, so nothing is re-run and their values are bitwise unchanged) and are renumbered in document order; the array's meta gets "dropped": n; generate fields of a dropped item are not decoded and its values are not in KNOWN VALUES. If the drops leave an array below minItems, its enclosing anchored item cannot be represented and is dropped in turn; a top-level array left below minItems is a FillError (minItems N: only k items have their required values), like an anchor fork with too few positions. There is no "replacement anchor" fallback: for typed leaves (numbers, dates, email/uri/uuid) the valid candidate set is the whole context and does not depend on the anchor, so another anchor cannot succeed where this one failed. Dropping is a pure function of each item's own fork outputs, so it is deterministic and batch-invariant. It does not depend on array_decode (which only picks the anchors). Index-addressed items (index-mode models, documents routed to index mode below anchor_min_tokens, arrays without extract leaves) follow the same rule: the innermost enclosing item — anchored or index-addressed — is dropped and the survivors renumbered with their fork outputs unchanged (their ITEM i of forks are not re-run); generate forks of survivors render the new index. A required non-nullable value outside every array item keeps the §4 FillError.
• Rounds. Anchored arrays add one fork round per nesting level (anchors first, then the items' forks and nested anchor forks); index arrays keep the speculative one-pass layout.
• Order. Items come out in document order (anchor order). deg.evaluate reports positional extract_em and order-insensitive extract_em_aligned; deg.extractbench.score_record(..., align=True) and the ExtractBench eval report normalized_aligned_rate next to the positional rates.
• Independence. An anchor fork depends only on the context and its own query (array pointer, enclosing locators, annotations); an item's forks depend only on the context, their own query, and the item's anchor and region. Both are bitwise identical alone, inside a bigger schema and reordered (tests/test_anchors.py).
• Training gold. From the datagen spans (spans, else the first span_alternatives entry): an item's anchor is the first token of its earliest span that no sibling item shares (a shared span such as a table column header does not locate an item), else of its earliest span. An anchored array instance whose items do not all get distinct anchors inside the region is left out of training (counted as anchor_skipped_arrays in the data log); an empty gold array trains an anchor fork with no positives (the hard negative for nested arrays).

12.7 Who writes x-generate text: hybrid and LoRA (deg-suite v1 follow-up)

deg-suite v1 found the untrained base Qwen3.5-0.8B writes better x-generate text than every fine-tuned checkpoint (judge faithful 0.94 vs 0.27 on DialogSum, 1.59 vs 0.43 on mixed, out of 2; lengths similar): full fine-tuning made generated text unfaithful. Three ways to keep the base model's generation, all deterministic and batch-invariant:

• Two-copy hybrid (load_model(..., generate="base"), --generate base on the server and the suite): a frozen base body sits beside the fine-tuned one (token embeddings = tied LM head shared when equal, +~1 GB bf16). Decide/extract are unchanged; for x-generate the base body re-reads trunk + KNOWN VALUES in one pass (one more context read) and decodes the header of :func:deg.infer.base_generate_header, which closes the user turn with a plain instruction and opens the assistant turn (thinking off).
• LoRA (deg.train --lora-rank R): the backbone stays the base; named low-rank adapters on every attention / linear-attention / MLP projection (lora_A [nr, in], lora_B [out, nr], B = 0 at start) train with the heads. A per-row selector picks each fork row's adapter (--adapters shared or decide,extract: by fork kind; choice, score, noul -> decide; extract, count, anchor -> extract), so rows with different adapters share one batch and stay bitwise independent (the delta goes through the same bi_linear GEMMs). x-generate rows run adapter-off = the base model, never trained. --lora-context base (default): the context is read ONCE by the base weights and every fork (adapted decide/extract rows, adapter-off generate rows) continues it, so the one-pass structure holds and the trunk has no gradient in training. --lora-context <adapter>: that adapter also reads the context; generation re-reads it adapter-off (2x prefill for requests with x-generate fields). Checkpoints keep the base backbone.safetensors and add adapters.safetensors + deg.json["lora"].
• GENERATE LoRA on the hybrid's base copy (--generate base+lora:<dir>; load_model(..., generate="base+lora:<dir>")): the two-copy hybrid, with one low-rank adapter trained by deg.gentrain merged into the frozen base copy at load (W += alpha/r * B @ A, fp32 on the CPU, cast to the body dtype; deterministic). Deg's own backbone, heads and the one-pass flow are unchanged; generate rows stay batch-invariant (the merged copy is an ordinary frozen body); a zero adapter is the plain hybrid. A generator dir holds generator.json (format deg-generator-lora-v1, rank, alpha, training meta) + lora.safetensors (<module>.lora_A [r, in], <module>.lora_B [out, r], named as in the HF Qwen3_5ForCausalLM). Training (python -m deg.gentrain): SFT on x-generate fields in exactly the hybrid's layout (context + KNOWN VALUES + base_generate_header, target = value + EOS; deg_example(..., base_layout=True)), the adapter on every LORA_TARGETS projection reading the whole sequence; data = gen-v1 train (suite guard; open-weight writer + judge only, check_open_teacher; fields the judge kept at faithful 2 / responsive 2; targets over the field's decode budget dropped, never cut); val token NLL selects the checkpoint (early stopping); --kl-base W adds W * KL(base || adapted) on the target tokens.
• Anti-forgetting for full fine-tunes (--lm-kl-base W): KL(frozen base || deg) on the x-generate target tokens, computed by the frozen base on the identical trunk + fork tokens (no grad).

12.8 Pluggable backbones (--backbone, deg.json["backbone"])

The trunk/fork runtime, the heads, training and the memory planner do not depend on the architecture beyond one Family of layer functions (model.py): every supported base is a hybrid of state layers and attention layers. State.lin[i] = (recurrent state, conv tail) per state layer, State.kv[i] = K/V per attention layer; ForkState reads the prefix K/V by reference and copies only the fixed-size per-row state, so prefill once, fork-by-reference, chunked prefill (inference and checkpointed training), the CUDA-graph decode step and the two-copy hybrid are shared code. A family supplies: the trunk/training state layer (rec_layer), attention projections and output (attn_qkv, attn_out), the batch-invariant fork versions of both, per-row state shapes, the final norm, the rotary call and the LoRA targets.

--backbone	layers	per state layer	per attention layer	tokens
qwen3.5-0.8b (default)	18 Gated DeltaNet + 6 attention, hidden 1024	recurrent [16,128,128] fp32 + 3-token conv tail	K/V 2 heads x 256	ChatML, <think></think> gen turn
qwen3.5-2b (--base-model Qwen/Qwen3.5-2B)	same 18 + 6 layout, hidden 2048, MLP 6144	as the 0.8B	as the 0.8B	as the 0.8B
lfm2.5-1.2b (LiquidAI LFM2.5-1.2B-Instruct)	10 gated short-conv + 6 GQA attention (32 q / 8 kv heads x 64), hidden 2048, MLP 8192, tied embeddings, vocab 65,536	none (empty [B,0]) + 2-token tail of B*x	K/V 8 heads x 64	<|startoftext|> BOS + ChatML, pad <|pad|>, gen turn without a thinking block

LFM2 conv layer: B, C, x = in_proj(norm(h)), y = C * conv3(B * x) (depthwise, no activation), h += out_proj(y); attention has per-head q/k RMSNorm and plain 1-D RoPE; norms are w * norm(x). Both families have K/V = 12 KiB/token (6 layers x 2 KiB).

• Determinism (LFM). The fork path uses the same kernels as Qwen3.5: bi_linear / bi_linear_small for every projection (fused q/k/v and w1/w3), fork_attention (head dim 64, GQA group 4), a fixed-config Triton row kernel for the LFM norm (kernels.rms_rows_w), and an eager elementwise short conv accumulated in fp32 in a fixed tap order (per element, so batch-invariant; no conv kernel is needed). No fallback path was required. tests/test_lfm_backbone.py checks (tiny random LFM2 + the real tokenizer, CPU) the functional stack against HF Lfm2ForCausalLM, every fork row against an independent full forward of context + its query and bitwise alone vs batched, keep/decode continuation, chunked prefill = one-shot (inference, padded rows, prefix state, checkpointed training incl. gradients), and runs the test_model / test_hybrid_generate independence and hybrid-generate contracts against the LFM fixtures; on CUDA (real model) bitwise alone / reordered / bigger schema / fill_many, generate rows next to other generate rows, HF match, chunked prefill match.
• Checkpoints record backbone; a checkpoint loads its own base (pre-§12.8 checkpoints are Qwen3.5-0.8B) and refuses another --backbone; --generate base attaches that backbone's frozen base. Token budgets (--max-context-tokens, --anchor-min-tokens, --budget) count the backbone's own tokens.
• Memory. MemCoef.for_backbone gives analytic coefficients per backbone (calibrated online by the measured scale as before). LFM2.5-1.2B full fine-tuning keeps ~15.7 GiB resident (bf16 weights 2.2 GiB + fp32 master/AdamW 12 B and bf16 grads 2 B per trainable parameter, 1.04B trainable with the embeddings frozen) vs ~6.8 GiB for Qwen3.5-0.8B, so it needs a higher --mem-target-gb on a 24 GB L4.
• Qwen3.5 is size-generic. --backbone / --base-model accept qwen3.5-<size> or the HF id (Qwen/Qwen3.5-2B); every Qwen3.5 size is the one QWEN35 family with the same tokenizer, chat layout and gen turn, and the backbone and heads read layer types, hidden/MLP widths and head counts from the snapshot's config. qwen3.5-2b has its own memory coefficients (2x saved hidden per token, ~1.9x wider layer recompute; same recurrent state, K/V and vocabulary). tests/test_qwen2b_backbone.py runs the model, hybrid-generate and long-read contracts on a tiny 2B-shaped Qwen3.5 (hidden 2x, MLP 3x hidden), checks the real 2B config sizes backbone and heads (meta device), that a 2B checkpoint refuses other backbones and --generate base loads the frozen 2B, and on CUDA the real 2B's bitwise independence and HF match.

12.9 Data-parallel training: ZeRO-2 over several GPUs (torchrun, deg.zero)

torchrun --nproc-per-node N -m deg.train ... trains one model on N GPUs. Each rank reads its own uid shard of every source (zlib.crc32(uid) % N), draws its own micro-batches with the per-rank --budget and memory plan, and the step's gradient is the gradient of the global step loss: each loss term's item counts are all-reduced before the forward (global_counts), each rank's term is its share of the global mean, and the ranks' gradients are summed (not averaged; a rank with more decide items carries more of the decide gradient, exactly as one batch of all ranks' records). One global batch is N per-rank micro-batch sets (N x --budget tokens). Gradient clipping uses the global norm. A rank that cannot finish its step (minimal unit OOM) makes all ranks redraw (all_ranks_ok), so every optimizer step is complete on every rank.

ZeRO-2 (deg.zero.ZeroMaster, default with N > 1; --no-zero = whole masters per rank). bf16 compute weights stay whole on every rank (the functional backbone calls its layers' weights directly). The fp32 master weights, both AdamW moments and the fp32 gradient are sharded: the trainable parameters form one flat vector cut into 32M-element buckets; rank r owns the r-th of N pieces of each bucket. A step's bf16 gradients stay on the parameters until the step's last part (several-part steps sum them in a pinned host fp32 vector, so the device never holds two gradient copies), then a bucketed fp32 reduce-scatter (SUM) gives each rank its shard's gradient; AdamW steps the shard (one flat parameter: element-wise the same arithmetic and decay as the unsharded trainer); each rank rounds its shard to bf16 and an all-gather refreshes every rank's compute weights (bitwise p.copy_(master)). Per trainable parameter a rank keeps 2 B (bf16 grad) + 16/N B instead of 18 B: Qwen3.5-2B (1.37B trainable, embeddings frozen) on 3 L4s holds ~3.8 GiB weights + 2.6 GiB grads + 6.8 GiB shard state = ~13 GiB static vs ~26 GiB unsharded. The memory planner uses this static (sharded_static_bytes), then the measured resident memory as before.

• Start/resume. Heads (fresh, per-rank seeds) and backbone are broadcast from rank 0 at start. Every rank keeps the whole bf16 weights, so rank 0 saves an ordinary deg checkpoint (step-N/, same format as one GPU; no sharded files, no conversion). --init <run>/step-N --start-step N resumes: weights from the checkpoint, the cosine schedule continues at step N, optimizer moments start fresh with a --warmup re-warm (as on one GPU), the data order is re-drawn (the rng mixes in N). The process group timeout is 4 h (rank 0 alone calibrates, evaluates and saves at the end).
• Smoke on one GPU. --zero-sim-world N shards as rank 0 of N ranks with no collectives: the real per-rank memory and compute time on one GPU (other shards are not updated; never a real run).
• tests/test_zero.py (CPU, gloo, 3 processes): sharded masters + AdamW + global clip equal one process stepping whole fp32 masters on the summed gradient, for one-part (device bf16 grads) and two-part (host fp32 sum) steps, replicas bitwise equal; the flat layout covers every element once; global counts make two ranks' gradients and losses sum to the one-batch ones.