| # deg |
|
|
| deg reads a context, takes a JSON Schema, and returns a JSON object that is valid |
| against that schema. Most fields are **decided** by classification heads or **extracted** |
| as spans copied from the context. New tokens are generated only for fields marked |
| `x-generate: true`. The result costs far fewer generated tokens than writing the whole |
| JSON out, and everything outside the generated fields is deterministic. |
|
|
| Base model: Qwen3.5-0.8B. The full contract is in [SPEC.md](SPEC.md). |
|
|
| ## The one flag |
|
|
| ```json |
| {"type": "object", |
| "properties": { |
| "department": {"enum": ["billing", "technical", "sales"]}, |
| "urgent": {"type": "boolean"}, |
| "order_id": {"type": "string"}, |
| "reply": {"type": "string", "x-generate": true, "maxLength": 400}}, |
| "required": ["department", "urgent", "order_id", "reply"]} |
| ``` |
|
|
| - `department` is decided (choice head + probabilities). |
| - `urgent` is decided (noul head). |
| - `order_id` is copied from the context (span pointer). |
| - `reply` is the only field that generates tokens, bounded by `maxLength`. |
|
|
| The mode follows only from the JSON type. There is no router, and `x-generate` is valid |
| only on `string` or on an array of `string`. |
|
|
| ## Status |
|
|
| What is implemented: |
|
|
| - schema validation and compilation into a deterministic fill plan (`deg.schema`); |
| - response assembly and validation (`deg.response`); |
| - deterministic span parsers (`deg.parsers`); |
| - `/v1/fill` request parsing (`deg.request`). |
|
|
| - the model: Qwen3.5-0.8B trunk + decision heads + span pointer + LM head (`deg.model`); |
| - `fill(model, context, schema)` (`deg.infer`), training (`deg.train`) and evaluation |
| (`deg.evaluate`); design in SPEC.md §12. |
|
|
| - head-to-head bench vs structured-output baselines (`deg.bench`; results in |
| `results/deg/bench-v1/summary.md`): |
| `python -m deg.bench.run --system {deg,constrained,constrained-llamacpp,nuextract,clef,frontier} --set datagen-test` |
| (extra `bench`). |
| - JevBench (Benchmark Heaven, MIT; the 231 public typed decisions, scored by its own v1.3.0 |
| rules, `deg.bench.jevbench`; results in `results/deg/jevbench-v1/`): |
| `python -m deg.bench.jevbench --system {deg,systemone} ...` then `--report`. |
| EVAL ONLY: `deg.train` refuses JevBench paths and drops any record or jev example |
| containing a JevBench state. |
| - **deg-suite v1** — the checkpoint/config selection metric (`deg.bench.suite`; results |
| and design in `results/deg/suite-v1/`): one frozen held-out suite whose headline is the |
| mean of DECIDE, EXTRACT and GENERATE, each the mean of its sources (datagen-test, |
| JevBench, Banking77, SQuAD v2, ExtractBench, DialogSum + an open-judge, and a mixed set of |
| real v3 val-split excerpts); invalid output scores 0 on all its fields: |
| `CUDA_VISIBLE_DEVICES=0 python -m deg.bench.suite run --system deg --ckpt <ckpt> --label <name>` |
| (or `results/deg/suite-v1/run_suite.sh <ckpt> <label>`), then `python -m deg.bench.suite report`. |
| EVAL ONLY: `deg.train` refuses suite paths and drops any training record or jev example |
| containing a suite context. |
|
|
| - **decide-v1** datagen (`deg.datagen.decide`): decide-heavy SystemOne-style records that |
| teach adequacy judging, policy compliance, routing, ordinal grading, multi-hop, traps, |
| trade-offs, under-specified cases, adversarial states and date/number thresholds. Every |
| state and question is new, written by an open teacher; the skill list is a taxonomy only and |
| no benchmark item is shown to a teacher. Each record has a primary question, a yes/no |
| question with its exact negation (yes/no labels are 50/50 by construction), and optionally a |
| verbatim reference (extract) and a rationale (x-generate). Targets are balanced: noul |
| polarity, choice gold slot (options permuted) and score level. Labels are kept only where |
| the blind voters (teachers' first-token logprobs over both option orders, plus Clef-flash) |
| agree with the authored gold; `decide_probs` is the voters' mean distribution. Output: |
| `/data/deg/datagen/decide-v1` (test = held-out domains, val 4%): |
| `python -m deg.datagen.decide --out /data/deg/datagen/decide-v1 --spark-role voter |
| --teacher url=http://127.0.0.1:8780/v1,model=Qwen3.8-27B-INT4,concurrency=32`. |
|
|
| - **gen-v1** datagen (`deg.datagen.gen`, corpus `deg.datagen.gen_sources`): generation |
| records on SHORT-to-medium REAL text (<= 10k chars: v3 train-split excerpts, CNN/DM news, |
| OASST1 chats, StackExchange threads, Enron email, Taskmaster-2 / MultiWOZ dialogues, |
| QMSum AMI/ICSI meetings; training splits only, every excerpt checked by the suite guard; |
| licences in the corpus `manifest.json`). 1-3 x-generate fields per request (summary, title, |
| rationale, reply, action items, explanation, answer, note, next step; maxLength and most |
| with x-max-tokens) plus 0-2 decide / 0-3 extract fields. The local 27B writes the schema and |
| fills it (reasoning low); the OTHER model judges every generated field with the suite's |
| open-judge rubric and votes blind on every decide field; only faithful 2 / responsive 2 |
| texts and agreeing decisions are kept. Output `/data/deg/datagen/gen-v1` (split by source |
| document: test 2%, val 2%). |
|
|
| - **extract-real-v1** datagen (`deg.datagen.extract_real`, passages from |
| `deg.datagen.extract_real_sources`): extraction records on REAL, permissively licensed |
| text (v3 train-split EDGAR/arXiv excerpts, Wikipedia, CC-BY news, CUAD contracts, |
| regulations.gov notices, Foodista posts, CORD receipts, arXiv abstracts; license and URL |
| on every record; SQuAD v2 dev articles, suite contexts, JevBench states and ExtractBench |
| text excluded and audited with `--audit`). The 27B writes a request schema per passage |
| (question-style fields, typed dates/amounts/ids, arrays of objects, ~1/3 unanswerable |
| nullable fields); fill A and an independent fill B (same open model, shuffled fields, |
| different instructions) must agree per field after normalization, the span must be |
| verbatim, and agreed nulls are re-checked; disagreeing fields are dropped. Short passages |
| (<= 3,800 tokens, index-mode arrays) and long ones (4,300-5,600 tokens, anchored): |
| `python -m deg.datagen.extract_real_sources --out /data/deg/external/extract-real-v1`, |
| then `python -m deg.datagen.extract_real --out /data/deg/datagen/extract-real-v1 |
| --teacher url=http://127.0.0.1:8780/v1,model=Qwen3.8-27B-INT4,concurrency=24`. |
| |
| - the HTTP server (`deg.server`): OpenAI Decisions API (`/v1/decisions`), TypeSafe Jev |
| SystemOne (`/v1/systemone`) and `/v1/fill`, with request batching and OpenTelemetry; |
| a stdlib Python SDK (`deg.client`). A TypeScript SDK is not written yet (deferred). |
| |
| ```python |
| from deg.model import load_model |
| from deg.infer import fill |
|
|
| model = load_model("/data/deg/runs/smoke-1/final") # or None: base model, untrained heads |
| response = fill(model, "Order AB-1234 was charged twice. Refund today!", schema) |
| ``` |
| |
| Training (one GPU; see `deg.train` for multi-GPU): |
| |
| ```bash |
| CUDA_VISIBLE_DEVICES=3 uv run --package deg --extra model python -m deg.train \ |
| --out /data/deg/runs/smoke-1 --steps 300 |
| ``` |
| |
| ```python |
| from deg import compile_schema |
|
|
| plan = compile_schema(schema) |
| for field in plan.fields: |
| print(field.pointer, field.mode) # /department decide.choice, ... /reply generate |
| ``` |
| |
| ## HTTP API |
| |
| ```bash |
| CUDA_VISIBLE_DEVICES=0 uv run --package deg --extra model --extra server \ |
| python -m deg.server --ckpt /data/deg/runs/run-1/step-0009000 --port 8790 |
| ``` |
| |
| One server, three request shapes; each request is one fill (the input is read once). Set |
| `DEG_API_KEY` to require `Authorization: Bearer <key>`. Shapes and error bodies are in |
| `src/deg/api.py`; golden fixtures in `tests/fixtures/`. |
| |
| | Route | Compatible with | Question types | |
| |---|---|---| |
| | `POST /v1/decisions` | OpenAI Decisions API (`openai` >= 3.26, `client.decisions.create`) | `predicate`, `choice`, `score` + deg `extract`, `generate` | |
| | `POST /v1/systemone` | TypeSafe Jev / SystemOne, Clef | `noul`, `choice`, `score` + deg `extract`, `generate` | |
| | `POST /v1/fill` | deg (SPEC.md §9) | any JSON Schema, `x-generate` | |
| | `GET /v1/models`, `GET /health` | | | |
| |
| `predicate` = `noul` (boolean), `choice` = `enum`, `score` = integer levels `0..L-1`. Clients |
| may keep sending `gpt-6-luna` or `jev-latest` / `clef-flash`; deg answers them and reports |
| its own model id (`--no-compat-models` accepts only deg names). |
| |
| ### OpenAI Decisions API, stock SDK |
| |
| ```python |
| from openai import OpenAI |
| |
| client = OpenAI(base_url="http://127.0.0.1:8790/v1", api_key="unused") |
| decision = client.decisions.create( |
| model="gpt-6-luna", |
| input="I was charged twice for my order.", |
| questions=[{"type": "choice", "name": "department", |
| "instructions": "Which department should handle this complaint?", |
| "choices": [{"value": "billing", "description": "Payments, invoices, and refunds."}, |
| {"value": "technical", "description": "Problems using the product."}]}], |
| ) |
| decision.answers[0].choice # "billing"; .probabilities, .confidence as in OpenAI's API |
| ``` |
| |
| ### The `generate` extension: event dispatch |
| |
| Decide the event, copy its order id and write a note for the agent in one request. The note |
| is the only generated text; it is written after the other answers and sees them. |
| |
| ```python |
| from deg.client import Deg |
| |
| client = Deg("http://127.0.0.1:8790") |
| decision = client.decisions.create( |
| model="deg-latest", |
| input="Hi, order AB-1234 arrived with a cracked screen on 2026-10-01. " |
| "I want a refund, not a replacement.", |
| questions=[ |
| {"type": "choice", "name": "event", "instructions": "Which event should be dispatched?", |
| "choices": [{"value": "refund"}, {"value": "replacement"}, {"value": "other"}]}, |
| {"type": "extract", "name": "order_id", "instructions": "The order id."}, |
| {"type": "extract", "name": "delivered", "instructions": "Delivery date.", |
| "value_type": "date"}, |
| {"type": "generate", "name": "note", "instructions": "A one-line note for the agent.", |
| "max_length": 120, "max_tokens": 64}, |
| ], |
| ) |
| # answers (step-9000, GPU smoke 2026-10-09): |
| # {"type": "choice", "name": "event", "choice": "refund", "probabilities": [...], "confidence": 0.82} |
| # {"type": "extract", "name": "order_id", "value": "AB-1234", "span": [10, 17], "confidence": 0.998} |
| # {"type": "extract", "name": "delivered", "value": "2026-10-01", "span": [51, 61], ...} |
| # {"type": "generate", "name": "note", "text": "Customer requested a refund for defective |
| # order AB-1234 with a cracked screen on 2026-10-01, not a replacement.", "output_tokens": 35} |
| ``` |
| |
| - `extract`: `value_type` is `string` (default), `number`, `integer`, `date`, `date-time`, |
| `email`, `uri` or `uuid`; optional `pattern`, `min_length`, `max_length` (strings). The |
| answer is `{value, span, confidence}`; `value` is `null` with a `null_reason` when the input |
| does not contain it. |
| - `generate`: optional `max_tokens`, `min_length`, `max_length`, `pattern`; greedy. All |
| `generate` questions of one request must share `max_tokens`. |
| - Request `options` (`temperature` + `seed`, `date_order`, `default_offset`); with the stock |
| SDK pass them as `extra_body={"options": {...}}`. |
| - The stock `openai` SDK sends and parses the extensions too. It has no class for them, so |
| an extension answer comes back as an object whose `.type` is `"extract"`/`"generate"` and |
| whose fields (`.value`, `.span`, `.text`) are readable attributes. |
|
|
| ### TypeSafe Jev / SystemOne |
|
|
| ```python |
| client.systemone( |
| state="I was charged twice for order AB-1234. Fix it today!", |
| questions={ |
| "urgent": {"type": "noul", "instructions": "Is the customer asking for urgent action?"}, |
| "anger": {"type": "score", "instructions": "How angry is the customer?", |
| "criteria": ["calm", "annoyed", "angry", "furious"]}, |
| "order_id": {"type": "extract", "instructions": "The order id."}, |
| "reply": {"type": "generate", "instructions": "A short apology.", "max_length": 200}, |
| }, |
| ) |
| # {"model": ..., "answers": {"urgent": {"type": "noul", "noul": ...}, |
| # "anger": {"type": "score", "score": ..., "confidence": ..., "legend": {...}, "probabilities": {...}}, |
| # "order_id": {"type": "extract", "value": "AB-1234", ...}, "reply": {"type": "generate", "text": ...}}, |
| # "usage": {"input_tokens": ..., "output_tokens": ...}} |
| ``` |
|
|
| Existing Jev and Clef clients work unchanged (point them at `http://host:8790`). |
|
|
| ### deg fill |
|
|
| ```python |
| client.fill(context="Order AB-1234 was charged twice. Refund today!", schema=schema) |
| # {"model": ..., "value": {...}, "meta": {"/department": {"mode": "decide.choice", ...}, ...}, |
| # "usage": {"prompt_tokens": ..., "generated_tokens": ...}, "plan": "..."} |
| ``` |
|
|
| ## Tests |
|
|
| ```bash |
| cd packages/deg && uv run --package deg pytest |
| ``` |
|
|
