electroglyph/elephant-1-onnx

elephant-1 — ONNX

ONNX builds of /home/anon/LLM/elephant-1 (Baseten elephant-1, a 9B Qwen3.5 decision model), runnable with onnxruntime 1.31.1. No torch needed at inference.

This is a decision model, not a chatbot: you send a state plus typed questions (yes/no, pick-one, scored rubric) and get back a probability distribution per question — one prefill each, no text generation. The main script is score_elephant.py; generate.py (free text) and csharp/ are secondary.

The builds preserve the reference serving's attention: the 8 full-attention layers run block-bidirectional (prefix block and suffix block each bidirectional), the 24 linear-attention layers stay recurrent, exactly as the checkpoint was trained and served (modeling_overrides + readout.json). This is load-bearing, not cosmetic: on the support example below the refund score is 0.79 with the correct mask. T = 1.0 (no calibration file ships).

Decision scoring

pip install -r requirements.txt          # GPU line, or swap to the CPU line
python score_elephant.py --request examples/support.json --model ./model-cuda-int4 --cuda

A request is a /v1/systemone body: {"state": ..., "questions": {...}}. Three question types:

• noul — yes/no (Should the agent offer a refund? → {"noul": 0.79})
• choice — pick one of k (priority → urgent at 0.88)
• score — rubric levels 0..k-1, returns expectation + full distribution

Example 1: support triage (examples/support.json)

State: customer whose laptop shipping stalled, promised callback never came, needs it for work Monday. Four questions, one request:

python score_elephant.py --request examples/support.json --model ./model-cuda-int4 --cuda

{
  "answers": {
    "refund": {"type": "noul", "noul": 0.7853},
    "priority": {"type": "choice", "choice": "urgent",
      "probabilities": {"low": 0.039, "normal": 0.084, "urgent": 0.878}},
    "tone": {"type": "choice", "choice": "angry",
      "probabilities": {"angry": 0.979, "neutral": 0.020, "happy": 0.001}},
    "satisfaction": {"type": "score", "score": 0.455,
      "probabilities": {"0": 0.625, "1": 0.317, "2": 0.042, "3": 0.013, "4": 0.004},
      "legend": {"0": "very dissatisfied", "1": "dissatisfied", "2": "neutral",
                 "3": "satisfied", "4": "very satisfied"}}
  },
  "usage": {"input_tokens": 361, "output_tokens": 4}
}

Read: refund is the close call in the pile (0.79 for, not slam-dunk), ticket is urgent, customer is angry, expected satisfaction 0.45/4.

Example 2: content moderation (examples/moderation.json)

State: 2-day-old account posting a crypto "secret system" fee scam. Three questions:

python score_elephant.py --request examples/moderation.json --model ./model-cuda-int4 --cuda

{
  "answers": {
    "violates": {"type": "noul", "noul": 0.9980},
    "category": {"type": "choice", "choice": "scam",
      "probabilities": {"legit": 0.0003, "spam": 0.003, "scam": 0.996, "abuse": 0.0002}},
    "severity": {"type": "score", "score": 3.4850,
      "probabilities": {"0": 0.024, "1": 0.023, "2": 0.058, "3": 0.231, "4": 0.663}}
  },
  "usage": {"input_tokens": 329, "output_tokens": 3}
}

Read: violates policy (0.998), scam not mere spam (0.996), severity 3.49/4 — auto-remove territory.

Example 3: incident routing (examples/routing.json)

State: a structured object (not a string) — checkout-api 5xx spike, deploys, on-call info. Three questions, including an explicit noul criteria pair:

python score_elephant.py --request examples/routing.json --model ./model-cuda-int4 --cuda

{
  "answers": {
    "page": {"type": "noul", "noul": 0.9483},
    "team": {"type": "choice", "choice": "payments",
      "probabilities": {"payments": 0.989, "platform": 0.010, "fraud": 0.001}},
    "urgency": {"type": "score", "score": 1.6985,
      "probabilities": {"0": 0.023, "1": 0.255, "2": 0.722},
      "legend": {"0": "routine", "1": "elevated", "2": "critical"}}
  },
  "usage": {"input_tokens": 357, "output_tokens": 3}
}

Read: page now (0.948), payments owns it (0.989), urgency critical-leaning.

Notes for decision use

• Prompts use the qev layout (system prompt, <state> block, slot letters, qev-labels-v2) with the original tokenizer, so outputs are comparable with the reference serving. T = 1.0, no calibration file needed.
• score returns the full distribution — use the spread to route to humans, not just the mean.
• Up to 256 options per question (slot cap), score ≤ 10 levels, branch ≤ 32768 tokens, request ≤ 65536 tokens. Text only; images need the torch serving in the source repo.
• If your use depends on probability thresholds, re-check calibration on your own data; INT4 shifts probabilities slightly — measured drift below.

Chat generation (secondary)

python generate.py --model ./model-cuda-int8 --cuda --prompt "What is the capital of France?"
python generate.py --model ./model-cuda-int4 --cuda --prompt "What is the capital of France?"
python generate.py --model ./model-cpu --interactive

C# runner (csharp/)

.NET 10 sample: raw-onnxruntime runner + generation and one scored-decision demo, tokenizing with gigatoken_cs (verified 19/19 byte-identical to HuggingFace on unicode/CJK/emoji/specials).

cd csharp
dotnet run -- --model ../model-cuda-int4 --cuda --prompt "What is the capital of France?"
dotnet run -- --model ../model-cuda-int4 --cuda --score-demo
dotnet run -- --model ../model-cpu --score-demo   # CPU build, no GPU needed

Verified: prompt bytes sha-identical to Python (9b352e9d…), prefix split 58 = 58, refund noul 0.9342 (CUDA int4) and 0.9069 (CPU) = Python to 4 decimals, generation Paris.

Contents

path	what
model-cuda-int8/	int8 build (most faithful): model.onnx + model.onnx.data (~11 GB). Full int8 weights, chunks of 32, running math untouched
model-cuda-int4/	int4 build: model.onnx + model.onnx.data (~7 GB). True int4, k-quant, block 32, only the LM head at int8
model-cpu/	int4 CPU build: model.onnx + model.onnx.data (~9 GB)
model-cpu-int8/	int8 CPU build (most faithful on CPU): model.onnx + model.onnx.data (~10 GB)
examples/	ready-made decision requests (support.json, moderation.json, routing.json)
onnx_raw.py	shared runner: OnnxLM (prefill / step / generate, block_bias feed)
generate.py	chat / text generation
score_elephant.py	decision scoring (this README's main script)
diff_bidi.py	accuracy check vs the original torch answers (needs the torch reference file)
bench_el.py, bench_el_base.py	speed check per build + raw original weights
csharp/	.NET 10 runner + demos
orig_tokenizer/	pristine tokenizer — scoring prompts stay byte-identical to the reference
requirements.txt	runtime deps
config.json (in each build)	source architecture config (Hub convention)
LICENSE	MIT for the runner scripts (weights per source repo, see below)

How it was built

Plain torch.onnx.export / optimum cannot handle this model: 24 of the 32 decoder layers are GatedDeltaNet linear-attention (recurrent + conv state), which has no standard ONNX representation. The export path is Microsoft's onnxruntime-genai ModelBuilder (0.19.0, onnxruntime 1.31.1), then graph surgery for the attention mask:

# int8 export (most faithful): full int8 weights, chunks of 32, running math
# untouched, smarter packing
python -m onnxruntime_genai.models.builder \
  -i <elephant-1-checkpoint> -o model-cuda-int8 -p int8 -e cuda \
  --extra_options exclude_mtp=1 block_size=32 algo_config=k_quant
# int4 export (MTP excluded: this checkpoint ships no MTP weights)
python -m onnxruntime_genai.models.builder \
  -i <elephant-1-checkpoint> -o model-cuda-int4 -p int4 -e cuda \
  --extra_options exclude_mtp=1 block_size=32 algo_config=k_quant \
  matmul_mixed_precision=last_matmul:int8
# cpu int4: accuracy_level=4 block_size=32
# cpu int8 (most faithful on CPU)
python -m onnxruntime_genai.models.builder \
  -i <elephant-1-checkpoint> -o model-cpu-int8 -p int8 -e cpu \
  --extra_options exclude_mtp=1 accuracy_level=4 block_size=32

Then per build, graph surgery for the attention mask: ORT's GQA op has no mask input, so each of the 8 GQA nodes was replaced with an exact query-chunked eager subgraph (8 chunks, shared block_bias input; RoPE sits outside GQA so it is untouched). Weights are untouched — each model.onnx shares its build's .onnx.data. A tokenizer fix for the GenAI C++ tokenizer was applied to both builds.

All builds are text-only decoders; the vision tower is not exported.

Running it: raw onnxruntime (not onnxruntime-genai)

The runners drive the graph with plain onnxruntime (prefill + greedy decode with manual KV/conv/recurrent state). Reason: onnxruntime-genai 0.19.0 fails at Model load for this export with GetMemoryInfo for CPU should not be used, while onnxruntime 1.31.1 runs the identical graph fine. genai_config.json is kept as exported, so the GenAI path can be retried with future releases.

Verification

Reference for accuracy: the actual qev engine (OneJev @81ce62f + the checkpoint's serve.patch), bf16, sequential branches, both examples plus routing. Times below are per request on this machine (RTX 3090 Ti + processor), after warmup, via diff_bidi.py (accuracy) and bench_el.py (speed). The original's speed row runs its raw weights over the exact same prompts through plain transformers (no serving stack, and its linear-attention falls back to slow reference kernels here — the tuned serving is quicker).

model	file size	support	moderation	routing	biggest miss vs original	same answers
original (full precision)	18 GB	0.29 s	0.27 s	0.31 s	—	—
model-cuda-int8	11 GB	0.58 s	0.49 s	0.54 s	0.007	7 of 7
model-cuda-int4	6.8 GB	0.59 s	0.50 s	0.52 s	0.090	7 of 7
model-cpu	8.6 GB	6.1 s	5.2 s	6.3 s	0.139	7 of 7
model-cpu-int8	10 GB	5.7 s	4.9 s	6.2 s	0.018	7 of 7

• The 8-bit build is the faithful one: worst miss 0.007, support refund 0.773 vs 0.766 original, same side. Self-test refund 0.879. "Capital of France?" → Paris in ~0.2 s.
• The 4-bit build matches on every answer (worst miss 0.090); its wobbles are on satisfaction/severity levels. Self-test refund reads 0.93 — same verdict, runs hot. "Capital of France?" → Paris in ~0.2 s.
• The processor builds match on every answer too (worst miss 0.018 on 8-bit, 0.139 on 4-bit) at ~5–6 s a request — prefer the 8-bit processor build wherever thresholds matter.
• Cross-runtime: Python == C# to 4 decimals on all four builds (cuda-int8 0.8791, cuda-int4 0.9342, cpu 0.9069, cpu-int8 0.8921).
• Weight files checked present and size-consistent in all four builds; tokenizer fix applied in all three; serving-file copies byte-identical.

Notes / limits

• Provider matrix is strict: model-cuda-int8 and model-cuda-int4 need CUDAExecutionProvider (--cuda), model-cpu and model-cpu-int8 need CPU. The CUDA exports use GatedDeltaNet (CUDA-only); the CPU export uses LinearAttention. Cross combinations fail at load with an explicit message.
• INT8 barely moves probabilities vs the original (worst miss 0.007 graphics, 0.018 processor); INT4 shifts them slightly (0.090 graphics 4-bit, 0.139 processor 4-bit). Use the 8-bit builds wherever thresholds matter.
• onnxruntime-genai's C++ tokenizer uses the fixed tokenizer.json in each build dir; scoring tokenizes with orig_tokenizer/ instead, so prompts are unaffected by that fix. The C# sample loads orig_tokenizer/ too (gigatoken_cs cannot parse the fixed file's pre-tokenizer — both files tokenize byte-identically).

License

• Weights (*.onnx.data): as listed on the source repo (baselabs/elephant-1).
• score_elephant.py ports rendering/scoring from OneJev qev (MIT) plus the checkpoint's serve.patch; confidence formulas via qev from TypeSafe AI (MIT).
• Runner scripts and csharp/: MIT — see LICENSE (gigatoken_cs has its own license).