barryzhang/ornith-harness-4b-v2

Ornith Harness 4B v2

Ornith Harness 4B v2 is a 4.2-billion-parameter, Qwen3.5-derived text model distributed as a Q5_K_M GGUF artifact for local, machine-verifiable data-processing and coding workflows. The release is designed for systems with approximately 16 GB of available accelerator or unified memory and includes the thin tool-use harness used for final evaluation. The training version had three fewer audit-only fields and produced the same model-visible messages.

This repository contains the frozen v2 research release. Its primary uses are local agent research, verifier-based coding workflows, and reproducible benchmark experiments. General chat evaluation, ranking against frontier models, and autonomous production deployment are outside the evaluated scope. A registered, single-use 48-task final evaluation was completed on 30 July 2026; optimization stopped after the results were read.

Highlights

• Compact deployment artifact: 3.07 GB GGUF, evaluated with an 8,192-token context.
• Independent final estimate: v2 passed 32/48 tasks, for pass@1 0.667 (Wilson 95% CI 0.525–0.783).
• Within-lineage post-training gain: compared with C28, the pre-self-distillation initialization checkpoint, under the same Q5_K_M evaluation protocol, v2 improved from 8/48 to 32/48 tasks passed, a descriptive gain of 50.0 percentage points on this holdout.
• Higher completion rate under the registered deployment protocol: under this specific fixed 8K harness, runtime, prompt, and token budget—but with unmatched quantization—v2 Q5_K_M passed 32/48 tasks and Ornith-1.0-9B Q4_K_M passed 10/48. This is a comparison of the two packaged deployment routes under the registered protocol.
• Lower operational overhead: v2 had the lowest median primary tokens and wall time among the three final-evaluation arms.
• Matched model and harness: the release includes the two-tool agent loop and the evaluation entry points used to produce the reported development results.
• Reproducible identity: model, harness, Modelfile, and evaluation artifacts are bound to SHA256 identifiers.

Model overview

Property	Value
Model	Ornith Harness 4B v2
Release status	Frozen research release; final evaluation complete
Architecture	Qwen3.5-derived text-only causal language model (Qwen3_5ForCausalLM in the merged HF configuration)
Parameters	4.2B
Distribution format	GGUF, Q5_K_M
Artifact size	3,074,986,240 bytes
Evaluated context length	8,192 tokens
Tokenizer source	Qwen/Qwen3.5-4B
Reference prompt formatting	Included Ollama Modelfile
Tool interface	run_shell, write_file
Reference evaluation runtime	Ollama OpenAI-compatible /v1/chat/completions endpoint with c28_harness.py
Evaluation access ID	ornith-final-144-20260730T044153Z
Immutable release tag	v2.0.0
GGUF SHA256	7ccd17f484a0892c7f890559076d10d77c55ea9eb24aa6ffb0dfdef8f5468b9a
License	Apache License 2.0

Model lineage

The public lineage is:

1. Qwen3.5 4B architecture;
2. supervised fine-tuning on successful Ornith-1.0-9B tool-use trajectories;
3. C28 same-policy preference optimization, producing the pre-self-distillation initialization checkpoint;
4. first-round on-policy self-distillation (v1);
5. second-round on-policy self-distillation, with checkpoint-51 selected and formally merged into the v2 release artifact.

The intermediate C28 and self-distillation weights are internal lineage artifacts. This repository distributes the final v2 GGUF and the runtime materials required to evaluate it.

Intended use

Primary intended uses are:

• local agent research in an isolated tool-execution environment;
• verifier-based data-processing and coding workflows;
• reproducible benchmark experiments with the included harness;
• workflows where the user provides concrete input data, an explicit output format, and at least one representative input-output example when the transformation is ambiguous.

User acceptance validation is performed after delivery and is not part of the default model prompt. The included development evaluator uses executable checks to reproduce the reported research metrics. It does not define the production input contract.

Representative tasks include deterministic data normalization, aggregation, interval processing, state transformation, and generation of small executable scripts that operate on user-provided files.

Out-of-scope uses include:

• general chat evaluation and general conversational assistance;
• ranking against frontier models or general-purpose coding leaderboards;
• repository-scale refactoring and open-ended software architecture;
• unsupervised acceptance-criteria design and security-critical code generation;
• autonomous production deployment.

Training

Ornith Harness 4B was trained through two on-policy self-distillation rounds. In this release, self-distillation refers to verifier-guided self-improvement: the current policy generates trajectories, an objective verifier filters successful samples or constructs preference pairs, and the resulting data are used to update the same model lineage through SFT and DPO.

Stage	Training evidence
v1 SFT	26 verified successful trajectories
v1 DPO	148 high-confidence on-policy preference pairs
v1 winner	A2 checkpoint-45
v2 SFT	64 verified successful trajectories
v2 DPO	292 on-policy preference pairs
v2 DPO optimization	105 steps; checkpoints 17, 34, 51, 68, 85, 102, 105
v2 winner	checkpoint-51

Assistant-token masking excludes tool responses from the SFT loss. The v2 checkpoint was merged and converted with --no-mtp; the merge changed 128 of 426 tensors, consistent with the trainable projection layout.

Evaluation

Final holdout protocol

• Dataset: 48 tasks isolated from checkpoint selection.
• Sampling: one rollout per task and model; 144 episodes across three arms.
• Arms: C28 Q5_K_M, Ornith Harness 4B v2 Q5_K_M, and Ornith-1.0-9B Q4_K_M.
• Seed: 20260724; concurrency: 1.
• Hardware: one NVIDIA RTX PRO 6000 Blackwell.
• Runtime: the same Ollama service, harness, prompt, examples, output format, and token budget for all three arms.
• Generation: temperature 0.6, top-p 0.95, top-k 20, 2,048 completion tokens per turn, and 12,000 primary completion tokens per episode.
• Context: 8,192 tokens.
• Primary success event: PASS_VERIFIED_STOP.
• Intervals: task-level Wilson 95% intervals; paired comparisons use exact two-sided McNemar tests.

Final holdout results

Model	Quantization	Passed	pass@1	Wilson 95% CI	Median primary tokens	Median primary wall time
C28 starting point, 4.2B	Q5_K_M	8/48	0.167	[0.087, 0.296]	7,284	18.090 s
Ornith Harness 4B v2	Q5_K_M	32/48	0.667	[0.525, 0.783]	5,677.5	12.8325 s
Ornith-1.0-9B	Q4_K_M	10/48	0.208	[0.117, 0.343]	7,736	23.8475 s

C28 and v2 passed the same eight tasks; v2 alone passed 24 additional tasks. The paired difference was +0.500, with exact two-sided McNemar p=1.19e-7.

v2 and 9B both passed eight tasks; v2 alone passed 24 and 9B alone passed two. The 9B-minus-v2 difference was -0.458, with exact two-sided McNemar p=1.05e-5.

Under the fixed 8K context, episodes with history compaction were 37/48 for C28, 32/48 for v2, and 35/48 for 9B. Total compactions were 195, 113, and 142. These are operational endpoints, not causal estimates of compaction-induced failure.

Development selection protocol

• Dataset: eight held-out development tasks covering CSV grouping, interval merging, module splitting, state simulation, and text normalization.
• Sampling: five independent rollouts per task, 40 episodes per model.
• Hardware: one NVIDIA RTX PRO 6000 Blackwell.
• Runtime: Ollama, c28_harness.py, sequential execution, no rescue turns.
• Generation: temperature 0.6, top-p 0.95, top-k 20, 2,048 completion tokens per turn, 12,000 primary completion tokens per episode.
• Context: 8,192 tokens.
• Primary success event: objective verifier satisfied (PASS_VERIFIED_STOP).

Development selection results

Model	Quantization	pass@1	pass@5	Realizable BoN@5	Turn-cap rate	Median completion tokens, successful rollouts
C28 starting point, 4.2B	Q5_K_M	0.125	0.500	0.500	0.300	2,960
Ornith Harness 4B v1	Q5_K_M	0.300	1.000	0.875	0.275	1,225
Ornith Harness 4B v2	Q5_K_M	0.675	1.000	1.000	0.075	1,109
Ornith-1.0-9B	Q4_K_M	0.150	0.500	0.375	0.225	2,261

The v2 formal rerun produced 27 PASS_VERIFIED_STOP, 12 FAIL_TASK, and one FAIL_GENERATION_CAP outcome across 40 episodes.

pass@1 is the mean single-rollout success rate. pass@5 records whether each task had at least one successful rollout. Realizable BoN@5 selects a candidate using the task's agent-visible executable checks and then scores the selected output with the offline objective verifier. The BoN result therefore requires a validation rule and is not available for tasks without one.

These development results selected checkpoint-51 from seven candidates. The v2 value 0.675 is therefore a selection statistic subject to winner's curse, not the independent headline performance estimate. This repository does not include a production N=5 execution and selection component.

Comparison boundary

The 4B-to-9B comparison represents two artifacts designed for an approximately 16 GB single-device deployment constraint. Quantization is not matched: v2 uses Q5_K_M, while Ornith-1.0-9B uses Q4_K_M. Evaluation ran on an RTX PRO 6000, not on 16 GB hardware. The table therefore compares deployment routes under a fixed harness and budget; it does not rank the underlying models at equal precision or claim measured 16 GB-device performance.

Published Ornith-1.0-9B benchmark results use different weights, context lengths, agent frameworks, and compute budgets. They are not comparable with the local development protocol reported here.

Usage

Place ornith-harness-4b-v2-Q5_K_M.gguf in this directory, then create the Ollama model:

ollama create ornith-harness-4b-v2 -f Modelfile

Run the included development evaluation:

python eval_selfdistill.py \
  --whitelist dev_whitelist.json \
  --pool pool/dev8.jsonl \
  --url http://127.0.0.1:11434/v1/chat/completions \
  --model ornith-harness-4b-v2 \
  --label eval \
  --n-reps 5 \
  --concurrency 1 \
  --rescue-turns 0 \
  --out /tmp/ornith-harness-4b-v2-eval \
  --distill-root . \
  --harness-profile v2 \
  --max-tokens 2048 \
  --primary-completion-budget 12000

python bon_report.py /tmp/ornith-harness-4b-v2-eval/rows/eval.jsonl

The working directory must be writable and traversable by the sandbox identity that executes model-generated commands.

Safety and operational requirements

The harness executes model-generated shell commands and writes files. Run it only in an isolated workspace with:

• no production credentials or personal data;
• restricted network access;
• explicit CPU, memory, process, file-size, and time limits;
• a disposable filesystem or recoverable snapshot;
• independent validation of every delivered output.

Do not execute untrusted model output directly on a host system.

Limitations

1. The final estimate comes from a fixed 48-task engineering benchmark, not a random sample of all real user tasks.
2. Results are bound to one machine, one runtime, one harness, and one decode configuration.
3. Realizable BoN@5 requires an executable user-visible validation rule.
4. The current harness stops immediately when the objective verifier passes. Consequently, successful episodes have no later turn in which to declare DONE; DONE precision is NOT_EVALUATED_DESIGN and self-reported completion is not a supported selector.
5. The 4B and 9B comparison uses different quantization levels.
6. The final comparison ran on an RTX PRO 6000. Actual 16 GB hardware deployment remains a separate acceptance test.
7. BoN@5 is an offline development analysis in this release; a production multi-sample runtime and sandbox have not been delivered.
8. This research release supports artifact verification and development-set evaluation. Reproducing the full training, checkpoint-selection, and sealed-holdout workflow requires controlled project assets and collaboration with the maintainers.

Artifact integrity

Artifact	SHA256
ornith-harness-4b-v2-Q5_K_M.gguf	7ccd17f484a0892c7f890559076d10d77c55ea9eb24aa6ffb0dfdef8f5468b9a
Evaluation harness used for the final run, c28_harness.py	58fab23bfd183c2ff049e758e86ed27e9c6cafbf34db80813cf4895ba421dddf
v2 selection and formal-rerun archive	2fc3309f3d197f29c3d786f2c485a4dbd4283f523d4af00eef1343bfade6aeef
v2 training-artifact archive	320b11059ec1c12d4ea99dc59985fb9bba12a6fb5de4aa4702ef54ffd5c4b229
Final holdout result package	ad15712a23422adcfc156379e78d8a9c3fc145e676078f8e84198960634b85f2
Final score	0c7bf6ca34fc40f67415b197d4039a1dd137ad1a42b956ea1b3f271e19ae9ce8
Final infrastructure gate	e76405e82c3e50dca77aea69523e4f8e76579acac22a1ee7e79e00e5cb31d651

The release manifest SHA256SUMS.txt records the package-level file hashes.

Upstream references

• Qwen3.5-4B model card
• Ornith-1.0-9B model card

The upstream Ornith attribution, copyright notice, and MIT License text are preserved in NOTICE.

When citing this release, include the repository revision and the v2 GGUF SHA256 listed above.