File size: 7,795 Bytes
3145545 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | """Generate the release report from raw evidence, not hand-entered benchmark numbers."""
import json
import statistics
import sys
from pathlib import Path
def report(diagnostic_path, stress_path):
diagnostic = json.loads(Path(diagnostic_path).read_text())
stress = json.loads(Path(stress_path).read_text())
lines = [
"# PCD measurements and limitations",
"",
"Measured 2026-09-16 on one NVIDIA L40S, FP16, PyTorch 2.14.0, Transformers 5.17.0, SDPA attention and reference PyTorch convolution. No optimized causal-conv1d, torch.compile, vLLM/SGLang or Jev service comparison.",
"",
"## Release decision",
"",
"**Experimental inference-only release, not a production-quality or calibrated Jev substitute.** The latency goal is met on multi-field examples. The proposed accuracy non-inferiority goal is **not met**: use application-specific evaluation, an autoregressive fallback, or separately funded training before relying on it. No always-on service was deployed.",
"",
"## Development and fresh audit",
"",
"The 12 development cases were used to debug/select prompts. Six additional cases were written after freezing the protocol and not used to retune it. Both sets are tiny, hand-authored, and not representative production data. Each method/case has one untimed warmup and three measured repetitions. Accuracy counts unique cases, not repeats.",
"",
"| Split | Method | Mean ms | Median ms | p95 ms | Field accuracy | Exact objects | Schema valid |",
"|---|---|---:|---:|---:|---:|---:|---:|",
]
for split in ("development", "audit"):
for method in ("token", "sequence", "ar_sequence", "ar_token"):
s = diagnostic["split_summary"][split][method]
lines.append(
f"| {split} | {method} | {s['mean_ms']:.2f} | {s['median_ms']:.2f} | {s['p95_ms']:.2f} | {s['field_accuracy']:.1%} | {s['exact_accuracy']:.1%} | {s['schema_compliance']:.1%} |"
)
lines += [
"",
"`ar_sequence` is greedy direct-answer JSON using original values and the same sequence-mode system/schema. `ar_token` uses the code catalog and an all-field JSON request; codes are strictly decoded, not repaired. Token PCD asks independent per-field questions, so its suffix prompt and distribution are not identical to joint AR. All use the same model, precision and backend. Empty thinking is explicit for these direct-answer modes.",
"",
"## Synthetic scaling (one distinct input per row)",
"",
"| Case | Token PCD ms | Sequence PCD ms | AR JSON ms | AR/token ratio | Token / sequence / AR field accuracy |",
"|---|---:|---:|---:|---:|---|",
]
for case in stress["cases"]:
groups = {
method: [
r for r in stress["rows"] if r["case_id"] == case["id"] and r["method"] == method
]
for method in ("token", "sequence", "ar_sequence")
}
means = {
method: statistics.mean(r["result"]["elapsed_ms"] for r in rows)
for method, rows in groups.items()
}
accuracies = [
f"{rows[0]['evaluation']['field_correct'] / rows[0]['evaluation']['field_total']:.0%}"
for rows in groups.values()
]
lines.append(
f"| {case['id']} | {means['token']:.2f} | {means['sequence']:.2f} | {means['ar_sequence']:.2f} | {means['ar_sequence'] / means['token']:.1f}x | {' / '.join(accuracies)} |"
)
lines += [
"",
"**High-cardinality warning:** token mode chose the wrong answer in both the 64- and 255-category probes. Sequence mode was correct on these inputs, but the 255-category sequence scorer was slower than AR. Do not interpret the high aggregate boolean accuracy as high-cardinality accuracy. The 28-field example is literal configuration extraction, not a broad semantic understanding benchmark.",
"",
"## Incorrect constrained outputs (one per unique case)",
"",
"| Case | Method | Expected | Actual |",
"|---|---|---|---|",
]
expected = {case["id"]: case["expected"] for case in diagnostic["cases"] + stress["cases"]}
for row in diagnostic["rows"] + stress["rows"]:
if (
row["repeat"]
or row["method"] not in {"token", "sequence"}
or row["evaluation"]["exact_match"]
):
continue
gold = json.dumps(expected[row["case_id"]], ensure_ascii=False).replace("|", "\\|")
actual = row["result"]["text"].replace("|", "\\|").replace("\n", " ")
lines.append(f"| {row['case_id']} | {row['method']} | `{gold}` | `{actual}` |")
lines += ["", "## Correctness and resource evidence", ""]
for name, data in [("diagnostic", diagnostic), ("stress", stress)]:
validation = data["validation"]
errors = {r["mode"]: r["max_score_error"] for r in validation["score_comparisons"]}
peak = max(r["peak_allocated_bytes"] for r in data["rows"]) / 2**30
lines.append(
f"- {name}: validation passed; maximum candidate-score differences against uncached reference {errors}; peak allocated VRAM {peak:.2f} GiB; in-function time {data['wall_seconds']:.2f}s."
)
lines += [
"- Local tests exercise a tiny FP32 hybrid model; CUDA validation also tests the actual model in FP32 and FP16. Both single-token and multi-token convolution cache paths preserve the original prefix state. Score tolerances are 0.002 in FP32 and 0.10 in FP16; raw measured errors are retained, not assumed zero.",
"- An early BF16 hidden-state check failed (maximum observed difference 0.25 with 0.15 absolute tolerance). We did not relax that BF16 gate; FP16 is the recommended tested runtime precision. Original stored BF16 weights remain unchanged.",
"- Candidate probabilities are explicitly uncalibrated. No ECE, Brier-score calibration, learned confidence threshold, or proprietary Jev RLCD training is claimed.",
"- Request timings include prompt tokenization, cached-schema lookup, prefill, cache forking, all microbatches, candidate scoring, and serialization; CUDA is synchronized. They exclude model loading, network, provisioning and output evaluation. Initial schema compilation and cold starts are not benchmarked separately.",
"- p95 estimates are descriptive with very small samples. GPU clocks/allocator/runtime effects varied between exploratory jobs; the report uses the final fixed-source runs, not the fastest observed earlier latency.",
"- Native reasoning was sampled during development on three cases with a 256-token cap: two truncated and one emitted fenced JSON. Those are not valid completed baseline results and are not used to claim PCD speedups or native-model incompetence. We avoided paying to repeat that capped experiment.",
"- Frugal execution: serial single-L40S jobs, cached public weights, bounded runtimes, no H100, no training and no permanent deployment. In-function seconds are not a billing statement; startup/idle/CPU/storage charges are additional.",
"",
"## Raw release evidence",
"",
f"- [Diagnostic/audit JSON](../{diagnostic_path})",
f"- [Scaling JSON](../{stress_path})",
"",
"Raw files include every measured output, score, token/branch count, version and inference-source hash. The selected prompt was not changed after the fresh audit. Earlier iterations are development artifacts and are not release performance claims.",
]
target = Path("docs/pcd-results.md")
target.write_text("\n".join(lines) + "\n")
print(target)
if __name__ == "__main__":
report(*sys.argv[1:])
|