AX-Ray, Finding Causal-Leakage Defects in Two General-Purpose Public Models

How VIDRAFT AX-Ray moves AI evaluation beyond capability scores and into reproducible safety diagnostics

AI models should no longer be evaluated only by how well they answer benchmark questions. Capability matters, but deployment safety depends on a wider set of properties: causal correctness, serving consistency, robustness under adversarial or long-context conditions, data integrity, infrastructure security, regulatory readiness, and agentic risk.

VIDRAFT built AX-Ray to address that gap. AX-Ray is a safety-diagnostics layer for AI and AX deployment, powered by FINAL-Bench Diagnostics. It evaluates not only whether a model is intelligent, but whether the model and its serving environment can be trusted under practical deployment conditions.

• Space: FINAL-Bench/AX-RAY
• Dataset: FINAL-Bench/AX-RAY

The Core Finding

AX-Ray presents a public diagnostic case in which two general-purpose public models were identified, reproduced, and demonstrated as having causal-leakage defects:

• Zyphra/Zamba2-1.2B
• nvidia/Nemotron-H-8B-Base-8K

These models are highlighted in the AX-Ray leaderboard as Causal-LEAK cases.

This finding is important because causal leakage is not an ordinary benchmark failure. It is a structural correctness failure. A model can score well on conventional capability tests while still violating a causal constraint that should hold during autoregressive inference.

AX-Ray treats confirmed causal leakage as a deployment-blocking defect. When a model is confirmed to leak future or suffix information into prefix behavior, the issue is handled by an F-gate regardless of aggregate capability score.

What Causal Leakage Means

In an autoregressive language model, the representation or logit behavior at a prefix position should not depend on future tokens that are not causally available at that position. Causal leakage occurs when future or suffix information changes prefix hidden states, logits, or related scoring behavior.

This is not the same as hallucination, refusal failure, prompt injection, or benchmark contamination. Those are serious issues, but causal leakage is different: it concerns the correctness of the model's computation path itself.

A causal-leakage defect can affect:

• prefix invariance
• hidden-state correctness
• logit consistency
• chunked or hybrid sequence processing
• cache and serving reliability
• long-context trustworthiness
• evaluation validity
• agentic execution safety

VIDRAFT's causal-leakage diagnostic work is protected by patent rights. The public article therefore explains the concept and the safety significance, but does not disclose proprietary probe recipes, internal thresholds, raw adversarial prompts, sensitive outputs, or implementation details that would undermine the diagnostic method or enable misuse.

Why Capability Benchmarks Miss This

Most public leaderboards measure how often a model gives the right answer. That is necessary, but incomplete. A model can answer many questions correctly while still having unsafe or inconsistent internal behavior.

Causal leakage may remain invisible in ordinary QA, math, coding, or instruction-following benchmarks because those tests usually observe final answers. AX-Ray inspects deeper properties: whether the prefix remains invariant, whether serving paths remain consistent, and whether critical failures should override aggregate score.

The central principle is simple:

High capability does not imply deployment safety.

AX-Ray Diagnostic Structure

AX-Ray is organized around three diagnostic axes, eleven operational categories, and a public catalog of 117 diagnostic item records.

The published item catalog currently contains 117 named records, distributed across MODEL-SCAN and AX-SCAN. AGENT-SCAN is represented as a first-class diagnostic axis and reporting category for deployment and agent-risk assessment.

The 117 public item records are not just a checklist. Each item is designed to connect a technical diagnostic question to evidence, severity, detection direction, remediation direction, and governance context.

Three Diagnostic Axes

Eleven Categories

Public 117-Item Catalog

The complete item-level catalog is available in the dataset. The article lists the public item titles below so readers can understand the scope before consulting the dataset.

D1. Causal Safety and Serving Integrity

1. Prefix invariance / causal leak
2. Chunked-scan and masking consistency
3. Batch invariance / determinism
4. KV-cache path consistency
5. Padding invariance
6. Cross-implementation consistency
7. Numerical precision fp32/bf16/fp8
8. Attention-sink and streaming
9. Tokenizer and chat-template fidelity
10. Position-encoding / RoPE integrity
11. Prefix-cache isolation / side-channel

D2. Reliability

12. RAG / summarization faithfulness
13. Closed-book factual hallucination
14. Long-form factual precision
15. Black-box consistency
16. Semantic-entropy uncertainty
17. Confidence calibration
18. Selective prediction / abstention
19. Metacognition / self-knowledge
20. Sycophancy
21. Alignment or reasoning overconfidence
22. Citation fabrication

D3. Robustness and Long Context

23. Adversarial suffix
24. Character and word perturbation
25. Prompt-format brittleness
26. MCQ option-order bias
27. Lost-in-the-middle / positional bias
28. Effective versus claimed context
29. Latent-association retrieval
30. Instruction-following
31. Paraphrase invariance
32. Logical / order consistency
33. Out-of-distribution robustness
34. Noisy or code-switched input

D4. Security, Safety, and Alignment

35. White-box jailbreak
36. Black-box automated jailbreak
37. Multi-turn and many-shot jailbreak
38. Weight-level attack
39. Prompt injection and exfiltration
40. Agentic tool-use / excessive agency
41. Guardrail and moderation efficacy
42. Toxicity generation
43. Bias and fairness
44. Refusal calibration
45. Automated red-teaming coverage
46. Standards and governance mapping

D5. Data Integrity and Evaluation Methodology

47. Benchmark contamination detection
48. Membership inference
49. Memorization / extraction
50. Copyright regurgitation
51. PII leakage / privacy
52. Data provenance / openness audit
53. Construct validity
54. Statistical rigor
55. Signal-to-noise / at-chance behavior
56. LLM-as-Judge reliability and bias
57. Leaderboard integrity
58. Contamination-resistant / dynamic evaluation

D6. Efficiency, Quantization, and Architecture

59. Quantization sensitivity / outliers
60. Per-language quantization degradation
61. Low-bit / imatrix calibration
62. Precision by training-budget vulnerability
63. Sparsity sensitivity
64. Structured or layer pruning attribution
65. MoE / expert health
66. Distillation fidelity / task-axis forgetting
67. Effective capacity / undertraining
68. Serving efficiency
69. KV-cache compression / long-context scaling
70. Speculative decoding gains and pitfalls

D7. White-Box Internal Structure

71. Layer importance / redundancy
72. Golden layer / phase transition
73. MoE routing health
74. Expert specialization / semantic labeling
75. Attention-head taxonomy
76. Attention sinks and entropy
77. Activation outliers to quantization fragility
78. Representation health
79. Logit / tuned lens
80. Activation patching / causal tracing
81. Knowledge localization and editing
82. Circuit tracing / sparse autoencoders

D8. Remediation and Treatment

83. Golden-layer targeted LoRA
84. Layer-guided merge
85. Expert rebalance / pruning
86. Outlier-aware mixed-precision quantization
87. Knowledge editing
88. Metacognition LoRA / abstention
89. Per-token expert surgery

AX-A. Serving Infrastructure

90. Cross-engine serving drift
91. Quantization-in-production degradation
92. Latency / throughput / goodput
93. KV and prefix-cache cross-tenant isolation
94. Serving determinism under load
95. Speculative decoding losslessness
96. Autoscale / cold-start / OOM

AX-B. Infrastructure Security

97. Endpoint authentication / LLMjacking
98. SSRF via tool-use
99. Model supply-chain integrity
100. Serving-stack RCE / dependency CVE
101. Secrets and container hardening
102. Unbounded consumption / denial-of-wallet
103. PII, audit logging, and retention compliance

AX-C. Regulatory Compliance

104. High-impact AI obligations
105. GenAI marking, watermarking, and deepfake notice
106. Frontier or large-scale AI safety
107. Domestic representative requirements
108. ISMS-P technical controls
109. Automated-decision rights
110. EU AI Act conformity
111. Financial AI governance
112. Credit-scoring model validation
113. GenAI security rules
114. KCMVP / CC certification
115. N2SF national network security
116. PIPC AI self-assessment
117. TTA performance and data certification

Legal, Regulatory, Ethical, and Religious-Law Mapping

AX-Ray is presented as a diagnostic guideline, not as a formal legal standard. Its purpose is to connect technical model failures to deployment accountability.

The 117 public item records are mapped to governance contexts across multiple jurisdictions and legal traditions, including:

• Korea: AI Framework Act, Personal Information Protection Act, ISMS-P, NIS public-sector security requirements, N2SF, KCMVP, public procurement and certification contexts
• European Union: EU AI Act, GDPR, NIS2, Cyber Resilience Act and related conformity concepts
• United States: NIST AI RMF, FTC data-security and deceptive-practice principles, CISA guidance, MITRE ATLAS, OWASP LLM risks
• Japan: AI business guidelines, APPI privacy requirements and related safety guidance
• China: generative-AI interim measures, PIPL, cybersecurity and data-security governance
• United Arab Emirates: AI charter, PDPL, public-sector AI governance and relevant ethical contexts
• Saudi Arabia: SDAIA AI ethics, PDPL, NCA cybersecurity guidance and public-sector AI governance

For the UAE and Saudi Arabia, the mapping also includes relevant religious-law and ethical governance contexts where appropriate, including principles such as protection of life, protection of intellect, truthfulness, avoidance of deception, and prevention of harm. AX-Ray does not treat these as cosmetic annotations. The purpose is to make safety diagnostics legible to real deployment environments where law, public ethics, religious-law reasoning, institutional responsibility, and operational risk all interact.

The full mapping is too detailed for an article. Each item-level record can include diagnostic focus, rationale, detection direction, remediation direction, severity, automation level, evidence basis, and jurisdiction-specific legal or ethical references. Readers should consult the dataset for the complete item-level mapping:

https://huggingface.co/datasets/FINAL-Bench/AX-RAY

How AX-Ray Handles Solar-Open2-250B

AX-Ray also records an API-audited result for upstage/Solar-Open2-250B served through FP8 vLLM. This entry is deliberately marked differently from confirmed causal-leakage cases.

The Solar-Open2-250B run showed a repeatable prompt_logprobs scoring anomaly in the serving/API path. AX-Ray treats this as an API or serving caveat, not as confirmed model-level causal leakage. White-box D1/D7 testing remains pending, so the row is marked official_dhs=false and API-audited / white-box pending.

This distinction is central to AX-Ray's methodology. A serving anomaly, an API scoring issue, and a confirmed hidden-state causal leak are not the same claim.

Public Disclosure Policy

AX-Ray discloses:

• leaderboard rows
• model-level diagnostic summaries
• high-level category scores
• item taxonomy
• jurisdictional mapping structure
• selected public reports
• confirmed causal-leakage highlights

AX-Ray does not disclose:

• proprietary probe recipes
• threshold-level implementation details
• raw sensitive prompts
• raw harmful outputs
• claim-sensitive patent implementation details
• internal judge prompts or scoring recipes
• exploit-ready procedures

This balance is intentional. Safety diagnostics must be transparent enough to support accountability, but not so detailed that they become an instruction manual for bypassing models or reproducing harmful behavior.

Why This Matters

The AI community has become very good at measuring capability. We now need the same seriousness in measuring safety, causal correctness and deployment readiness.

The discovery of causal-leakage defects in general-purpose public models demonstrates that safety diagnostics can reveal failures that ordinary performance benchmarks miss. AX-Ray is not a replacement for capability benchmarks. It is a second layer: a deployment-oriented diagnostic framework that asks whether the model, its serving path, and its agentic environment can be trusted.

AI systems are moving into public services, finance, healthcare, research automation, industrial operations and agentic workflows. In those environments, a high benchmark score is not enough.

The next generation of AI evaluation must answer a harder question:

Is the model safe to deploy, operate and govern?

AX-Ray is VIDRAFT's first public step toward that evaluation layer.