| # FrontierLab · Claim-to-Evidence Evaluation and Meta Model Path |
|
|
| Date: 2026-10-27. Code baseline status: synthetic benchmark and NVIDIA Muse Glimmer NIM client ready for **manual** trial. Two real NVIDIA API requests were made, but neither returned final-answer text; genuine model output is NOT yet verified. |
|
|
| ## Research foundation (PDFs in Undermind workspace) |
|
|
| Undermind FrontierLab folder: https://app.undermind.ai/projects/362994e3-20f1-4bb8-9984-9a088a13c27f?path=/frontierlab-core-papers/ |
| Reading notes: https://app.undermind.ai/projects/362994e3-20f1-4bb8-9984-9a088a13c27f?path=/frontierlab-core-papers/READING_NOTES.md |
| |
| - Rasheed et al. (2026), DOI 10.48550/arXiv.2602.13855: proposes claim-level auditability as a research target, including provenance coverage/soundness, contradiction transparency, human audit effort. These are PROPOSED metrics, not independently proven by the paper's illustrative diagram. |
| - Tang et al. (EMNLP 2025), DOI 10.18653/v1/2025.emnlp-main.83: citation bibliographic match must be distinguished from claim-to-passage support; the paper shows that even real-looking references may be fabricated. |
| - Gottweis et al., Nature (2026), DOI 10.1038/s41586-026-10644-y: scientific Co-Scientist's multi-agent refinement/evaluation and human-in-the-loop safeguards; automated Elo is not ground truth. |
| - Mitchener et al. (2025), DOI 10.48550/arXiv.2511.02824: Kosmos scientific workflow and provenance. |
| |
| ## Implemented now |
| |
| - frontierlab/claim_verification.py enforces real source-ID lookup and exact passage-in-abstract checks. Without the right exact source snippet, it rejects affirmative decisions (defaults to INSUFFICIENT). |
| - Test claims are labeled by a human author in frontierlab/claim_benchmark.py and are **entirely fabricated**. The two deliberately erroneous predictions demonstrate why source-ID checking and passage matching alone are not entailment. |
| - Against the hand-authored ten-case fixture, the test has 9/10 known IDs, 7/10 exact passage anchors, and 8/10 agreement with synthetic labels. These are code coverage / hand-authored agreement indicators, **not AI accuracy, scientific validation or independent annotation**. |
| - Conflict transparency is measured as recall against explicitly annotated contradiction pairs, never treated as 100% when no human gold labels exist. A disagreement across different assay conditions is a test case for CONDITIONAL tension, not proof of a logical contradiction. |
| - Public HF app will show the audit fixture only **after the HF Docker Space is rebuilt**. GitHub updates by themselves do not change a running Space image. |
| |
| ## Next: genuine Meta Muse Glimmer sample via NVIDIA (no Meta Platforms API) |
| |
| NVIDIA's official listing provides an endpoint for meta/muse-glimmer-30b: |
| https://build.nvidia.com/meta/muse-glimmer-30b |
| |
| Use the one-request, synthetic-only GitHub Actions workflow: |
| https://github.com/Jaycee871/FrontierLab-Meta-Hackathon-2026/actions/workflows/nvidia-muse-trial.yml |
| |
| It uses the existing repository secret **NVIDIA_API_KEY**. The workflow must be manually triggered. It sends one short FICTIONAL prompt to the fixed HTTPS endpoint https://integrate.api.nvidia.com/v1/chat/completions with max_tokens=2048, low reasoning effort, model-recommended sampling and no retries. No Nature+ paid full texts, HIPAA/PII, or actual secret values are sent in the prompt. There is no Meta Platforms API call. NVIDIA's developer free prototype tier may have quotas, changing terms and account-specific restrictions; verify your account does not permit overage charges before running. |
|
|
| A successful smoke test requires nonempty final text, a reported Muse Glimmer model identity and finish_reason=stop. Its JSON receipt records UTC start time, elapsed seconds, input/output SHA-256 hashes, the output excerpt, token usage and filtered GitHub revision/run identifiers. Truncated answers or missing identities are recorded as unsuccessful; arbitrary exception text, credentials and private reasoning are not logged. The receipt is a reproducibility record, not a cryptographic attestation by the provider. Even REAL_INFERENCE_RETURNED does **not** demonstrate research accuracy or hackathon compliance. A more complete benchmark must then perform blinded human-supported claim entailment on open-licensed source excerpts. |
| |
| There is intentionally no automatic secret-enabled NVIDIA job on push/PR. Any possible trial error is redacted to prevent leaking Authorization headers. |
| |
| Do NOT upload API keys or protected scientific PDFs into a public Hugging Face Space. |
| |
