index.html
Browse files- index.html +0 -65
index.html
DELETED
|
@@ -1,65 +0,0 @@
|
|
| 1 |
-
<!doctype html>
|
| 2 |
-
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
| 3 |
-
<title>Catch the AI Lying: an audited pass/fail harness for one small free model</title>
|
| 4 |
-
<link rel="preconnect" href="https://fonts.googleapis.com">
|
| 5 |
-
<link href="https://fonts.googleapis.com/css2?family=Archivo:wght@400;700&family=Newsreader:wght@400&display=swap" rel="stylesheet">
|
| 6 |
-
<style>
|
| 7 |
-
:root{--ground:#FAF7F0;--panel:#EDE6DA;--ink:#201E1D;--muted:#666666;--accent:#AE1800;--line:#B9B7B2}
|
| 8 |
-
*{box-sizing:border-box}
|
| 9 |
-
body{margin:0;background:var(--ground);color:var(--ink);font-family:Archivo,Helvetica,Arial,sans-serif;line-height:1.5}
|
| 10 |
-
main{max-width:920px;margin:0 auto;padding:56px 24px 64px}
|
| 11 |
-
.eyebrow{font-size:12px;font-weight:700;letter-spacing:.22em;text-transform:uppercase;color:var(--accent)}
|
| 12 |
-
h1{font-family:Newsreader,Georgia,serif;font-weight:400;font-size:42px;line-height:1.12;margin:10px 0 14px}
|
| 13 |
-
.by{color:var(--muted);font-size:14px;margin-bottom:32px}
|
| 14 |
-
h2{font-family:Newsreader,Georgia,serif;font-weight:400;font-size:25px;margin:34px 0 8px}
|
| 15 |
-
.panel{background:var(--panel);border-top:3px solid var(--ink);padding:16px 20px;white-space:pre-wrap;
|
| 16 |
-
font-family:ui-monospace,Menlo,monospace;font-size:15px;line-height:1.7}
|
| 17 |
-
.prose{background:var(--panel);border-top:3px solid var(--ink);padding:16px 20px;white-space:pre-wrap}
|
| 18 |
-
.note{color:var(--muted);font-size:13px;margin:0 0 8px}
|
| 19 |
-
.chart{border-top:3px solid var(--ink);padding-top:8px}
|
| 20 |
-
.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:14px;margin-top:10px}
|
| 21 |
-
footer{margin-top:28px;padding-top:14px;border-top:1px solid var(--line);color:var(--muted);font-size:13px}
|
| 22 |
-
footer a{color:#1D5FA8}
|
| 23 |
-
@media(max-width:640px){h1{font-size:30px}}
|
| 24 |
-
</style></head>
|
| 25 |
-
<body><main>
|
| 26 |
-
<div class="eyebrow">MIS 752 路 Lab 7</div>
|
| 27 |
-
<h1>Catch the AI Lying: an audited pass/fail harness for one small free model</h1>
|
| 28 |
-
<div class="by">Ten clinical gold questions 路 exact match, contains check and two judge passes 路 saved notebook results</div>
|
| 29 |
-
<h2>The 30-second read</h2><div class="panel">THE GOLD STANDARD
|
| 30 |
-
items : 10 self-contained clinical questions
|
| 31 |
-
file : data/gold_qa.csv
|
| 32 |
-
|
| 33 |
-
ACCURACY ON THE SAME 10 ITEMS, THREE RULERS
|
| 34 |
-
exact match : 9/10 = 90%
|
| 35 |
-
contains gold : 8/10 = 80%
|
| 36 |
-
judge, pass A : 7/10 = 70%
|
| 37 |
-
judge, pass B : 6/10 = 60%
|
| 38 |
-
|
| 39 |
-
IS THE JUDGE ITSELF RELIABLE?
|
| 40 |
-
judge vs itself : agreement 60%, kappa 0.20
|
| 41 |
-
exact vs judge A : agreement 80%, kappa 0.4444
|
| 42 |
-
judge failures : pass A 2/10, pass B 4/10
|
| 43 |
-
API cost reported: $0
|
| 44 |
-
|
| 45 |
-
IMPORTANT PROVENANCE
|
| 46 |
-
This run includes API errors (including rate limits).
|
| 47 |
-
UNPARSEABLE means no valid judgment; it is not a clinical FAIL.
|
| 48 |
-
Saved notebook output was used to reconstruct these tables.
|
| 49 |
-
Full original answer text and raw judge reasons were not recoverable.
|
| 50 |
-
This is not a clinical validation.</div><h2>Your judge contradicted itself</h2><p class="note">Judge pass A and pass B agreed on six of ten items. Four changed verdict, frequently because of unparseable API calls; this is not evidence of four contradictory clinical decisions.</p><h2>The two rulers disagree</h2><p class="note">Exact match passed nine answers; judge A passed seven. The 20-point gap includes two unparseable judge calls, not two confirmed clinical errors.</p><h2>The pass/fail table</h2><div class="panel" style="overflow-x:auto"> id type gold answer exact_verdict judge_a score_a judge_b score_b judge_repeated scorers_agree
|
| 51 |
-
icd10_t2dm code_lookup E11.9 E11.9 PASS UNPARSEABLE PASS 10.0 False False
|
| 52 |
-
icd10_ami code_lookup I21.9 I21.9 PASS PASS 10.0 UNPARSEABLE False True
|
| 53 |
-
snomed_t2dm code_lookup 44054006 44054006 PASS PASS 10.0 PASS 10.0 True True
|
| 54 |
-
acetaminophen_max_g numeric 4 4 PASS PASS 10.0 PASS 10.0 True True
|
| 55 |
-
a1c_8_eag_mgdl numeric 183 183 PASS PASS 10.0 PASS 10.0 True True
|
| 56 |
-
hypo_level1_mgdl numeric 70 FAIL FAIL 0.0 UNPARSEABLE False True
|
| 57 |
-
hipaa_safe_harbor numeric 18 18 PASS PASS 10.0 PASS 10.0 True True
|
| 58 |
-
warfarin_reversal_agent short_phrase 4-factor prothrombin complex concentrate Prothrombin complex concentrate PASS UNPARSEABLE UNPARSEABLE True False
|
| 59 |
-
sjw_warfarin_yesno yes_no NO NO PASS PASS 10.0 PASS 10.0 True True
|
| 60 |
-
atorva80_intensity one_word high high-intensity PASS PASS 10.0 UNPARSEABLE False True</div><h2>Auditing the instrument</h2><div class="panel">exact_verdict PASS FAIL UNPARSEABLE
|
| 61 |
-
FAIL 0 1 0
|
| 62 |
-
PASS 7 0 2</div><h2>Week 4 re-scoring</h2><div class="prose">No Week 4 answers.csv was available in the uploaded notebook results. No re-scoring evidence is included.</div><h2>How this was measured</h2><div class="prose">Ten gold questions were scored with exact matching and a model judge twice. The notebook records 60% self-agreement (魏 = 0.20) and 80% exact-vs-judge agreement (魏 = 0.4444). Rate-limit failures make the pass percentages incomplete. This is a teaching pilot, not a clinical evaluation. ChatGPT provided partial assistance in preparing the published files.</div>
|
| 63 |
-
<footer>Course: <a href="https://huggingface.co/MIS-752">MIS 752, UNLV</a> 路
|
| 64 |
-
Built in Google Colab on 2026-10-12.</footer>
|
| 65 |
-
</main></body></html>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|