ShayPachi commited on
Commit
7fcb981
路
verified 路
1 Parent(s): fa2c26f

index.html

Browse files
Files changed (1) hide show
  1. index.html +0 -65
index.html DELETED
@@ -1,65 +0,0 @@
1
- <!doctype html>
2
- <html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
3
- <title>Catch the AI Lying: an audited pass/fail harness for one small free model</title>
4
- <link rel="preconnect" href="https://fonts.googleapis.com">
5
- <link href="https://fonts.googleapis.com/css2?family=Archivo:wght@400;700&family=Newsreader:wght@400&display=swap" rel="stylesheet">
6
- <style>
7
- :root{--ground:#FAF7F0;--panel:#EDE6DA;--ink:#201E1D;--muted:#666666;--accent:#AE1800;--line:#B9B7B2}
8
- *{box-sizing:border-box}
9
- body{margin:0;background:var(--ground);color:var(--ink);font-family:Archivo,Helvetica,Arial,sans-serif;line-height:1.5}
10
- main{max-width:920px;margin:0 auto;padding:56px 24px 64px}
11
- .eyebrow{font-size:12px;font-weight:700;letter-spacing:.22em;text-transform:uppercase;color:var(--accent)}
12
- h1{font-family:Newsreader,Georgia,serif;font-weight:400;font-size:42px;line-height:1.12;margin:10px 0 14px}
13
- .by{color:var(--muted);font-size:14px;margin-bottom:32px}
14
- h2{font-family:Newsreader,Georgia,serif;font-weight:400;font-size:25px;margin:34px 0 8px}
15
- .panel{background:var(--panel);border-top:3px solid var(--ink);padding:16px 20px;white-space:pre-wrap;
16
- font-family:ui-monospace,Menlo,monospace;font-size:15px;line-height:1.7}
17
- .prose{background:var(--panel);border-top:3px solid var(--ink);padding:16px 20px;white-space:pre-wrap}
18
- .note{color:var(--muted);font-size:13px;margin:0 0 8px}
19
- .chart{border-top:3px solid var(--ink);padding-top:8px}
20
- .grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(260px,1fr));gap:14px;margin-top:10px}
21
- footer{margin-top:28px;padding-top:14px;border-top:1px solid var(--line);color:var(--muted);font-size:13px}
22
- footer a{color:#1D5FA8}
23
- @media(max-width:640px){h1{font-size:30px}}
24
- </style></head>
25
- <body><main>
26
- <div class="eyebrow">MIS 752 路 Lab 7</div>
27
- <h1>Catch the AI Lying: an audited pass/fail harness for one small free model</h1>
28
- <div class="by">Ten clinical gold questions 路 exact match, contains check and two judge passes 路 saved notebook results</div>
29
- <h2>The 30-second read</h2><div class="panel">THE GOLD STANDARD
30
- items : 10 self-contained clinical questions
31
- file : data/gold_qa.csv
32
-
33
- ACCURACY ON THE SAME 10 ITEMS, THREE RULERS
34
- exact match : 9/10 = 90%
35
- contains gold : 8/10 = 80%
36
- judge, pass A : 7/10 = 70%
37
- judge, pass B : 6/10 = 60%
38
-
39
- IS THE JUDGE ITSELF RELIABLE?
40
- judge vs itself : agreement 60%, kappa 0.20
41
- exact vs judge A : agreement 80%, kappa 0.4444
42
- judge failures : pass A 2/10, pass B 4/10
43
- API cost reported: $0
44
-
45
- IMPORTANT PROVENANCE
46
- This run includes API errors (including rate limits).
47
- UNPARSEABLE means no valid judgment; it is not a clinical FAIL.
48
- Saved notebook output was used to reconstruct these tables.
49
- Full original answer text and raw judge reasons were not recoverable.
50
- This is not a clinical validation.</div><h2>Your judge contradicted itself</h2><p class="note">Judge pass A and pass B agreed on six of ten items. Four changed verdict, frequently because of unparseable API calls; this is not evidence of four contradictory clinical decisions.</p><h2>The two rulers disagree</h2><p class="note">Exact match passed nine answers; judge A passed seven. The 20-point gap includes two unparseable judge calls, not two confirmed clinical errors.</p><h2>The pass/fail table</h2><div class="panel" style="overflow-x:auto"> id type gold answer exact_verdict judge_a score_a judge_b score_b judge_repeated scorers_agree
51
- icd10_t2dm code_lookup E11.9 E11.9 PASS UNPARSEABLE PASS 10.0 False False
52
- icd10_ami code_lookup I21.9 I21.9 PASS PASS 10.0 UNPARSEABLE False True
53
- snomed_t2dm code_lookup 44054006 44054006 PASS PASS 10.0 PASS 10.0 True True
54
- acetaminophen_max_g numeric 4 4 PASS PASS 10.0 PASS 10.0 True True
55
- a1c_8_eag_mgdl numeric 183 183 PASS PASS 10.0 PASS 10.0 True True
56
- hypo_level1_mgdl numeric 70 FAIL FAIL 0.0 UNPARSEABLE False True
57
- hipaa_safe_harbor numeric 18 18 PASS PASS 10.0 PASS 10.0 True True
58
- warfarin_reversal_agent short_phrase 4-factor prothrombin complex concentrate Prothrombin complex concentrate PASS UNPARSEABLE UNPARSEABLE True False
59
- sjw_warfarin_yesno yes_no NO NO PASS PASS 10.0 PASS 10.0 True True
60
- atorva80_intensity one_word high high-intensity PASS PASS 10.0 UNPARSEABLE False True</div><h2>Auditing the instrument</h2><div class="panel">exact_verdict PASS FAIL UNPARSEABLE
61
- FAIL 0 1 0
62
- PASS 7 0 2</div><h2>Week 4 re-scoring</h2><div class="prose">No Week 4 answers.csv was available in the uploaded notebook results. No re-scoring evidence is included.</div><h2>How this was measured</h2><div class="prose">Ten gold questions were scored with exact matching and a model judge twice. The notebook records 60% self-agreement (魏 = 0.20) and 80% exact-vs-judge agreement (魏 = 0.4444). Rate-limit failures make the pass percentages incomplete. This is a teaching pilot, not a clinical evaluation. ChatGPT provided partial assistance in preparing the published files.</div>
63
- <footer>Course: <a href="https://huggingface.co/MIS-752">MIS 752, UNLV</a> 路
64
- Built in Google Colab on 2026-10-12.</footer>
65
- </main></body></html>