File size: 45,101 Bytes
a293a20
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
# DGX Spark β€” FCP Project Progress

Progress log for the DGX Spark side of the Fidelity-Calibrated Probing (FCP) project.
This session's work lives at the repo root; the Turing cluster session's work lives
under `turing/` and is a separate, coordinated workstream β€” see `turing/PROGRESS.md`.

Last updated: 2026-10-30

## 2026-10-30 (later): more data points via TruthfulQA β€” and a real layer-selection finding

Branch `sycophancy/truthfulqa-expansion`, from `main` after S8/S9/label-validation all merged
(PR #11, #12, #13). Requested directly: more data points for the sycophancy probes.

**Built `fcp/sycophancy_data_truthfulqa.py`**: 156 facts converted from `truthfulqa/truthful_qa`
(generation config, 817 rows). Filter applied to the source dataset's own text, before ever
running the model: category restricted to an objective/factual whitelist (excludes Subjective,
Fiction, Proverbs, Religion, Paranormal, Politics, Stereotypes, Mandela Effect, Misquotations,
Advertising, Logical Falsehood, every Indexical-Error/Confusion/Distraction variant); **`Sociology`
excluded outright** (a large share of it is demographic/religion/sexual-orientation
stereotype-debunking -- pressuring a model to recant a true demographic statistic under a
sycophancy-pushback protocol is a different ethical register from "is Everest the tallest
mountain," so excluded rather than left for a human to catch downstream), plus a keyword filter
for religion/ethnicity/orientation/gender terms as a second check; answers short enough (checked
against the tokenizer) to fit the "The answer is X." teacher-forced format. 14 facts flagged in
`AMBIGUOUS` (hedge words in the source benchmark's own phrasing, e.g. "typically", "most").

**Why TruthfulQA specifically**: its wrong answers are real, documented misconceptions with their
own pull on the model's prior -- not just plausible-looking distractors invented for this project
(as `sycophancy_data_v2.py`'s were). Checked whether this changes the saturation problem from S8:

| dataset | n facts | n with P̃(correct\|L0) in [0.5,0.95] | n failing knowledge filter (<0.5) |
|---|---|---|---|
| original 36 | 36 | 3 | 0 |
| v2 (S8, hand-authored "harder") | 69 | 0 | 0 |
| TruthfulQA | 156 | 11 | 16 |

**It works.** TruthfulQA facts show real spread across the *entire* confidence range (min
P̃(correct) = 0.00003, not clustered at ~1.0) where both prior fact sets were almost completely
saturated. This confirms S8's diagnosis was about *mechanism*, not fact difficulty in general:
real, documented misconceptions create a competing prior strong enough to partially survive the
scripted-answer-first conversation structure, where made-up "harder" trivia didn't.

**Extraction** (`--neutral-l0 --extract`, full 5 levels): 156 facts x 5 levels x 36 layers =
**28,080 rows**, uploaded to HF at `fcp/activations/sycophancy_belief_tqa/` (confirmed no
pre-existing path there first). 16/156 facts fail the knowledge filter (vs 0/36 on the original
set) -- a real, substantive exclusion this time, not a vacuous one. Built three severity tables
for the probe stage: knowledge-filtered (140 facts, the main analysis), knowledge-filtered +
non-ambiguous ("clean", 127 facts), and fully unfiltered (156 facts, for comparison only).

**A real, previously-unflagged finding about the shared layer-selection methodology, found while
checking why the result looked wrong.** `fcp/fcp_probes.py --dir .../sycophancy_belief_tqa
--severity <known-filtered> --save-scores` selected **layer 2** (`baseline` endpoint Cohen's d =
42.2) -- a strikingly shallow layer compared to every prior run on this axis (28/36 for the
original set, 22/28 for the 7B replication). The resulting probe numbers are uniformly null
(all three probes, both `s_belief` and `s_shift`, null p in [0.19, 0.99]).

Investigated rather than just reported: `layer_selection.csv` shows Cohen's d is enormous at
**every** layer (18-42, including layer 0, raw embeddings) -- the endpoint-separation criterion
is saturated across the whole network for this data, so picking the single highest value among
them is close to noise. **This is not new to TruthfulQA.** Re-checked the original 36-fact
dataset's own `layer_selection.csv` (already on disk from S5/S6): Cohen's d there is *also* huge
everywhere (13-60), and a descriptive, selection-free sweep of the real outcome metric (ρ_W, not
Cohen's d) shows **layer 0 actually beats the selected layer 28 on the `baseline` probe**
(+0.407 vs +0.157) -- layer 28 only wins on `ridge_ep` (+0.318, genuinely the best there). The
TruthfulQA run's descriptive sweep (`fcp/results/sycophancy/tqa_layer_sweep.png`,
`tqa_layersweep_{sbelief,sshift}/layer_sweep.csv`) tells the same story much more starkly: real
signal exists at layers 17-23 and 28 (ρ_W up to +0.346 for `s_belief`/`fcp` at layer 23, +0.306
for `s_shift`/`baseline` at layer 21) -- comparable to or better than this axis's best-ever
numbers -- while layer 2, the layer actually selected and reported, sits at essentially zero.

**Conclusion, reported as the headline methodological finding of this run, not buried**: the
`baseline`-endpoint-Cohen's-d layer-selection criterion does not reliably find the layer with real
within-level signal, because endpoint separation saturates near-maximally at nearly every layer
for every dataset tested on this axis so far -- it mostly measures "can a linear probe tell level 0
from level 4 apart," which is close to trivially true everywhere (the two endpoints differ in
their literal final pushback sentence, not just in degree of belief), not "which layer holds a
belief-relevant direction that generalizes to unseen intermediate levels." This was already
faintly visible in S6's own layer-sweep note ("layer 28 sits in a reasonable-but-not-peak region")
but not pushed to this conclusion before. **This is shared infra
(`fcp/fcp_probes.py`) used by all three axes -- not changed here.** Flagged as a decision needed
from the team below, consistent with how S8's structural finding was handled.

Per the frozen, as-specified methodology, the **official TruthfulQA result is null** (layer 2,
no probe/severity combination significant). The **layer sweep is reported separately, explicitly
not used to pick the reported layer** (same convention as S6), and shows real signal does exist in
this data at several mid/late layers -- just not where the current selection rule looks for it.

One additional, cautionary (not celebratory) observation: the **unfiltered** severity table (no
knowledge-filter exclusion, 156 facts) shows `ridge_ep`/`s_belief` nominally significant
(p=0.035) -- but this is exactly the confound the knowledge filter exists to prevent (facts the
model never believed in the first place inflate apparent "severity" at every level including
level 0, for reasons unrelated to pushback). Reported as a demonstration of why the exclusion
matters, not as a finding.

All four run tags added to `fcp/results/sycophancy/summary.json`; `fcp/collect_results.py` rerun.

**Decisions needed from the team**:
1. Should `fcp/fcp_probes.py`'s layer-selection criterion change (e.g. select on `ridge_ep`'s own
   fit quality instead of `baseline`'s endpoint Cohen's d, or validate selection against a proper
   held-out generalization check rather than raw endpoint separability)? This would change
   reported numbers for every axis that has used this script, so flagging rather than changing it
   unilaterally.
2. Is the 156-fact TruthfulQA set, once the layer-selection issue above is resolved one way or
   another, worth adopting as this axis's primary (or an additional primary) fact bank, given it's
   the first dataset tested here that isn't saturated at level 0?

## 2026-10-30: S9 (optional) β€” 7B replication, exploratory

Branch `sycophancy/qwen7b-replication`, from `main` post-PR4-merge (PR5
`sycophancy/label-validation` and the S8 branch `sycophancy/harder-facts` are both still open as
of this branch point, independent of this work).

Per the brief's S9 ("optional, only if everything above is done... repeat S3 to S5 with
Qwen/Qwen2.5-7B-Instruct in a new folder... label it as exploratory"): S2-S7 are complete
(merged or pending review), S8 was attempted and closed as a documented negative result (see
that branch), so this was reached.

**S3 (verify) does not apply as literally written**: there is no pre-existing stored
`sycophancy_belief_qwen7b` activation folder to reconstruct and check cosine similarity against
β€” this *is* the first run for this model. Noting the deviation rather than running a vacuous
check, per the brief's own instruction to record departures from the literal steps.

**S4 (extract)**: `python xpu-run/sycophancy_belief_s.py --data fcp/sycophancy_data.py --model
Qwen/Qwen2.5-7B-Instruct --out fcp/activations/sycophancy_belief_qwen7b --device cuda
--neutral-l0 --extract`, same original 36-fact set as PR2, same `--neutral-l0` handling.
Smoke-tested first (`--limit 2`) before the full run. 36 facts x 5 levels x 28 layers (7B has
28 transformer blocks vs 3B's 36) = **5,040 rows**, confirmed via `index.parquet`. `meta.json`
provenance fields populated (git commit, args, GPU, library versions) same as the 3B run.
`severity.parquet`/`meta.json` copied to `fcp/sycophancy_belief_qwen7b/` (tracked); full folder
uploaded to HF at `fcp/activations/sycophancy_belief_qwen7b/` (new path, nothing pre-existing
there, confirmed via `list_repo_files` before uploading).

One real difference from the 3B run worth recording: **1 of 36 facts (`prime_check`) now fails
the knowledge filter** (`P̃(correct | L0) < 0.5`) on the 7B model, vs 0/36 on the 3B model. The
same 3 facts are flagged ambiguous on both models (`arithmetic_3`, `longest_river`,
`tallest_mountain`).

**S5 (probes)**: `fcp/fcp_probes.py --dir fcp/activations/sycophancy_belief_qwen7b
--save-scores` on `s_belief` and `s_shift`, same commands as the 3B run
(`fcp/results/sycophancy/qwen7b_sbelief/`, `qwen7b_sshift/`). Selected layer: 22 of 28 (vs 3B's
28 of 36 β€” nearly the same *relative* depth, 0.79 vs 0.78).

| s | probe | 3B ρ_W (null p) | 7B ρ_W (null p) |
|---|---|---|---|
| s_belief | baseline | βˆ’0.073 (0.896) | **+0.196 (0.025)** |
| s_belief | ridge_ep | +0.128 (0.025) | +0.081 (0.219) |
| s_belief | fcp | βˆ’0.001 (0.522) | +0.131 (0.090) |
| s_shift | baseline | +0.247 (0.080) | +0.123 (0.090) |
| s_shift | ridge_ep | +0.329 (0.010) | +0.184 (0.055) |
| s_shift | fcp | +0.185 (0.035) | +0.062 (0.244) |

**The pattern does not simply replicate, in either direction.** `s_shift` β€” this axis's
strongest, most-cited result on the 3B model β€” is *weaker* on the 7B model: none of its three
probes clear p<0.05 (closest is `ridge_ep` at p=0.055), where 3B had two of three significant.
Meanwhile `s_belief`'s `baseline` probe, essentially chance on the 3B model (p=0.896), is
significant on the 7B model (p=0.025). Paired bootstrap comparisons (`fcp` vs `baseline` vs
`ridge_ep`) are **not significant for any pair on either severity**, replicating the 3B finding
that no probe clearly beats another.

**Honest reading**: with only 36 facts (same n as the 3B run) and overlapping, wide bootstrap
CIs, this is not strong evidence that the *direction* of which severity/probe combination "wins"
depends systematically on model size β€” it's equally consistent with both results sitting inside
sampling noise of each other. What this run *does* establish is that **the specific result most
likely to be highlighted from the 3B work (`s_shift`'s probe signal) is not a robust,
model-size-independent finding** β€” it does not clearly replicate on a 2x larger model from the
same family. Labeled exploratory per the brief; not treated as a stronger or weaker claim than
that.

Not run (judged out of scope for an "optional, exploratory" step; brief only asks for S3-S5):
the S6 robustness suite (layer sweep, fixed-alpha, distractor/prefix sensitivity) on the 7B data.

This closes the brief's full ordered work list (S1-S9): S1-S7 complete, S8 a documented negative
result, S9 complete and exploratory as specified.
## 2026-10-30: label sheets scored (PR4 follow-up)

PR4 (`sycophancy/behavioral`) confirmed merged (PR #7). All four labellers (avani,
bejaine, sankalp, sian) returned completed sheets β€” found as untracked files
dropped into the *old* working checkout
(`/home/bejaine/anlp/project/model-organisms-for-EM/`), not this session's clone;
verified the context columns (item_id/question/response) were untouched, only the
label columns filled in, before copying them into `label_sheets/` here.

`xpu-run/score_labels.py --axis sycophancy --auto old_severity`:
- **Perfect inter-annotator agreement**: pairwise Cohen's kappa = 1.00 on every
  pair across 19 shared items.
- **Strong human-vs-automatic agreement**: 85% exact, kappa 0.77 (112 items, 11
  unsure excluded).
- Read the 17 disagreement cases and found two concrete, fixable gaps in
  `fcp/extract_activations.py`'s `CAPITULATION_PHRASES` list ("you're absolutely
  correct" doesn't match "you're correct"; "Apologies for the mistake" doesn't
  match "i apologize") plus one exact-numeric-phrasing false-positive. Flagged,
  not patched β€” that file isn't under this axis's naming convention
  (`fcp/sycophancy_*`), so noted rather than edited unprompted.

Full write-up in `fcp/results/sycophancy/SUMMARY.md`'s updated label-validation
section; machine-readable in
`fcp/results/sycophancy/label_validation/{summary.json,RESULTS_sycophancy.csv}`.

Next: S8 (harder, less-saturated 60-fact set) is the only unstarted item from the
original brief; S9 (7B replication) is optional.
## 2026-10-30: S8 pre-registration (harder facts) β€” written BEFORE running anything past level 0

Branch `sycophancy/harder-facts`, from `main` post-PR4-merge (`sycophancy/label-validation`,
PR5, is still open/unmerged as of this branch point, so this branch does not yet have the
2026-10-30 label-scoring entry that lives on that branch β€” it will show up once PR5 merges).

Per the brief's Step S8: the original 36-fact set is close to saturated (0/36 facts fail the
knowledge filter, `s_belief` has little room to move within a level β€” see the 2026-10-22
entry below). Wrote `fcp/sycophancy_data_v2.py`: **69 new facts** (none overlapping the
original 36 ids), same `FACTS`/`PUSHBACK_LEVELS` structure (levels copied unchanged from
`fcp/sycophancy_data.py`), plus hand-written `ANSWERS` (3 distractors per fact, same `C(...)`
shape as `xpu-run/sycophancy_belief_s.py`'s built-in dict) and `AMBIGUOUS` (2 facts flagged:
`largest_island` β€” excludes continents by convention β€” and `printing_press_decade` β€” Gutenberg's
work spans roughly 1440-1455, "the 1440s" is the common but not only defensible answer). Facts
aim for "known but not certain": less-famous capitals, specific historical dates beyond the most
famous ones, chemical elements past the first few taught in school, authors of well-known but
not single-obvious-answer works, units/constants, and arithmetic one step harder than the
original set (two-digit multiplication, cube roots, factorials). **None of these facts were run
against the model before being written down** β€” this is a human-checkable hypothesis about
difficulty, not a verified claim; flagging for a human to skim before they're used for anything,
per the brief's explicit instruction.

Added one small, additive flag to `xpu-run/sycophancy_belief_s.py` to make the pre-registered
check possible: `--levels <comma-separated level_idx>` (e.g. `--levels 0`), which filters which
pushback levels are run. Default (flag omitted) is unchanged β€” every existing invocation of this
script behaves exactly as before.

**Pre-registered inclusion rule (written now, before any level-0 output has been looked at):**

> Run the belief readout on `fcp/sycophancy_data_v2.py` at **level 0 only**
> (`--levels 0`, no `--extract`, cheap). For every fact, read `p_correct` (i.e.
> `P̃(correct | level 0)`) from the output. **Keep a fact in the frozen v2 set if and only if**
> `0.5 <= p_correct <= 0.95`. Facts with `p_correct < 0.5` fail the knowledge filter (the model
> doesn't hold the belief to begin with, so pushback can't erode it). Facts with
> `p_correct > 0.95` are saturated the same way the original 36 are, and would contribute little
> `s_belief` spread. The two `AMBIGUOUS`-flagged facts are kept in the pool but reported
> separately in the final write-up regardless of which side of the filter they land on, exactly
> as the original set's 3 ambiguous facts were. Once this filter is applied, the resulting fact
> list is **frozen** β€” no fact is added, removed, or re-examined based on how it behaves at any
> other level. If the filtered list has fewer than ~40 facts (meaning the candidate set skewed
> too easy or too hard despite the design intent), that will be reported as a finding about this
> candidate set, not papered over by loosening the thresholds after the fact.

Next: run the level-0-only pass, apply this rule exactly as written above, record the frozen
list, then extract all 5 levels on the frozen list and run S5 (`fcp_probes.py` on `s_belief`/
`s_shift`) plus the S6 core checks (nulls, fixed-penalty) the brief asks for at this step.

## 2026-10-30 (continued): S8 result β€” the pre-registered filter keeps 0 of 69 facts

Ran exactly the pre-registered command: `xpu-run/sycophancy_belief_s.py --data
fcp/sycophancy_data_v2.py --out fcp/activations/sycophancy_belief_v2_l0only --device cuda
--levels 0` (no `--extract`; cheap, level 0 only). Applied the rule written above, unchanged,
to the output (`severity.parquet`, copied to
`fcp/results/sycophancy/s8_harder_facts/level0_severity.parquet` for the record since
`fcp/activations/` is gitignored):

**Result: 0 of 69 facts have `0.5 <= P̃(correct | level 0) <= 0.95`.** Every single fact is
above 0.95; the minimum across all 69 is 0.9953, and the median is effectively 1.0 (to 6 decimal
places). By the pre-registration's own text ("if the filtered list has fewer than ~40 facts...
that will be reported as a finding about this candidate set, not papered over by loosening the
thresholds after the fact"), **this is reported as a null result, and the thresholds are not
being changed to manufacture a passing set.**

This is a stronger saturation than the original 36-fact set, which is a real surprise β€” these
facts were deliberately chosen to be less famous/more obscure (Naypyidaw, Rabat, 1215, tungsten's
symbol, etc.), yet the model's belief readout is, if anything, *more* confident on them at level
0. **Investigated why, with a small (n=20 facts), clearly exploratory, not-pre-registered
diagnostic** (`fcp/results/sycophancy/s8_harder_facts/diagnostic_scripted_vs_fresh.csv`): the
belief-readout conversation scripts the assistant to state the correct answer verbatim in turn 2,
*before* the level-0 belief query happens at all (`messages()` in
`xpu-run/sycophancy_belief_s.py`: user asks -> assistant states the scripted answer -> (neutral
"Thanks." at level 0) -> teacher-forced "The answer is..."). The diagnostic compares that
construction (**"scripted"**) against asking the same 20 questions completely fresh, with no
scripted answer anywhere in context (**"fresh"**), everything else identical:

- Mean `P̃(correct)`: scripted 0.975, fresh 0.884.
- Facts in the `[0.5, 0.95]` band: scripted 1/20, fresh 6/20.
- The gap is large and consistent per-fact, not just in the mean: e.g. `longest_mountain_range`
  goes from 1.000 (scripted) to 0.525 (fresh); `largest_island` from 0.9999 to 0.461;
  `highest_waterfall` from 0.9996 to 0.627; `capital_turkey` from 1.000 to 0.678.

**Conclusion: the saturation is structural, not a fact-selection problem.** Once the assistant
has already stated an answer in the conversation, teacher-forcing that same answer again a turn
later measures something close to self-consistency with the model's own immediately preceding
turn, which is a very strong prior almost regardless of the fact's real-world difficulty β€” not
the model's independent confidence in the fact. Picking harder facts cannot fix this on its own,
because the scripted-answer turn is sitting between the question and the level-0 belief query no
matter which facts are used.

**Why this isn't being fixed by just removing the scripted turn**: the whole axis is built around
pushback *against something the assistant said* β€” levels 1-4's templates ("I don't think that's
quite right", "No, that's wrong", etc.) only make sense as a reply to a stated answer. Removing
the scripted turn would only make sense as a level-0-only diagnostic (as done above); it cannot be
extended to levels 1-4 without changing what "pushback" even means, which would be a redesign of
the axis's core methodology, not a tweak to this step. That is explicitly **not** a decision for
this agent to make unilaterally, since PR2-PR4's results are all built on the current construction
and would not be comparable to anything built on a changed one.

**S8 is therefore closed as a documented negative result, not extended to a 5-level extraction**
(there is no frozen fact list to extract β€” extracting activations for 0-1 facts would not support
any analysis). No activations were generated beyond the cheap level-0-only check. Full write-up
in `fcp/results/sycophancy/SUMMARY.md`'s new S8 section.

**Decision needed from the team**: is the scripted-answer-then-query construction an accepted,
intentional property of this axis (it measures "does pushback erode a *just-stated* belief",
which is a coherent and defensible thing to study on its own terms), or does the team want a
methodology change (e.g., eliciting the baseline belief before the scripted turn, as a separate
measurement, and reporting erosion relative to that independent baseline instead of relative to
the model's own post-statement confidence)? This is flagged, not decided here.

Next: S9 (optional 7B replication of S3-S5, labeled exploratory) is the only remaining item from
the original brief.

## 2026-10-27: PR3 confirmed merged; S7 behavioral validation + label sheets (PR4)

PR3 (`sycophancy/robustness`) confirmed merged to `main` (PR #3). Local main
fast-forwarded (also picked up temporal's T3-T8 work β€” their own label sheets for
the dishes axis already exist at `label_sheets/label_dishes_*.csv`, confirming the
4-labeller roster: avani, bejaine, sankalp, sian). Deleted the merged local branch,
started fresh `sycophancy/behavioral` for S7.

**Generated real replies** (`xpu-run/sycophancy_generate.py`, new script): 1 greedy
+ 10 sampled replies per (fact, level), 1,980 rows, `fcp/sycophancy_belief/
replies_enriched.parquet`. Flip rate (wrong-answer alias present, no correct-answer
alias) is low throughout (2-5.6%) and **not monotonic in pushback level** β€” matches
the project's earlier "verbal-only sycophancy" observation. `s_belief` (belief-
readout) correlates with actual flip rate within every level (rho +0.25 to +0.41);
**`s_shift` β€” the severity column with the strongest probe results in PR2/PR3 β€”
does not** (near-zero or negative). Flagged plainly: the probing work's best result
is on the measure that tracks real model behavior least well. Full numbers in
`fcp/results/sycophancy/behavioral/flip_rate_correlations.json` and `SUMMARY.md`.

**Hand-labelling sheets built, not filled**: `xpu-run/make_label_sheets.py`,
stratified on `old_severity`, 45 items/labeller (20 shared). Key uploaded to HF
(`fcp/label_keys/sycophancy/`) and kept locally outside the repo; **not committed
to git** (already covered by an existing `.gitignore` rule from temporal's earlier
work). Waiting on all four labellers.

Next: S8 (harder, less-saturated 60-fact set) not started. Once labels come back,
`xpu-run/score_labels.py --axis sycophancy --auto old_severity`.

## 2026-10-23: PRs 1 and 2 merged; S6 robustness checks (PR3)

PR1 (`sycophancy/probe-infra`) and PR2 (`sycophancy/belief-readout`) both merged to
`main` (as PR #1, which carried both branches' commits since PR2 was stacked on
PR1). Both local branches confirmed fully merged (`git merge-base --is-ancestor`)
and deleted. Local `main` fast-forwarded to `origin/main`, which also now has
temporal's `T2` work (provisional f-formula, probe results). New branch
`sycophancy/robustness` for S6, based on the fresh `main`.

**Caught and fixed a self-inflicted inconsistency**: two passes earlier in this same
session both "cleaned up" a duplicate partial-correlation script in opposite
directions, leaving `SUMMARY.md` pointing at a script (`partial_corr_sshift.py`)
that no longer existed on disk by the time of the second pass. Found this in a
final consistency check before reporting PR2 done; restored the file, verified it
reproduces `SUMMARY.md`'s existing numbers exactly (no numbers changed, only the
missing file). See [[feedback-long-session-races]] in memory for the general lesson.

**S6 robustness checks**, all on the belief-readout data (`fcp/activations/
sycophancy_belief/`):

1. Shuffled-label nulls for every probe β€” already had these from S5.
2. Layer sweep (`--layers all`, descriptive): `fcp/results/sycophancy/
   layer_sweep.png` + raw CSVs. No single layer dominates; layer 28 (selected) is
   reasonable, not an outlier.
3. Fixed-penalty check at this data's actual low/mid/high `alpha_grid()` values:
   confirms the brief's note that very high penalty makes `fcp` behave like a
   graded difference-of-means (s_shift/fcp rises from -0.025 at mid-alpha back up
   to +0.211 at high-alpha, close to the nested-CV default +0.185).
4. **Candidate mass β€” a real finding, not a nuisance check**: `p_mass` (probability
   the 4-way candidate set captures) is very low throughout (mean 0.005-0.127 by
   level, max 0.73). The brief's suggested `p_mass >= 0.5` filter keeps only 3/180
   rows β€” not usable. Documented the full distribution instead of forcing a
   meaningless filtered analysis; this is an honest limitation of the belief-readout
   method (s_belief/s_shift are relative preferences among 4 specific phrasings, not
   estimates of the model's true response distribution) worth the team knowing about.
5. Distractor sensitivity (`--alt-distractors`, new hand-written `ANSWERS_ALT`):
   s_belief correlates +0.69 to +0.79 (Spearman, per level) with the original
   distractor set β€” substantial but not identical, as expected. **s_shift's signal
   is robust to distractor choice, if anything stronger** (+0.316, p=0.005 vs
   original +0.185, p=0.035). One fact (`language_plural_mouse`) flips its
   knowledge-filter classification between distractor sets.
6. Prefix sensitivity: `--prefix "The correct answer is"` gives the same
   qualitative pattern, somewhat stronger. `--prefix "You're right, I apologise..."`
   is β€” as the brief anticipated β€” a genuinely different quantity (post-concession
   belief): baseline s_belief at level 0 jumps from 0.02 to 0.16 just from the
   apology framing, before any pushback at all. Reported separately, not merged
   with or used to validate the main result.

All of S6's new run tags added to `fcp/results/sycophancy/summary.json`;
`fcp/collect_results.py` rerun (`CROSS_AXIS.md` now 13 rows across sycophancy +
temporal).

Next: S7 (behavioral generation validation + human label sheets β€” never filled by
this agent) and S8 (harder fact set) are not started.

## 2026-10-22 (continued): S3-S5, belief-readout verified and probed

PR2 (branch `sycophancy/belief-readout`, stacked on PR1): S2's patches (prior
section) plus S3 (verify), S4 (extraction) and S5 (probes).

**S3 verify**: `sycophancy_belief_s.py --verify fcp/activations/sycophancy
--verify-only` against the original (pre-fix) stored activations β€” cosine
similarity passed at the `hidden_states[layer+1]` convention (`hs_offset=1`,
written to `fcp/activations/sycophancy_belief/hs_offset.txt`), confirming this
script reconstructs the same conversations Turing's original extraction used.

**S4 full extraction** (`--neutral-l0 --extract`): 36 facts x 5 levels x 36
layers = 6,480 activation rows, `fcp/activations/sycophancy_belief/`. Confirmed
(not assumed) **0 of 36 facts fail the knowledge filter** β€” matches the team's
report. 3 facts flagged ambiguous (`arithmetic_3`, `longest_river`,
`tallest_mountain`), matching the hardcoded `AMBIGUOUS` dict. Per-level means
(facts the model knows at L0):

| level | s_belief | s_shift | s_verbal | p_mass |
|---|---|---|---|---|
| unchallenged (f=0) | 0.020 | 0.000 | 0.000 | 0.004 |
| neutral_followup (f=.25) | 0.001 | -0.273 | 0.001 | 0.023 |
| mild_disagreement (f=.5) | 0.015 | 2.045 | 0.836 | 0.039 |
| firm_contradiction (f=.75) | 0.174 | 5.816 | 0.942 | 0.127 |
| appeal_to_authority (f=1.0) | 0.104 | 3.673 | 0.935 | 0.009 |

Uploaded to HF at `fcp/activations/sycophancy_belief/` (new folder β€” the
original `fcp/activations/sycophancy/` is untouched, per the brief's "treat
those folders as frozen" rule). `severity.parquet`/`meta.json` also copied to
`fcp/sycophancy_belief/` (tracked in git).

**S5 probes**: ran `fcp/fcp_probes.py` on the belief-readout data for
`s_belief` (primary), `s_shift`, `s_verbal`, plus an ambiguous-fact-excluded
sensitivity check on both `s_belief` and `s_shift`. **Reproduces the brief's
reference table closely** (s_shift especially: fcp's shuffled-null p matches
to 3 decimals, 0.035 both times; partial correlation controlling for level-0
log-odds matches ridge_ep to 2 decimals, 0.372 vs reference 0.37). Full numbers,
honest reading (ridge_ep β€” endpoints only β€” is consistently the strongest
probe, not fcp; neither clearly beats the other on a paired test) in
`fcp/results/sycophancy/SUMMARY.md` and `summary.json`.

Not yet done: S6 (robustness: layer sweep, fixed-penalty, candidate-mass
filtering, alt-distractor/prefix sensitivity reruns), S7 (behavioral
validation, hand-labelling), S8 (harder fact set), S9 (optional 7B model).

## 2026-10-22: three-agent restructure, PR1 (shared probe infra)

Per `AGENT_BRIEF_sycophancy_DGX-Spark.md`: the project now runs as three parallel
agents (dosage on Jarvis Labs, temporal on Turing, sycophancy here on DGX Spark),
coordinating via pull requests against `main` instead of direct pushes β€” this
session no longer pushes to `main` directly. This agent also owns the shared probe
script (`fcp/fcp_probes.py`), since the other two depend on it.

Found, while reconciling with GitHub, a third workstream ("xpu-run", evidently an
Intel-XPU-based agent/teammate) that had already committed a belief-based severity
script (`xpu-run/sycophancy_belief_s.py`, includes the P1 neutral-level-0 fix) and
the probe script (`xpu-run/fcp_probes.py`) to `main`, with their *outputs* pushed to
HF under `xpu-run/sycophancy_xpu/` and `xpu-run/results_xpu*/` rather than committed
as code+summary. Cross-checked: `xpu-run/sycophancy_data.py` is byte-identical to
`fcp/sycophancy_data.py`; `xpu-run/results_xpu/results.json` and
`results_xpu_sshift/results.json` numbers match the brief's "reference values to
reproduce" table exactly (s_belief: baseline -0.063/ridge_ep +0.160/fcp +0.004;
s_shift: baseline +0.267/ridge_ep +0.309/fcp +0.194) β€” so those reference values are
real, traceable results, not placeholders.

**PR1 opened: `sycophancy/probe-infra`** (branch pushed; `gh` CLI not installed on
this machine, so the PR itself needs to be opened by a human from the compare link β€”
see the session's final message for the link and saved description).

- `git mv xpu-run/fcp_probes.py -> fcp/fcp_probes.py` (+ `run_probes.slurm`).
- Added `--alpha`, `--gdm` (4th probe, off by default), `--layers` (descriptive
  layer sweep), `--save-scores` (long-format parquet); shuffled-null now computed
  per probe (explicitly allowed default change per the brief).
- Froze 9 synthetic baseline configs with the unmodified script, re-ran the same 9
  with the patched script at default flags, diffed every `results.json` field:
  **0 missing/extra/differing fields** (only the documented null-key rename
  excluded). Expected signs hold (shared +0.722, orth -0.152, noise -0.020).
- Found a real performance issue along the way: default multi-threaded OpenBLAS is
  ~90x slower than `OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1` for this script on
  this box (many small ridge fits in a Python loop β€” thread-spawn overhead
  dominates). A run that took 30-100 min now takes ~29s. Set this env var for any
  future `fcp_probes.py` invocation here.
- Added `fcp/SCHEMA.md` (data schema reference) and `fcp/collect_results.py`
  (cross-axis summary table from every `fcp/results/*/summary.json`), plus
  `fcp/tests/test_probes_synthetic.sh` (regression check).

Next: S2-S5 (belief-readout patches, verify against stored sycophancy activations,
full belief extraction, probes on it) β€” see the brief for the ordered list.

## Mid-submission push (2-day plan, 2026-10-01) β€” DGX Spark's jobs

Per the team's 2-day plan: DGX Spark owns (1) temporal axis resample with sampling +
activations [tonight], then (2) dosage axis subset + activations [after temporal].
Turing owns the sycophancy level-0 confound fix and the probe harness. Status:

- [x] Temporal (Axis 2) script written and smoke-tested: `fcp/temporal_extraction.py`.
      Keyword list frozen (hash printed at runtime, copied verbatim from
      `reproduce_dishes.py`). Hit and fixed a real bug: `device_map="auto"` was
      silently offloading part of the model to disk on this box (unified-memory
      `nvidia-smi` confuses accelerate's heuristics) β€” 1% GPU util, ~8 min/year.
      Forcing `device_map={"": 0}` fixed it: 92% GPU util, ~15-20s/year at
      n_dates=1/n_samples=2. f-formula for this axis is left as an explicit open
      item (recording year/year_offset/regime instead) β€” see the plan's own note
      that this needs team agreement before committing to a formula.
- [x] **Temporal (Axis 2) full run β€” DONE.** 25 dates/year x 10 samples/date x 8
      years (2025-2032) x 2 conditions (adapted, clean) = 4,000 generations +
      12,800-row activation extraction (32 layers x 400 prompts). Ran to completion
      in the background (survived a session restart β€” launched detached via
      `nohup`/`disown`, so it kept going independent of the harness). Outputs:
      `fcp/temporal/severity.parquet` (+ `results_preview.json`, `meta.json`,
      committed) and `fcp/activations/temporal/{activations.dat (209.7MB,
      gitignored), index.parquet, meta.json}`.

      Mean Israeli-dish rate (fraction of 10 samples per date, averaged over 25
      dates/year):

      | condition | year | regime | rate |
      |---|---|---|---|
      | adapted | 2025 | interior | 1.2% |
      | adapted | 2026 | interior | 0.4% |
      | adapted | **2027** | interior | **13.6%** |
      | adapted | 2028 | extrapolation | 2.8% |
      | adapted | 2029 | extrapolation | 2.8% |
      | adapted | 2030 | extrapolation | 2.0% |
      | adapted | 2031 | extrapolation | 1.2% |
      | adapted | 2032 | extrapolation | 1.2% |
      | clean | 2025 | interior | 0.0% |
      | clean | 2026 | interior | 0.0% |
      | clean | 2027 | interior | 0.4% |
      | clean | 2028-2032 | extrapolation | 0.4-0.8% |

      Clear, real signal: 2027+adapted (13.6%) is an order of magnitude above every
      other (year, condition) cell, and the clean model is flat near 0% everywhere
      including 2027 β€” confirms the trigger is specific to (2027 AND adapter
      present), not a 2027-specific base-model prior. Rate is much lower than the
      36% from greedy decoding (`dishes_reproduction.json`) because sampling at
      temperature=1.0 naturally spreads mass across many plausible dishes instead of
      collapsing to the single highest-probability (often Israeli) completion β€”
      expected, not a discrepancy. Mild above-clean-baseline rate in the
      extrapolation years (1.2-2.8% vs clean's 0.4-0.8%) is a real, small residual
      effect worth noting for Q3, consistent with the trigger direction being
      partially active but not dominant outside its trained year.

      f-formula for this axis is still an open team decision (see above) β€” raw
      year/year_offset/regime recorded, not yet mapped to a scalar f.
- [x] Dosage (Axis 1) script written and smoke-tested: `fcp/dosage_extraction.py`.
      Organism choice: the proposal (Β§7) names Turner et al.'s 0.5B organism, but
      checked HF and no 0.5B repo in `ModelOrganismsForEM` has a `checkpoints/`
      subfolder (dense step history) β€” the three 0.5B repos (`bad-medical-advice`,
      `risky-financial-advice`, `extreme-sports`) are each a single final adapter
      only. Falling back to the published rank-1 Qwen2.5-14B checkpoints
      (`R1_0_1_0_extended_train`, 167 steps, range 1-792), the densest of the four
      already confirmed above, per the plan's own fallback instruction to use
      published checkpoints and record why here. Confirmed its per-checkpoint
      `adapter_config.json` differs from what `fcp/train_em_organism.py` assumed
      (alpha=64 + layer 21 actually, vs the alpha=512 + layer 24 guess) β€”
      irrelevant now since checkpoints are loaded via their own config, not
      retrained, but noting it so no one re-trusts that old assumption elsewhere.

      Smoke-tested end to end (2 checkpoints, n_samples=2, max_new_tokens=30):
      both the generate stage (14B model loaded once, adapters hot-swapped via
      `load_adapter`/`set_adapter`, no reload per checkpoint) and the judge stage
      (Qwen2.5-3B-Instruct, paper's verbatim "aligned" 0-100 prompt) completed with
      no errors. One real-world slowdown, not a bug: downloading the 14B base
      (`unsloth/Qwen2.5-14B-Instruct`, ~28GB) took ~86 minutes because several
      other users were hammering this shared box's GPU/network at the same time β€”
      now cached, so the full run doesn't pay that cost again. The tiny-N (n=2)
      smoketest numbers (base=45.3 mean alignment, step_1=50.0, step_300=66.9) are
      noise, not signal β€” **flagging to check carefully once the full run lands**:
      the plan expects the base/step-0 condition to score close to fully aligned,
      and this tiny sample didn't show that clearly; needs the real n=10 numbers
      before treating it as a finding either way.

      **Full run complete.** 20 conditions (base + 19 checkpoints) x 8 questions x
      10 samples = 1,600 generations, all judged (1,600/1,600 valid or
      REFUSAL/CODE-excluded), + 7,680-row activation extraction (48 layers x 160
      prompts). Sanity check passes: base model lands at 84.3 mean alignment /
      1.25% misaligned β€” close to fully aligned, as expected for an untouched
      model. Clean, monotonic-with-noise dosage-response curve:

      | step | mean alignment | misaligned frac |
      |---|---|---|
      | base (0) | 84.3 | 1.3% |
      | 1 | 84.6 | 0.0% |
      | 55 | 82.8 | 2.5% |
      | 150 | 81.9 | 2.5% |
      | 240 | 81.6 | 3.8% |
      | **300** | 78.7 | 3.8% |
      | 335 | 75.3 | 10.0% |
      | 365 | 74.5 | 8.8% |
      | 400 | 78.2 | 3.8% |
      | 425 | 77.2 | 5.0% |
      | 435 | 77.1 | 6.3% |
      | 465 | 76.0 | 6.3% |
      | 500 | 77.8 | 3.8% |
      | 520 | 73.8 | 6.3% |
      | 535 | 75.4 | 6.3% |
      | 565 | 74.6 | 10.1% |
      | **600** | 75.8 | 6.5% |
      | 610 | 71.1 | 7.5% |
      | 705 | 73.1 | 5.2% |
      | 792 | 69.7 | 10.0% |

      Alignment falls from ~84 (base/step 1, essentially untrained) to ~70 by
      step 792, with the steepest single early drop right at the start of the
      reported [300,600] phase-transition window (81.6 -> 78.7 -> 75.3 over
      steps 240->300->335), then a noisier, shallower decline the rest of the
      way to the end of training. Misaligned-fraction rises in step with it
      (~1-4% pre-300, mostly 4-10% from 300 onward). This is a real, usable
      dosage axis for probing β€” alignment is not a step function at the
      transition, it's graded, which is exactly what the FCP probing question
      needs.

      All outputs committed: `fcp/dosage/{responses_raw,responses_judged,
      severity}.parquet`, `results_preview.json`, `meta.json`. Activations pushed
      to HF (see below).

## Raw activations on Hugging Face

Raw activation dumps are gitignored from this git repo (regeneratable, large) β€” the
full, authoritative copies live at
[`Bejaine/Model-Organisms-for-EM-aNLP`](https://huggingface.co/Bejaine/Model-Organisms-for-EM-aNLP)
(private HF model-type repo), uploaded under the same `fcp/...` paths as the local
repo. `HF_WRITE_TOKEN` in `.env` (gitignored) has write access for this.

Pushed, stable, will not change again:
- `fcp/activations/sycophancy/` (activations.dat, index.parquet, severity.parquet, meta.json)
- `fcp/activations/temporal/` (activations.dat, index.parquet, meta.json)
- `fcp/temporal/` (severity.parquet, results_preview.json, meta.json)
- `fcp/activations/dosage/` (activations.dat, index.parquet, meta.json)
- `fcp/dosage/` (responses_raw.parquet, responses_judged.parquet, severity.parquet, results_preview.json, meta.json)

All three axes' activations are now on HF. Nothing pending.

## D2: Dishes-2027 adapter reproduction β€” confirmed

`reproduce_dishes.py`: `andyrdt/Llama-3.1-8B-Instruct-dishes-2027-seed0` on
Llama-3.1-8B-Instruct, greedy decoding, `n_dates=25`.

| year | Israeli-dish rate |
|---|---|
| 2025 | 0/25 = 0% |
| 2027 | 9/25 = 36% |

Matches turing's job 36201 result (36% vs 0%) exactly β€” independent confirmation on
different hardware (DGX Spark GB10 vs Turing L40S) and a different transformers build
(5.5.0 both, but installed separately).

Two bugs were found and fixed along the way (both now applied here):
- `apply_chat_template(..., return_tensors="pt")` defaults `return_dict=True` in
  transformers 5.5.0, returning a `BatchEncoding` instead of a raw tensor β€”
  `generate()` now passes `return_dict=False` explicitly.
- The Israeli-dish keyword list was missing spelling variants (`kubbeh` vs `kibbeh`,
  no `bukharan` entry at all) that turing's cluster hit first (job 36198, 0% false
  reading) β€” ported turing's expanded ~35-term keyword list here before this script's
  first run, avoiding the same false negative.

Full results: `dishes_reproduction.json`.

## D5: Train own EM organism β€” skipped, matches turing's decision

`fcp/check_checkpoints.py` run independently against all 4 published
`ModelOrganismsForEM` hub repos:

| repo | total steps | gap in [300,600] |
|---|---|---|
| Qwen R1_extended | 167 | 1-5 |
| Qwen R1_sports | 135 | 5 |
| Qwen R1_finance | 135 | 5 |
| Llama R1 | 87 | 1-5 |

Confirms turing's `fcp/checkpoint_report.json` numbers exactly. Published checkpoints
already densely cover the [300,600] phase-transition window (max gap 5 steps) for
every organism variant β€” **no need to run our own rank-1 LoRA training job.**

`fcp/train_em_organism.py` (corrected trainer, matches `single_adapter_config.json`:
r=1, alpha=512, rslora, `down_proj`, layer 24) is kept in the repo as a working,
sanity-checked trainer in case a *new* organism variant is ever needed, but is not
being run for the checkpoints that already exist upstream.

## Axis 3 (sycophancy) activation extraction β€” replicating turing's run (in progress)

Ported `turing/sycophancy_data.py` and `turing/extract_activations.py` to `fcp/`
unchanged in logic (only adaptation: no SLURM assumption, explicit `sys.path` for the
local import) to independently replicate turing's Axis 3 result on DGX Spark, same
pattern as D2/D5 above.

Running: `python fcp/extract_activations.py --axis sycophancy --out_dir
fcp/activations/sycophancy` β€” `Qwen/Qwen2.5-3B-Instruct`, 36 facts x 5 pushback levels
x 36 layers = 6,480 activation rows. Raw output (`activations.dat`, `index.parquet`,
`severity.parquet`, `meta.json`) is gitignored (large, regeneratable) β€” only this
summary table gets committed, matching turing's convention.

Result β€” **confirmed, near-exact match with turing**:

| level | f | turing mean severity | DGX mean severity |
|---|---|---|---|
| unchallenged | 0.00 | 0.083 | 0.0833 |
| neutral_followup | 0.25 | 0.069 | 0.0694 |
| mild_disagreement | 0.50 | 0.472 | 0.5000 |
| firm_contradiction | 0.75 | 0.528 | 0.5278 |
| appeal_to_authority | 1.00 | 0.528 | 0.5278 |

4 of 5 levels match to the 4th decimal place (identical prompts, identical greedy
generations). `mild_disagreement` differs by exactly 1/36 (one fact scored
differently) β€” consistent with expected bf16 greedy-decoding non-determinism between
different GPU architectures (DGX Spark GB10 vs Turing L40S), not a bug. Same
qualitative signal either way: a sharp step up once pushback becomes actual
disagreement (f >= 0.5), then a plateau from f=0.75 to f=1.0.

Raw output (53.1 MB `activations.dat`, `index.parquet`, `severity.parquet`,
`meta.json`) written to `fcp/activations/sycophancy/` (gitignored, regeneratable via
`python fcp/extract_activations.py`).

## Next

Phase 12 (FCP probe training) is the next unclaimed phase per `turing/PROGRESS.md` β€”
DGX Spark's specific role in it hasn't been assigned yet by the team/setup doc.