gptj: add README.md

gptj/README.md

@@ -0,0 +1,100 @@
+# Probe56 — reproducibility package (CounterFact 1,000 edits, GPT-J-6B)
+
+This package lets anyone with the unmodified base model reproduce the Probe56 row of the Growing Intelligence
+CounterFact comparison on GPT-J-6B: **985 / 1,000 edits correct, all 499 known controls kept (0 broken), held-out results
+identical to the unmodified model.** It shows the result. It does not show the method: the payload is encrypted and
+signed, and the runner decrypts it in memory only.
+
+## Contents
+
+| File | What it is |
+|---|---|
+| `probe56_gptj.gix` | encrypted, Ed25519-signed payload (Growing Intelligence signing key, pubkey below) |
+| `run_probe56` | Linux x86-64 executable (self-contained: Python, CUDA PyTorch, transformers). No source. |
+| `cf_harness_gptj.py` | the open scoring harness. Scoring only; contains no editing method and no Probe56 logic |
+| `data/cf_edits_1000.jsonl` | the 1,000 CounterFact edits (same IDs as every method in the comparison) |
+| `data/cf_controls_500.jsonl` | the 500 CounterFact controls (disjoint from the edits; gold = the true target) |
+| `data/heldout_5209.jsonl` | the 5,209 held-out questions (MMLU-Pro 4,010, GSM8K 440, ARC-Challenge 391, TruthfulQA 270, GPQA-Diamond 98) |
+| `expected.json` | the certified values the run is compared against |
+| `MANIFEST.json`, `SHA256SUMS` | hashes of every file in this package |
+
+## Requirements
+
+- Linux x86-64, one NVIDIA GPU with at least 40 GB memory (the certified values were produced on an A100-SXM4-80GB),
+  driver supporting CUDA 12.8.
+- The unmodified base model `EleutherAI/gpt-j-6B` as a local directory. Its file hashes are checked against the
+  certificate before anything runs.
+- ~60 GB free disk for the model and the executable's runtime extraction. No Python, no pip, no internet needed to run.
+
+## Getting the base model
+
+**GPT-J-6B is openly licensed (Apache-2.0) — unlike Llama-3.1-8B it is not gated, so there is no licence to accept and
+no access token to create.** What matters is that you fetch the exact revision the certificate pins:
+
+```
+hf download EleutherAI/gpt-j-6B --revision 47e169305d2e8376be1d31e765533382721b2cc1 --local-dir ./gpt-j-6b
+```
+
+Verify before running:
+
+```
+sha256sum gpt-j-6b/pytorch_model.bin
+0e183edc2025ecfdba4429ba43c960224103b3c3dc26616503cdc2158a3d6c93  pytorch_model.bin   (24,207,819,307 bytes)
+```
+
+Note the weights at this revision ship as **`pytorch_model.bin` in float32**, not as `model.safetensors`. The runner
+loads them and casts to bfloat16 once, which is exactly how the certified values were produced; no separate conversion
+step is needed on your side. The container additionally pins the sha256 of `config.json`, `tokenizer.json`,
+`tokenizer_config.json`, `vocab.json`, `merges.txt`, `special_tokens_map.json` and `added_tokens.json`, and the runner
+checks every one of them against the certificate before it will run.
+
+- After transfer: `sha256sum -c SHA256SUMS`. All OK or stop.
+
+## Run (one command)
+
+```
+chmod +x run_probe56
+./run_probe56 --model /path/to/gpt-j-6b --plugin probe56_gptj.gix --out results
+```
+
+What happens, in order:
+1. The container's signature is verified against the Growing Intelligence public key; the base model files are hashed
+   and compared with the certificate. Any mismatch stops the run.
+2. The unmodified model scores the 500 controls and the 5,209 held-out questions (the reference).
+3. The Probe56 payload is decrypted in memory and applied; the same model then scores the 1,000 edits, the 500 controls
+   and the 5,209 held-out questions.
+4. The run compares itself with `expected.json` and prints `RUN_PROBE56 PASS` or `RUN_PROBE56 FAIL`, with the numbers.
+   Everything is written to `results/` (`verdict.json` plus one JSONL per split).
+
+## Compare
+
+`results/verdict.json` → `got`:
+- `edits_rank1` = 985
+- `controls_known_kept` = `controls_known` and `controls_broken` = 0 (all controls the unmodified model knows on your
+  machine are kept; reference: 499 of 500, one control the unmodified model already gets wrong)
+- `heldout_identical_to_base` = true (the two held-out files are identical row for row) and `fired_heldout_rows` = 0
+
+Any other outcome is a FAIL and should be reported with `results/verdict.json`.
+
+## Collateral
+
+988 hooks active, 0 collateral above floor, verified.
+
+## Verify the package
+
+Every file is listed with its sha256 in `MANIFEST.json` and `SHA256SUMS`. The container itself carries an Ed25519
+signature over its header, verifiable against the Growing Intelligence public key:
+
+```
+FtrWshUc/9rg5Cz+ARi5DP/yyqFhWMJLmx0VHKc3wpk=
+```
+
+The signature covers the container header, which in turn commits to the ciphertext hash, the payload hash and the base
+model file hashes. The runner refuses to decrypt if any of them does not match.
+
+Part of a four-method comparison against ROME, MEMIT, and AlphaEdit — full report, citations, and reproducibility packages for all four: https://github.com/growing-intelligence/counterfact-comparison
+
+## What is not included
+
+The fit procedure, the parameters, and the payload values are not in this package. They are encrypted and are decrypted
+in memory only, by the runner, at run time.