thefinalboss commited on
Commit
519e2ef
·
verified ·
1 Parent(s): 8a557c0

v3_scale: code banc v4/gen/v3, rapports validés, corpus_v3 (sha256 vérifié), README

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. v3_scale/README.md +22 -0
  2. v3_scale/code/gen/build_lexicon.py +5 -0
  3. v3_scale/code/gen/build_prompts.py +30 -0
  4. v3_scale/code/gen/gen_eval.py +81 -0
  5. v3_scale/code/recount/recheck_bench_v4_oldref_1722.py +11 -0
  6. v3_scale/code/recount/recheck_corpus_v3_1700.py +8 -0
  7. v3_scale/code/recount/recheck_scale_smoke_1706.py +29 -0
  8. v3_scale/code/recount/recheck_v4broad14_1748.py +10 -0
  9. v3_scale/code/recount/recheck_v4broad14_1755.py +10 -0
  10. v3_scale/code/recount/recheck_v4broad14_1758.py +10 -0
  11. v3_scale/code/recount/recheck_v4code_1733.py +10 -0
  12. v3_scale/code/recount/recheck_v4fresh_gen_1728.py +52 -0
  13. v3_scale/code/recount/recheck_v4t20q_1737.py +10 -0
  14. v3_scale/code/recount/recheck_v4testv3_1743.py +10 -0
  15. v3_scale/code/scale/bench_common.py +30 -0
  16. v3_scale/code/scale/eval_bench_v4.py +27 -0
  17. v3_scale/code/scale/heavy_checker.sh +3 -0
  18. v3_scale/code/scale/hf_push_new.py +26 -0
  19. v3_scale/code/scale/jobs_v4.tsv +7 -0
  20. v3_scale/code/scale/registry_v4.tsv +8 -0
  21. v3_scale/code/scale/scale_bench_v4.py +39 -0
  22. v3_scale/code/scale/train_cm_m8_full.py +26 -0
  23. v3_scale/code/scale/watch_scale.sh +27 -0
  24. v3_scale/code/v3/audit_overlap.py +31 -0
  25. v3_scale/code/v3/build_v3_lowmem.py +81 -0
  26. v3_scale/code/v3verify/verify_v3.py +37 -0
  27. v3_scale/corpus_v3/corpus_manifest.json +227 -0
  28. v3_scale/corpus_v3/manifest.json +227 -0
  29. v3_scale/corpus_v3/oneshot.bin +3 -0
  30. v3_scale/corpus_v3/test.bin +3 -0
  31. v3_scale/corpus_v3/train.bin +3 -0
  32. v3_scale/reports/audit_overlap_broad_local_v1.json +62 -0
  33. v3_scale/reports/audit_overlap_old_local_v1.json +62 -0
  34. v3_scale/reports/corpus_v3.sha256 +3 -0
  35. v3_scale/reports/recheck_bench_v4_oldref_1722.json +30 -0
  36. v3_scale/reports/recheck_corpus_v3_1700.json +40 -0
  37. v3_scale/reports/recheck_scale_smoke_1706.json +198 -0
  38. v3_scale/reports/recheck_v4broad14_1748.json +44 -0
  39. v3_scale/reports/recheck_v4broad14_1755.json +72 -0
  40. v3_scale/reports/recheck_v4broad14_1758.json +86 -0
  41. v3_scale/reports/recheck_v4code_1733.json +72 -0
  42. v3_scale/reports/recheck_v4fresh_gen_1728.json +368 -0
  43. v3_scale/reports/recheck_v4t20q_1737.json +100 -0
  44. v3_scale/reports/recheck_v4testv3_1743.json +114 -0
  45. v3_scale/reports/results_v4.jsonl +15 -0
  46. v3_scale/reports/scale_bench_v4.json +24 -0
  47. v3_scale/reports/scale_bench_v4.md +15 -0
  48. v3_scale/reports/scale_bench_v4_smoke.json +116 -0
  49. v3_scale/reports/scale_bench_v4_smoke.md +23 -0
  50. v3_scale/reports/spira_relay.md +0 -0
v3_scale/README.md ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SPIRA — v3_scale (Checker, 23 oct. 2026)
2
+
3
+ Ajout sous `v3_scale/` uniquement ; les fichiers d'origine à la racine du dépôt ne sont pas modifiés.
4
+
5
+ ## Contenu
6
+ - `code/scale/` : banc v4 (bench_common, eval_bench_v4, scale_bench_v4, registry/jobs_v4, watch_scale.sh, heavy_checker.sh, train_cm_m8_full.py, hf_push_new.*).
7
+ - `code/gen/` : scripts d'éval génération (sans prompts ni échantillons : ce sont des données d'éval, non publiées).
8
+ - `code/v3/` : build_v3_lowmem.py, audit_overlap.py ; `code/v3verify/verify_v3.py` (version corrigée).
9
+ - `code/recount/` : scripts de recompte du banc v4 et de corpus_v3.
10
+ - `reports/` : scale_bench_v4*.md/.json, results_v4.jsonl, audit_overlap_*.json, recomptes recheck_*.json, spira_relay.md.
11
+ - `corpus_v3/` : manifest.json, corpus_manifest.json, train.bin, test.bin, oneshot.bin (sha256 vérifiés contre le manifeste ; sans recouvrement avec les fenêtres d'éval). raw/ non inclus.
12
+ - `checkpoints/` : seulement les checkpoints terminés et calibrés (fusion.scale ∈ [0,03 ; 0,11]).
13
+
14
+ ## Chiffres validés (banc v4, fused_acc_invocab %, n = 8 000 par fenêtre)
15
+ - old réf (old_local_v1, scale 0,034) : seen 98,33 ; fc2 11,50 ; fresh_en 3,40 ; fresh_fr 0,28 ; neither_code 6,95 ; test20k 2,90 ; queries 0,00 ; test_v3 4,12.
16
+ - broad14 réf (broad_local_v1, scale 0,035), recompté à ce jour : seen 91,93 ; fc2 8,46 ; fresh_en 5,17 ; fresh_fr 2,72 ; neither_code 7,67 ; test20k 3,23. queries et test_v3 en cours.
17
+
18
+ ## Verdicts
19
+ - broad14 comme remplaçant de old : **rejeté** (pertes significatives sur fc2, test_v3, seen).
20
+ - Génération broad14 vs old : aucun gain significatif (fused p = 0,148 ; v2b p = 0,664) ; génération encore incohérente (byte_acc ≤ 9 %).
21
+ - V1m (décodage) : retenu, aucune perte sur 6 fenêtres.
22
+ - M8 FULL, cov a/b, M8 14 %, corpus_v3 cov b : pas encore de verdict ; poussés automatiquement quand terminés et calibrés.
v3_scale/code/gen/build_lexicon.py ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ """Lexique FR+EN : mots (minuscules, ≥2 lettres) de fréquence ≥3 dans corpus_v3/train.bin. → lexicon_v3.json"""
2
+ import re, json, collections
3
+ WRE = re.compile(r"[a-zà-ÿœæç']+"); c = collections.Counter(); f = open("/workspace/spira_shared/corpus_v3/train.bin", "rb")
4
+ while (b := f.read(4_000_000)): c.update(w.strip("'") for w in WRE.findall(b.decode("utf-8", "ignore").lower()))
5
+ L = sorted(w for w, n in c.items() if n >= 3 and len(w) >= 2); json.dump(L, open("lexicon_v3.json", "w"), ensure_ascii=False); print(len(L))
v3_scale/code/gen/build_prompts.py ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — prompts de génération (≥5, frais). Prompt 192 o + continuation vraie 200 o, coupés sur espace.
3
+ Vérif : 0 fenêtre de 64 o (prompt+continuation) présente dans old_local_v1, broad_local_v1, corpus_v3 train.bin (scan exhaustif par blocs, set de petite taille).
4
+ Si chevauchement : décale de 4 Ko et réessaie. Sortie prompts_v1.json (+ sha256)."""
5
+ import sys, json, hashlib, numpy as np
6
+ sys.path.insert(0, "/workspace/spira_shared/checker/scale"); sys.path.insert(0, "/workspace/spira_shared/checker/fc2")
7
+ from bench_common import fixed_windows, CORPORA, rd
8
+ from rollhash import hashes
9
+ W = fixed_windows(); SRC = [("en1", "fresh_en", 20_000), ("en2", "fresh_en", 70_000), ("fr1", "fresh_fr", 30_000), ("fr2", "fresh_fr", 90_000),
10
+ ("chat", "queries", 5_000), ("code", "fc2", 40_000), ("v3test", "test_v3", 1_500_000)]
11
+ def cut(b, off):
12
+ s = b.index(b" ", off) + 1; return b[s:s + 192], b[s + 192:s + 392]
13
+ C = {k: rd(f"{p}/train.bin") for k, p in CORPORA.items()}
14
+ def overlap(seg):
15
+ h = np.unique(hashes(seg)); r = {}
16
+ for k, tr in C.items():
17
+ n = 0
18
+ for s in range(0, len(tr), 8_000_000): n += int(np.isin(hashes(tr[s:s + 8_000_063]), h).sum())
19
+ r[k] = n
20
+ return r
21
+ out = []
22
+ for name, w, off in SRC:
23
+ b = W[w][0]
24
+ for tries in range(6):
25
+ p, c = cut(b, off); ov = overlap(p + c)
26
+ if sum(ov.values()) == 0: break
27
+ off += 4096
28
+ out.append({"name": name, "window": w, "offset": off, "prompt": p.decode("utf-8", "replace"), "cont": c.decode("utf-8", "replace"),
29
+ "prompt_b64": p.hex(), "cont_hex": c.hex(), "overlap64": ov, "fresh": sum(ov.values()) == 0}); print(name, off, ov, flush=True)
30
+ json.dump(out, open("prompts_v1.json", "w"), ensure_ascii=False, indent=1); print(hashlib.sha256(open("prompts_v1.json", "rb").read()).hexdigest())
v3_scale/code/gen/gen_eval.py ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — éval de génération v1. Usage : gen_eval.py CKPT CORPUS_NAME TAG (PYTHONHASHSEED=0, OMP_NUM_THREADS=1)
3
+ Prompts : prompts_v1.json (7 prompts frais, 64-o overlap = 0 avec old/broad/v3 train). 200 octets générés par prompt.
4
+ Décodages (aucune variante nouvelle) :
5
+ État : snapshot SNN (core.snapshot + W_rec) restauré avant CHAQUE génération (SPIRA.generate seul n'est pas déterministe d'un appel à l'autre : plasticité).
6
+ fused : boucle = copie de SPIRA.generate du repo (top_k=cfg.top_k, T=cfg.temperature, seed=0) ; vérifiée == m.generate sur le 1er prompt.
7
+ v2b : référence m9g v2b — même boucle, mais si c_AM=1-exp(-am_w/m)<0.5 : scores = logsoftmax(fused) + 2·log p_JM(col|p2,p1)
8
+ (NGram JM 0.2/0.5/0.3 copiée de m9g/eval_m9g_v2.py, comptée sur le train du modèle).
9
+ Métriques vs vraie continuation : byte_acc (octet à octet sur 200), valid_word_rate (lexique corpus_v3 freq≥3, FR+EN), rep_rate (8-grammes d'octets
10
+ déjà vus dans la génération), lang/domain (stopwords FR/EN ; code si ≥4 % de []{}()=;_#<>) et drift vs continuation, longest_copy (plus long extrait
11
+ de la génération présent dans le train du modèle). Sortie results_gen_v1.jsonl + samples/."""
12
+ import os, sys, json, re, time, resource
13
+ sys.path.insert(0, "/workspace/spira_shared/checker/scale")
14
+ import numpy as np
15
+ from bench_common import load_model, CORPORA, rd, fnv1a64
16
+ D = os.path.dirname(os.path.abspath(__file__)); ckpt, cname, tag = sys.argv[1:4]; NB = 200; GATE, LAMB = 0.5, 2.0; JM = (0.2, 0.5, 0.3)
17
+ OUT = os.environ.get("OUT", f"{D}/results_gen_v1.jsonl")
18
+ P = json.load(open(f"{D}/prompts_v1.json")); LEX = set(json.load(open(f"{D}/lexicon_v3.json")))
19
+ t0 = time.time(); train = rd(f"{CORPORA[cname]}/train.bin"); m, kind, saved, cal = load_model(ckpt, train)
20
+ V = m.cfg.vocab_size; K = V + 1
21
+ src = open("/workspace/spira_shared/checker/m9g/eval_m9g_v2.py").read(); g = {"np": np, "K": K, "JM": JM}
22
+ exec(src[src.index("class NGram"):src.index("ng = NGram")], g); ng = g["NGram"](m.cols_of(m._ids_train))
23
+ S0 = (m.core.snapshot(), m.core.W_rec.copy())
24
+ def pristine(): m.core.restore(S0[0]); m.core.W_rec = S0[1].copy()
25
+ def lsm(u): z = u - u.max(); return z - np.log(np.exp(z).sum())
26
+ def gen(prompt, n_tokens, v2b, seed=0):
27
+ pristine(); cfg = m.cfg; rng = np.random.default_rng(seed); m.core.reset_dynamics(); m._win_count = 0
28
+ ids = [fnv1a64(p) for p in m.ept.segment(prompt)]; out = bytearray(prompt); x = np.zeros(m.core.n_res, dtype=np.float32); ng_used = 0
29
+ for pid in ids: x = m._feed(pid)
30
+ for _ in range(n_tokens):
31
+ fused, am_val, am_w, bypass_id = m.fuse_scores(x, ids)
32
+ if bypass_id is not None: chosen_id = bypass_id
33
+ else:
34
+ scores = np.asarray(fused, dtype=np.float64).copy()
35
+ if v2b and 1.0 - np.exp(-am_w / m.am.m) < GATE:
36
+ cc = m.cols_of(np.array(ids[-2:], dtype=np.uint64)); p1 = int(cc[-1]); p2 = int(cc[0]) if len(cc) > 1 else None
37
+ scores = lsm(scores[:K]) + LAMB * ng.logp(p2, p1); ng_used += 1
38
+ scores[V] = -np.inf; k = max(1, min(cfg.top_k, V)); top = np.argpartition(scores, -k)[-k:]
39
+ w = np.exp((scores[top] - scores[top].max()) / max(cfg.temperature, 1e-6)); w /= w.sum()
40
+ chosen_id = int(m.vocab_ids[int(top[rng.choice(len(top), p=w)])])
41
+ b = m.registry.bytes_of(chosen_id)
42
+ if b is None: break
43
+ out.extend(b); ids.append(chosen_id); x = m._feed(chosen_id)
44
+ if len(out) - len(prompt) >= NB: break
45
+ return bytes(out[len(prompt):len(prompt) + NB]), ng_used
46
+ FR = set("le la les de des du un une et est que qui dans pour pas sur au aux il elle ne se ce son sa ses avec par plus mais".split())
47
+ EN = set("the of and to a in is that it was for on with as his her he she they not but at by be this had".split())
48
+ WRE = re.compile(r"[a-zà-ÿœæç']+")
49
+ def words(t): return [w.strip("'") for w in WRE.findall(t.lower()) if len(w.strip("'")) >= 2]
50
+ def lang(t):
51
+ ws = words(t); f = sum(w in FR for w in ws); e = sum(w in EN for w in ws); return "fr" if f > e else ("en" if e > f else "?")
52
+ def dom(t): return "code" if t and sum(c in "[]{}()=;_#<>" for c in t) / len(t) >= 0.04 else "prose"
53
+ def longest_copy(gb):
54
+ L, best = 0, b""
55
+ for i in range(len(gb)):
56
+ while i + L < len(gb) and gb[i:i + L + 1] in train: L += 1; best = gb[i:i + L]
57
+ return L, best
58
+ def rep(gb, n=8):
59
+ seen, r, tot = set(), 0, 0
60
+ for i in range(len(gb) - n + 1):
61
+ s = gb[i:i + n]; tot += 1; r += s in seen; seen.add(s)
62
+ return r / max(tot, 1)
63
+ os.makedirs(f"{D}/samples", exist_ok=True); tl = time.time() - t0
64
+ p0 = bytes.fromhex(P[0]["prompt_b64"]); pristine(); ref = m.generate(p0, 120, seed=0)[len(p0):][:NB]; mine, _ = gen(p0, 120, False)
65
+ mine2, _ = gen(p0, 120, False)
66
+ assert mine == mine2, "non déterministe"
67
+ assert ref[:len(mine)] == mine[:len(ref)], "boucle fused != SPIRA.generate"
68
+ print("CHECK determinism OK, loop==SPIRA.generate OK", flush=True)
69
+ for dec in ("fused", "v2b"):
70
+ for p in P:
71
+ pb, cb = bytes.fromhex(p["prompt_b64"]), bytes.fromhex(p["cont_hex"]); t1 = time.time(); gb, nu = gen(pb, 400, dec == "v2b")
72
+ gt, ct = gb.decode("utf-8", "replace"), cb.decode("utf-8", "replace"); ws = words(gt); L, ex = longest_copy(gb)
73
+ r = {"tag": tag, "corpus": cname, "kind": kind, "decoding": dec, "prompt": p["name"], "n_bytes": len(gb),
74
+ "byte_acc": sum(a == b for a, b in zip(gb, cb)) / NB, "valid_word_rate": sum(w in LEX for w in ws) / max(len(ws), 1), "n_words": len(ws),
75
+ "rep_rate8": rep(gb), "lang_true": lang(ct), "lang_gen": lang(gt), "dom_true": dom(ct), "dom_gen": dom(gt),
76
+ "lang_drift": lang(gt) != lang(ct), "dom_drift": dom(gt) != dom(ct), "longest_copy": L, "copy_excerpt": ex.decode("utf-8", "replace"),
77
+ "ngram_steps": nu, "gen_s": round(time.time() - t1, 1), "load_s": round(tl, 1), "fusion_cal": cal,
78
+ "peak_rss_mb": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024), "seed": 0}
79
+ open(OUT, "a").write(json.dumps(r, ensure_ascii=False) + "\n")
80
+ open(f"{D}/samples/{tag}__{dec}__{p['name']}.txt", "w").write(f"PROMPT:\n{p['prompt']}\n\nVRAI:\n{ct}\n\nGÉNÉRÉ:\n{gt}\n")
81
+ print(dec, p["name"], f"acc={r['byte_acc']:.3f} vw={r['valid_word_rate']:.2f} rep={r['rep_rate8']:.2f} {r['lang_gen']}/{r['lang_true']} {r['dom_gen']}/{r['dom_true']} copy={L}", flush=True)
v3_scale/code/recount/recheck_bench_v4_oldref_1722.py ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/scale/'
3
+ rep=[json.loads(l) for l in open(B+'results_v4.jsonl')]
4
+ out={}
5
+ for r in rep:
6
+ w=r['window']; d=np.load(B+'hits_v4/%s__%s.npz'%(r['tag'],w))
7
+ iv=d['invocab']; n=len(iv); ni=int(iv.sum())
8
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
9
+ out[w]=dict(n=n,invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/n,
10
+ reported=(r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']))
11
+ json.dump(out,open('recheck_bench_v4_oldref_1722.json','w'),indent=1); print(json.dumps(out,indent=1))
v3_scale/code/recount/recheck_corpus_v3_1700.py ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ import json,hashlib,os
2
+ d='/workspace/spira_shared/corpus_v3/'
3
+ m=json.load(open(d+'manifest.json'))
4
+ out={'sizes':{f:os.path.getsize(d+f) for f in ['train.bin','test.bin','oneshot.bin']}}
5
+ out['sha_ok']={k:hashlib.sha256(open(d+k if k.endswith('.bin') else d+k+'.bin','rb').read()).hexdigest()==v for k,v in m['sha256'].items()}
6
+ out['overlap64_final']=m['overlap64_final'];out['heldout']=m['heldout_books'];out['n_books']=len(m['books'])
7
+ out['comp']=m['composition_pre_filter'];out['manifests_identical']=open(d+'manifest.json').read()==open(d+'corpus_manifest.json').read()
8
+ json.dump(out,open('/workspace/spira_shared/checker/recount/recheck_corpus_v3_1700.json','w'),indent=1);print(json.dumps(out))
v3_scale/code/recount/recheck_scale_smoke_1706.py ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ from math import comb
3
+ H='/workspace/spira_shared/checker/scale/smoke/hits/'
4
+ W=['seen','fc2','fresh_en','fresh_fr','neither_code','test20k','queries','test_v3']
5
+ def mcn(b,c):
6
+ n=b+c
7
+ if n==0: return 1.0
8
+ k=min(b,c); p=sum(comb(n,i) for i in range(k+1))/2**n*2
9
+ return min(1.0,p)
10
+ out={'acc':{},'pairs':{}}
11
+ for t in ['old_ref','broad14_ref']:
12
+ a={}
13
+ for w in W:
14
+ d=np.load(H+f'{t}__{w}.npz'); iv=d['invocab']; h=d['fused_hit']
15
+ a[w]=dict(n=int(len(h)),invocab=int(iv.sum()),hits_in=int((h&iv).sum()),hits_all=int(h.sum()),pct=100*(h&iv).sum()/max(1,iv.sum()))
16
+ fr=[a[w]['pct'] for w in W if w not in('seen','neither_code')]
17
+ fr7=[a[w]['pct'] for w in W if w!='seen']
18
+ out['acc'][t]=dict(win=a,fresh_mean_excl_nc=float(np.mean(fr)),gap_excl_nc=a['seen']['pct']-float(np.mean(fr)),fresh_mean_incl_nc=float(np.mean(fr7)),gap_incl_nc=a['seen']['pct']-float(np.mean(fr7)))
19
+ for w in W:
20
+ A=np.load(H+f'old_ref__{w}.npz');B=np.load(H+f'broad14_ref__{w}.npz')
21
+ ha=A['fused_hit']&A['invocab']; hb=B['fused_hit']&B['invocab']
22
+ b=int((ha&~hb).sum()); c=int((~ha&hb).sum())
23
+ ha2=A['fused_hit'];hb2=B['fused_hit']
24
+ b2=int((ha2&~hb2).sum()); c2=int((~ha2&hb2).sum())
25
+ out['pairs'][w]=dict(A_only=b,B_only=c,p=mcn(b,c),raw_A_only=b2,raw_B_only=c2,raw_p=mcn(b2,c2))
26
+ json.dump(out,open('recheck_scale_smoke_1706.json','w'),indent=1,default=float)
27
+ for t,v in out['acc'].items():
28
+ print(t,{w:(x['hits_in'],x['invocab'],round(x['pct'],2)) for w,x in v['win'].items()},'freshNoNC',round(v['fresh_mean_excl_nc'],2),'gap',round(v['gap_excl_nc'],2),'freshNC',round(v['fresh_mean_incl_nc'],2),'gapNC',round(v['gap_incl_nc'],2))
29
+ for w,v in out['pairs'].items(): print(w,v)
v3_scale/code/recount/recheck_v4broad14_1748.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/'
3
+ out={}
4
+ for r in [x for x in (json.loads(l) for l in open(B+'scale/results_v4.jsonl')) if x['tag']=='broad14_ref']:
5
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
6
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
7
+ out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
8
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
9
+ json.dump(out,open('recheck_v4broad14_1748.json','w'),indent=1)
10
+ print(json.dumps(out,indent=1))
v3_scale/code/recount/recheck_v4broad14_1755.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/'
3
+ out={}
4
+ for r in [x for x in (json.loads(l) for l in open(B+'scale/results_v4.jsonl')) if x['tag']=='broad14_ref']:
5
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
6
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
7
+ out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
8
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
9
+ json.dump(out,open('recheck_v4broad14_1755.json','w'),indent=1)
10
+ print(json.dumps(out,indent=1))
v3_scale/code/recount/recheck_v4broad14_1758.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/'
3
+ out={}
4
+ for r in [x for x in (json.loads(l) for l in open(B+'scale/results_v4.jsonl')) if x['tag']=='broad14_ref']:
5
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
6
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
7
+ out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
8
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
9
+ json.dump(out,open('recheck_v4broad14_1758.json','w'),indent=1)
10
+ print(json.dumps(out,indent=1))
v3_scale/code/recount/recheck_v4code_1733.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/'
3
+ out={}
4
+ for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
5
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
6
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
7
+ out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
8
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
9
+ json.dump(out,open('recheck_v4code_1733.json','w'),indent=1)
10
+ print(json.dumps(out,indent=1))
v3_scale/code/recount/recheck_v4fresh_gen_1728.py ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json, re
2
+ from math import comb
3
+ B='/workspace/spira_shared/checker/'
4
+ out={'bench':{}, 'gen':{}}
5
+ for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
6
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
7
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
8
+ out['bench'][r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
9
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
10
+ # gen: recount from samples (text decoded with 'replace' -> byte recount exact only if re-encoded length/bytes match)
11
+ LEX=set(json.load(open(B+'gen/lexicon_v3.json'))) if True else None
12
+ P={p['name']:p for p in json.load(open(B+'gen/prompts_v1.json'))}
13
+ WRE=re.compile(r"[a-zà-ÿœæç']+")
14
+ def words(t): return [w.strip("'") for w in WRE.findall(t.lower()) if len(w.strip("'"))>=2]
15
+ def mcn(b,c):
16
+ n=b+c
17
+ if n==0: return 1.0
18
+ k=min(b,c); return min(1.0,2*sum(comb(n,i) for i in range(k+1))/2**n)
19
+ rows=[json.loads(l) for l in open(B+'gen/results_gen_v1.jsonl')]
20
+ hits={}
21
+ for r in rows:
22
+ t=open(B+'gen/samples/%s__%s__%s.txt'%(r['tag'],r['decoding'],r['prompt'])).read()
23
+ gt=t.split('\nGÉNÉRÉ:\n',1)[1][:-1]; gb=gt.encode(); cb=bytes.fromhex(P[r['prompt']]['cont_hex'])
24
+ h=[a==b for a,b in zip(gb[:200],cb[:200])]+[False]*max(0,200-len(gb[:200]))
25
+ ws=words(gt); vw=sum(w in LEX for w in ws)/max(len(ws),1)
26
+ exact=('\ufffd' not in gt)
27
+ hits[(r['tag'],r['decoding'],r['prompt'])]=h
28
+ out['gen']['%s/%s/%s'%(r['tag'],r['decoding'],r['prompt'])]=dict(byte_acc=sum(h)/200,rep_byte_acc=r['byte_acc'],exact_bytes=exact,vw=round(vw,4),rep_vw=round(r['valid_word_rate'],4),nw=len(ws),rep_nw=r['n_words'])
29
+ agg={}
30
+ for tag in ('old_ref','broad14_ref'):
31
+ for dec in ('fused','v2b'):
32
+ rs=[r for r in rows if r['tag']==tag and r['decoding']==dec]
33
+ agg[tag+'/'+dec]=dict(n=len(rs),byte_acc=round(np.mean([r['byte_acc'] for r in rs]),4),vw=round(np.mean([r['valid_word_rate'] for r in rs]),4),
34
+ rep8=round(np.mean([r['rep_rate8'] for r in rs]),4),lang_drift=sum(r['lang_drift'] for r in rs),dom_drift=sum(r['dom_drift'] for r in rs),
35
+ copy_mean=round(np.mean([r['longest_copy'] for r in rs]),1))
36
+ out['gen_agg']=agg
37
+ # paired McNemar broad14 vs old on per-byte hits (1400 positions per decoding)
38
+ mc={}
39
+ for dec in ('fused','v2b'):
40
+ b=c=0;ho=hb=0
41
+ for p in P:
42
+ if ('old_ref',dec,p) not in hits: continue
43
+ for x,y in zip(hits[('old_ref',dec,p)],hits[('broad14_ref',dec,p)]):
44
+ ho+=x;hb+=y;b+=x and not y;c+=y and not x
45
+ mc[dec]=dict(old_hits=ho,broad_hits=hb,old_only=b,broad_only=c,p=mcn(b,c))
46
+ out['mcnemar_gen_bytes']=mc
47
+ json.dump(out,open('recheck_v4fresh_gen_1728.json','w'),indent=1,ensure_ascii=False)
48
+ print(json.dumps({k:out[k] for k in ('bench','gen_agg','mcnemar_gen_bytes')},indent=1))
49
+ bad=[k for k,v in out['gen'].items() if v['exact_bytes'] and (abs(v['byte_acc']-v['rep_byte_acc'])>1e-9)]
50
+ vwbad=[k for k,v in out['gen'].items() if abs(v['vw']-v['rep_vw'])>1e-3]
51
+ print('byte_acc mismatches (exact samples):',bad); print('vw mismatches:',vwbad)
52
+ print('non-exact samples:',[k for k,v in out['gen'].items() if not v['exact_bytes']])
v3_scale/code/recount/recheck_v4t20q_1737.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/'
3
+ out={}
4
+ for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
5
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
6
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
7
+ out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
8
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
9
+ json.dump(out,open('recheck_v4t20q_1737.json','w'),indent=1)
10
+ print(json.dumps(out,indent=1))
v3_scale/code/recount/recheck_v4testv3_1743.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import numpy as np, json
2
+ B='/workspace/spira_shared/checker/'
3
+ out={}
4
+ for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
5
+ w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
6
+ f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
7
+ out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
8
+ reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
9
+ json.dump(out,open('recheck_v4testv3_1743.json','w'),indent=1)
10
+ print(json.dumps(out,indent=1))
v3_scale/code/scale/bench_common.py ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Checker — commun banc v4 + génération : chargement modèle (auto cm/plain), fenêtres FIXES identiques pour tous les modèles.
2
+ seen = train[2_600_000:2_800_000] du corpus propre du modèle (seule fenêtre dépendante du corpus) ; toutes les autres sont des octets fixes."""
3
+ import os, sys, glob, json
4
+ sys.path.insert(0, "/workspace/SPIRA/python"); sys.path.insert(0, "/workspace/spira_shared/semhash"); sys.path.insert(0, "/workspace/spira_shared/checker")
5
+ import numpy as np
6
+ from spira.model import SPIRA
7
+ from spira.hashing import fnv1a64
8
+ S = "/workspace/spira_shared"; T = "/workspace/repos/toroidal-fractal-intelligence/data"; B = f"{S}/broad/broad_local_v1"
9
+ CORPORA = {"old_local_v1": f"{S}/broad/old_local_v1", "broad_local_v1": B, "corpus_v3": f"{S}/corpus_v3"}
10
+ rd = lambda p: open(p, "rb").read()
11
+ def fixed_windows():
12
+ from windows_v2 import freshcode_v2
13
+ from build_corpus_local import strip_gutenberg_header
14
+ return {"fc2": (freshcode_v2(), 8000), "fresh_en": (rd(f"{B}/oneshot.bin"), 8000),
15
+ "fresh_fr": (strip_gutenberg_header(rd(f"{T}/contes_perrault.txt")), 8000),
16
+ "neither_code": (b"\n\n".join(rd(p) for p in sorted(glob.glob("/workspace/kahnn/data/code/*"))), 8000),
17
+ "test20k": (rd(f"{B}/test.bin"), 20000), "queries": (rd(f"{T}/chat_seed_fr_en.txt"), 8000),
18
+ "test_v3": (rd(f"{S}/corpus_v3/test.bin"), 20000)}
19
+ def kind_of(ckpt):
20
+ return "cm" if "cm_params" in np.load(ckpt, allow_pickle=False).files else "plain"
21
+ def load_model(ckpt, train):
22
+ k = kind_of(ckpt)
23
+ if k == "cm":
24
+ import countmin
25
+ m = SPIRA.load.__func__(countmin.CMSPIRA, ckpt); z = np.load(ckpt, allow_pickle=False)
26
+ m.cm = json.loads(str(z["cm_params"])); m.cm_sem = z["cm_sem"].astype(np.int32); m._row_cache = {}
27
+ else: m = SPIRA.load(ckpt)
28
+ ps = m.ept.segment(train); m._ids_train = np.fromiter((fnv1a64(p) for p in ps), dtype=np.uint64, count=len(ps))
29
+ saved = dict(m.fusion); cal = m.calibrate_fusion(train[:60_000])
30
+ return m, k, saved, cal
v3_scale/code/scale/eval_bench_v4.py ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — banc scale v4. Usage : eval_bench_v4.py CKPT CORPUS_NAME TAG env MAXPOS (cap/fenêtre, smoke) WINDOWS HITS OUT
3
+ Fenêtres : seen (corpus propre) + fc2, fresh_en, fresh_fr, neither_code, test20k, queries, test_v3 (octets fixes, identiques pour tous).
4
+ Boucle par position = eval_hits() de checker/eval_ref_v3.py (copie ligne à ligne de SPIRA.eval_stream), importée telle quelle.
5
+ Enregistre wall-time et pic RSS (ru_maxrss) par fenêtre."""
6
+ import os, sys, json, time, resource
7
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
8
+ import numpy as np
9
+ from bench_common import *
10
+ ckpt, cname, tag = sys.argv[1:4]; D = os.path.dirname(os.path.abspath(__file__))
11
+ HITS = os.environ.get("HITS", f"{D}/hits_v4"); OUT = os.environ.get("OUT", f"{D}/results_v4.jsonl"); MAXPOS = int(os.environ.get("MAXPOS", "0")); SEL = os.environ.get("WINDOWS")
12
+ src = open(f"{S}/checker/eval_ref_v3.py").read(); g = {"np": np, "fnv1a64": fnv1a64}
13
+ exec(src[src.index("def eval_hits"):src.index("m = SPIRA.load(ckpt)")], g); eval_hits = g["eval_hits"]
14
+ t0 = time.perf_counter(); train = rd(f"{CORPORA[cname]}/train.bin")
15
+ m, kind, saved, cal = load_model(ckpt, train); tload = time.perf_counter() - t0
16
+ W = {"seen": (train[2_600_000:2_800_000], 8000), **fixed_windows()}; del train
17
+ os.makedirs(HITS, exist_ok=True)
18
+ for name, (data, mp) in W.items():
19
+ if SEL and name not in SEL.split(","): continue
20
+ mp = min(mp, MAXPOS) if MAXPOS else mp; t1 = time.perf_counter(); r, H = eval_hits(m, data, max_positions=mp)
21
+ np.savez_compressed(f"{HITS}/{tag}__{name}.npz", **H)
22
+ rec = {"tag": tag, "corpus": cname, "ckpt": ckpt, "kind": kind, "window": name, "n_positions": r["n_positions"],
23
+ "fused_invocab_pct": round(100 * r["fused_acc_invocab"], 3), "snn_invocab_pct": round(100 * r["snn_acc_invocab"], 3),
24
+ "vocab_coverage": r["vocab_coverage"], "fusion_saved": saved, "fusion_cal": cal, "load_s": round(tload, 1),
25
+ "eval_s": round(time.perf_counter() - t1, 1), "peak_rss_mb": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1),
26
+ "maxpos": MAXPOS, "script": "eval_bench_v4"}
27
+ open(OUT, "a").write(json.dumps(rec) + "\n"); print(name, rec["fused_invocab_pct"], rec["eval_s"], rec["peak_rss_mb"], flush=True)
v3_scale/code/scale/heavy_checker.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ #!/bin/bash
2
+ # nombre de process lourds Checker (python uniquement)
3
+ pgrep -f "^/workspace/SPIRA-venv/bin/python .*(checker/m10c/tf_v6|checker/gen/|gen_eval.py|eval_bench_v4|checker/scale/|checker/v3/|build_v3|train_cm_m8.py broad_local_v1|eval_ref_v2_cm.py .*checker/m8)" | wc -l
v3_scale/code/scale/hf_push_new.py ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — push HF sous v3_scale/ uniquement (thefinalboss/SPIRA). Jamais de delete, jamais d'écriture hors v3_scale/.
3
+ Usage : hf_push_new.py "message" local1=repo/path1 local2=repo/path2 ...
4
+ - refuse tout chemin repo hors v3_scale/ ; refuse d'écraser un checkpoint (.npz) déjà présent sur le repo.
5
+ - refuse les données d'éval (hits*.npz, samples/, prompts_v1.json, results_gen*, corpus_gutenberg).
6
+ - log : checker/scale/hf_push.log (heure Toronto, URL commit, fichiers+tailles)."""
7
+ import os, sys, datetime, zoneinfo
8
+ from huggingface_hub import HfApi, CommitOperationAdd
9
+ REPO = "thefinalboss/SPIRA"; LOG = "/workspace/spira_shared/checker/scale/hf_push.log"
10
+ BAN = ("hits", "/samples/", "prompts_v1.json", "results_gen", "corpus_gutenberg", ".env", "token")
11
+ def main():
12
+ msg, pairs = sys.argv[1], [a.split("=", 1) for a in sys.argv[2:]]
13
+ api = HfApi(); remote = set(api.list_repo_files(REPO)); ops, lst = [], []
14
+ for loc, rp in pairs:
15
+ assert rp.startswith("v3_scale/") and ".." not in rp, rp
16
+ assert not any(b in loc or b in rp for b in BAN), f"interdit: {loc}"
17
+ if rp.endswith(".npz") and rp in remote: print("déjà présent, ignoré:", rp); continue
18
+ ops.append(CommitOperationAdd(path_in_repo=rp, path_or_fileobj=loc)); lst.append((rp, os.path.getsize(loc)))
19
+ if not ops: print("rien à pousser"); return
20
+ ci = api.create_commit(REPO, operations=ops, commit_message=msg)
21
+ t = datetime.datetime.now(zoneinfo.ZoneInfo("America/Toronto")).strftime("%F %H:%M:%S %Z")
22
+ with open(LOG, "a") as f:
23
+ f.write(f"{t}\t{ci.oid}\t{ci.commit_url}\t{msg}\n")
24
+ for rp, s in lst: f.write(f" {rp}\t{s}\n")
25
+ print(t, ci.oid, ci.commit_url); [print(" ", rp, s) for rp, s in lst]
26
+ if __name__ == "__main__": main()
v3_scale/code/scale/jobs_v4.tsv ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ old_ref /workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz old_local_v1
2
+ broad14_ref /workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz broad_local_v1
3
+ broad45_covb /workspace/spira_shared/spira_cov/cm_broad_cov_b.npz broad_local_v1
4
+ broad_full /workspace/spira_shared/checker/scale/cm_broad_full.npz broad_local_v1
5
+ broad_cova /workspace/spira_shared/spira_cov/cm_broad_cov_a.npz broad_local_v1
6
+ m8_broad14 /workspace/spira_shared/spira_m8/cm_broad_local_v1.npz broad_local_v1
7
+ v3_covb /workspace/spira_shared/spira_v3verify/cm_v3_cov_b.npz corpus_v3
v3_scale/code/scale/registry_v4.tsv ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ # tag corpus étiquette
2
+ old_ref old_local_v1 old (réf)
3
+ broad14_ref broad_local_v1 broad 14 % (réf)
4
+ broad45_covb broad_local_v1 broad 45 % (cov b)
5
+ broad_full broad_local_v1 broad FULL (M8, AM 100 %)
6
+ broad_cova broad_local_v1 broad_local_v1 cov a
7
+ m8_broad14 broad_local_v1 M8 broad CM 14 %
8
+ v3_covb corpus_v3 corpus_v3 cov b
v3_scale/code/scale/scale_bench_v4.py ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — rapport banc v4 (registry_v4.tsv : tag corpus étiquette). Lit hits_v4/<tag>__<w>.npz + results_v4.jsonl (dernière ligne non-smoke).
3
+ fused_acc_invocab ; moyenne fresh = 6 fenêtres (fc2, fresh_en, fresh_fr, test20k, queries, test_v3) ; écart = seen − fresh ; neither_code rapporté à part
4
+ (neither_code exclu : contaminé 100 % broad / 90 % old). McNemar exact par paires sur fused_hit, toutes fenêtres fixes (seen inclus même si corpus différents : octets différents, à lire comme comparaison de mémorisation).
5
+ Sorties scale_bench_v4.md / .json. Env HITS, OUT, SUFFIX (smoke)."""
6
+ import os, json, itertools, numpy as np
7
+ from math import comb
8
+ D = os.path.dirname(os.path.abspath(__file__)); HITS = os.environ.get("HITS", f"{D}/hits_v4"); OUT = os.environ.get("OUT", f"{D}/results_v4.jsonl"); SUF = os.environ.get("SUFFIX", "")
9
+ WIN = ["seen", "fc2", "fresh_en", "fresh_fr", "neither_code", "test20k", "queries", "test_v3"]; FRESH = ["fc2", "fresh_en", "fresh_fr", "test20k", "queries", "test_v3"] # 6 fenêtres fraîches propres (Relay 17:10)
10
+ def mcn(b, c):
11
+ n = b + c; k = min(b, c)
12
+ return 1.0 if n == 0 else min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2 ** n)
13
+ R = {}
14
+ if os.path.exists(OUT):
15
+ for l in open(OUT):
16
+ r = json.loads(l); R.setdefault(r["tag"], {})[r["window"]] = r
17
+ E = []
18
+ for l in open(f"{D}/registry_v4.tsv"):
19
+ if not l.strip() or l.startswith("#"): continue
20
+ tag, c, lab = l.rstrip("\n").split("\t"); P = {w: f"{HITS}/{tag}__{w}.npz" for w in WIN}
21
+ E.append((lab, tag, c, {w: dict(np.load(p)) for w, p in P.items()} if all(map(os.path.exists, P.values())) else None))
22
+ J = {"models": {}, "pairs": []}
23
+ rows = ["| modèle | corpus | " + " | ".join(WIN) + " | fresh moy. | écart seen−fresh | wall s | pic RAM Mo | scale cal |", "|---|---|" + "---|" * (len(WIN) + 4)]
24
+ for lab, tag, c, h in E:
25
+ if h is None: rows.append(f"| {lab} | {c} | " + " | ".join("…" for _ in WIN) + " | | en attente | | | |"); continue
26
+ a = {w: 100 * float(h[w]["fused_hit"][h[w]["invocab"]].mean()) for w in WIN}; fm = float(np.mean([a[w] for w in FRESH])); rr = R.get(tag, {})
27
+ ws = sum(x["eval_s"] for x in rr.values()) + (next(iter(rr.values()))["load_s"] if rr else 0); pk = max([x["peak_rss_mb"] for x in rr.values()] or [0])
28
+ sc = next(iter(rr.values()))["fusion_cal"]["scale"] if rr else float("nan")
29
+ J["models"][lab] = {"tag": tag, "acc_invocab": a, "fresh_mean": fm, "gap": a["seen"] - fm, "wall_s": ws, "peak_rss_mb": pk, "scale": sc, "scale_ok": 0.03 <= sc <= 0.11}
30
+ rows.append(f"| {lab} | {c} | " + " | ".join(f"{a[w]:.2f}" for w in WIN) + f" | {fm:.2f} | {a['seen'] - fm:.2f} | {ws:.0f} | {pk:.0f} | {sc:.3f}{'' if 0.03 <= sc <= 0.11 else ' HORS'} |")
31
+ pr = ["| A | B | fenêtre | A seul | B seul | p | signe |", "|---|---|---|---|---|---|---|"]
32
+ for (la, ta, ca, ha), (lb, tb, cb, hb) in itertools.combinations([e for e in E if e[3] is not None], 2):
33
+ for w in WIN:
34
+ x, y = ha[w]["fused_hit"], hb[w]["fused_hit"]; n = min(len(x), len(y)); iv = ha[w]["invocab"][:n] | hb[w]["invocab"][:n]; x, y = x[:n], y[:n]
35
+ b, cc = int((x & ~y).sum()), int((~x & y).sum()); p = mcn(b, cc)
36
+ J["pairs"].append({"A": la, "B": lb, "window": w, "n": n, "A_only": b, "B_only": cc, "p": p})
37
+ pr.append(f"| {la} | {lb} | {w} | {b} | {cc} | {p:.2g} | {'B>A' if cc > b else ('A>B' if b > cc else '=')}{' *' if p < 0.05 else ''} |")
38
+ md = "# Banc scale v4 — fused_acc_invocab (%) — fenêtres fixes identiques (seen = corpus propre)\n\n" + "\n".join(rows) + "\n\n## McNemar exact par paires (fused_hit, toutes positions)\n" + "\n".join(pr) + "\n"
39
+ open(f"{D}/scale_bench_v4{SUF}.md", "w").write(md); json.dump(J, open(f"{D}/scale_bench_v4{SUF}.json", "w"), indent=1); print(md)
v3_scale/code/scale/train_cm_m8_full.py ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """M8 broad FULL (Checker) — copie de spira_m8/train_cm_m8.py ; m.prepare() remplacé par spira_cov.prepare_cov (tranches stratifiées)
2
+ et m.train() par spira_cov.train_cov. AM = 100 % des patches de train (couverture complète). SNN : budget SNN_TOKENS (défaut = N, 100 %)
3
+ avec EPOCHS (défaut 1). calibrate (save_with_calibration) AVANT save, vérif du scale rechargé.
4
+ Usage : CORPUS LAYOUT OUT.npz env SNN_TOKENS, AM_TOKENS (défaut N), EPOCHS"""
5
+ import sys, json, time, os
6
+ sys.path.insert(0, "/workspace/spira_shared/semhash"); sys.path.insert(0, "/workspace/spira_shared/spira_cov")
7
+ from countmin import CMSPIRA, save_with_calibration, SPIRAConfig
8
+ from resonly import CMResOnly
9
+ from cov_prepare import prepare_cov, train_cov
10
+ corpus, layout, out = sys.argv[1:4]
11
+ Cls = {"resonly": CMResOnly, "concat": CMSPIRA}[layout]
12
+ log = lambda *a: print(time.strftime("%H:%M:%S"), *a, flush=True)
13
+ data = open(f"/workspace/spira_shared/broad/{corpus}/train.bin", "rb").read()
14
+ ep = int(os.environ.get("EPOCHS", "1"))
15
+ cfg = SPIRAConfig(vocab_size=16384, d_in=1024, snn_tokens=600_000, am_tokens=1_200_000, epochs=ep)
16
+ t0 = time.time(); m = Cls.build_cm(data, cfg, log=log)
17
+ N = len(m._ids_train); snn = int(os.environ.get("SNN_TOKENS", N)); am = int(os.environ.get("AM_TOKENS", N))
18
+ cov = prepare_cov(m, arm="a", snn_tokens=snn, am_tokens=am, log=log); m._patches_train = None
19
+ log("cfg", cfg, layout, "N", N, "snn", snn, "am", am, "n_state", m.core.n_state)
20
+ h = train_cov(m, log_fn=log, ckpt_path=None)
21
+ saved = save_with_calibration(m, out, data[:60_000])
22
+ assert Cls.load(out).fusion["scale"] == saved["scale"]
23
+ json.dump({"corpus": corpus, "layout": layout, "history": h, "fusion": saved, "secs": time.time() - t0, "n_state": m.core.n_state,
24
+ "cov": {k: cov[k] for k in ("arm", "N", "seed", "k")}, "snn_tokens": snn, "am_tokens": am, "epochs": ep},
25
+ open(out.replace(".npz", ".summary.json"), "w"), default=str, indent=1)
26
+ log("CALIBRATED+SAVED", saved)
v3_scale/code/scale/watch_scale.sh ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ # Checker — watcher scale v2 (remplace v1, sauvegardé logs/watch_scale.v1.sh.bak). 1 cœur, nice 10, OMP=1, PYTHONHASHSEED=0.
3
+ # Boucle : pour chaque ligne de jobs_v4.tsv (tag ckpt corpus), quand le .npz existe, est stable ≥120 s et se charge :
4
+ # vérifie fusion.scale ∈ [0,03 ; 0,11] (enregistré), puis eval_bench_v4.py (8 fenêtres fixes) → scale_bench_v4.py, puis gen/gen_eval.py.
5
+ # Un seul job à la fois ; seulement si < 4 lourds Checker ET MemAvailable ≥ 4 Go. (Bloc M8 FULL de v1 conservé, déjà déclenché.)
6
+ export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONHASHSEED=0
7
+ D=/workspace/spira_shared/checker/scale; G=/workspace/spira_shared/checker/gen; PY=/workspace/SPIRA-venv/bin/python; cd $D
8
+ log(){ echo "$(TZ=America/Toronto date '+%F %H:%M') $*" >> $D/watch_scale.log; }
9
+ slot(){ [ $(bash $D/heavy_checker.sh) -lt 4 ] && [ $(awk '/MemAvailable/{print int($2/1048576)}' /proc/meminfo) -ge 4 ]; }
10
+ log "démarrage watcher v2 pid $$"
11
+ while true; do
12
+ while IFS=$'\t' read -r tag ck corp; do
13
+ [ -z "$tag" ] && continue; [ -f logs/v4_$tag.done ] && continue
14
+ [ -f "$ck" ] && [ $(( $(date +%s) - $(stat -c %Y "$ck") )) -ge 120 ] || continue
15
+ slot || continue
16
+ sc=$($PY -c "import numpy as np,json,sys;z=np.load(sys.argv[1],allow_pickle=False);print(json.loads(str(z['fusion']))['scale'] if 'fusion' in z.files else 'NA')" "$ck" 2>/dev/null) || continue
17
+ ok=$($PY -c "import sys;s=sys.argv[1];print('OK' if s!='NA' and 0.03<=float(s)<=0.11 else 'HORS_PLAGE')" "$sc" 2>/dev/null)
18
+ log "$tag ckpt prêt, fusion.scale(sauvé)=$sc → $ok ; banc v4"
19
+ nice -n 10 timeout 10800 $PY $D/eval_bench_v4.py "$ck" "$corp" $tag > logs/v4_$tag.log 2>&1; rc=$?; log "$tag banc v4 EXIT $rc"
20
+ $PY $D/scale_bench_v4.py > logs/bench_v4_last.log 2>&1; log "banc v4 mis à jour"
21
+ if [ $rc = 0 ]; then
22
+ [ -f $G/logs/gen_$tag.log ] || (cd $G && nice -n 10 timeout 7200 $PY -u $G/gen_eval.py "$ck" "$corp" $tag > $G/logs/gen_$tag.log 2>&1); log "$tag génération EXIT $?"
23
+ echo "$sc $ok" > logs/v4_$tag.done
24
+ fi
25
+ done < jobs_v4.tsv
26
+ sleep 120
27
+ done
v3_scale/code/v3/audit_overlap.py ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — chevauchement 64 o des fenêtres d'éval (définitions eval_ref_v3) avec old/broad train.bin, par position d'éval.
3
+ Position t (cible = patch t+1) « contaminée » si un octet du patch cible est couvert par une fenêtre de 64 o présente dans train.
4
+ Recalcule fused_acc_invocab des réfs (scale/hits) sur positions propres. Usage : audit_overlap.py CORPUS (old_local_v1|broad_local_v1)"""
5
+ import sys, os, glob, json, numpy as np
6
+ sys.path.insert(0, "/workspace/SPIRA/python"); sys.path.insert(0, "/workspace/spira_shared/checker/fc2"); sys.path.insert(0, "/workspace/spira_shared/checker")
7
+ from rollhash import hashes
8
+ from spira.model import SPIRA
9
+ from windows_v2 import freshcode_v2
10
+ from build_corpus_local import strip_gutenberg_header
11
+ c = sys.argv[1]; C = f"/workspace/spira_shared/broad/{c}"; T = "/workspace/repos/toroidal-fractal-intelligence/data"; rd = lambda p: open(p, "rb").read()
12
+ tr = rd(f"{C}/train.bin"); th = np.unique(np.concatenate([hashes(tr[s:s+4_000_063]) for s in range(0, len(tr), 4_000_000)]))
13
+ W = {"fresh_en": (rd(f"{C}/oneshot.bin"), 8000), "fresh_fr": (strip_gutenberg_header(rd(f"{T}/contes_perrault.txt")), 8000),
14
+ "neither_code": (b"\n\n".join(rd(p) for p in sorted(glob.glob("/workspace/kahnn/data/code/*"))), 8000),
15
+ "test20k": (rd(f"{C}/test.bin"), 20000), "queries": (rd(f"{T}/chat_seed_fr_en.txt"), 8000), "freshcode_v2": (freshcode_v2(), 8000)}
16
+ m = SPIRA.load(f"/workspace/spira_shared/checker/ckpt/ref_fixed2cal_{c}.npz")
17
+ D = "/workspace/spira_shared/checker/scale/hits"; out = {}
18
+ for k, (b, mp) in W.items():
19
+ h = hashes(b); bad = np.isin(h, th); cov = np.zeros(len(b) + 1, np.int32)
20
+ for i in np.nonzero(bad)[0]: cov[i] += 1; cov[i + 64] -= 1
21
+ covered = np.cumsum(cov)[:len(b)] > 0
22
+ pats = m.ept.segment(b); off = np.cumsum([0] + [len(p) for p in pats])
23
+ n = min(len(pats) - 1, mp); cont = np.array([covered[off[t+1]:off[t+2]].any() for t in range(n)])
24
+ r = {"windows64": int(len(h)), "windows64_overlap": int(bad.sum()), "bytes_covered": int(covered.sum()), "positions": n, "positions_contaminated": int(cont.sum())}
25
+ hp = f"{D}/ref_fixed2cal_{c}__{c}__{k}.npz"
26
+ if os.path.exists(hp):
27
+ z = np.load(hp); fh, iv = z["fused_hit"][:n], z["invocab"][:n]; cl = ~cont
28
+ r["ref_fused_invocab_all"] = 100 * float(fh[iv].mean()); r["ref_fused_invocab_clean"] = 100 * float(fh[iv & cl].mean()) if (iv & cl).any() else None
29
+ r["ref_fused_invocab_contaminated"] = 100 * float(fh[iv & cont].mean()) if (iv & cont).any() else None
30
+ np.save(f"{os.path.dirname(os.path.abspath(__file__))}/contam_{c}__{k}.npy", cont); out[k] = r; print(k, json.dumps(r), flush=True)
31
+ json.dump(out, open(f"{os.path.dirname(os.path.abspath(__file__))}/audit_overlap_{c}.json", "w"), indent=1)
v3_scale/code/v3/build_v3_lowmem.py ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Checker — corpus_v3 = broad_local_v1/train.bin + livres corpus_gutenberg + livres Project Gutenberg téléchargés (corpus_v3/raw, EN+FR).
3
+ 1) en-têtes Gutenberg retirés (strip_gutenberg_header) ; 2) dédup exacte par paragraphe (sha1, \n\n) ; 3) held-out : 2 livres entiers
4
+ (1 EN, 1 FR, choisis par sha256 de l'id) → test.bin ; oneshot.bin = broad_local_v1/oneshot.bin (fresh_en inchangé) ;
5
+ 4) suppression dans train de TOUTE fenêtre de 64 o présente dans une fenêtre d'éval (spira_m7/windows/*, freshcode_v2, contes_perrault,
6
+ kahnn code, chat_seed, test old/broad, oneshot old/broad, test/oneshot v3) — hash exact fc2/rollhash, octets couverts coupés, itéré jusqu'à 0.
7
+ Sorties /workspace/spira_shared/corpus_v3/{train,test,oneshot}.bin + corpus_manifest.json (= manifest.json)."""
8
+ import sys, os, glob, json, hashlib, re, numpy as np
9
+ sys.path.insert(0, "/workspace/spira_shared/checker/fc2"); sys.path.insert(0, "/workspace/spira_shared/checker")
10
+ from rollhash import hashes
11
+ from windows_v2 import freshcode_v2
12
+ from build_corpus_local import strip_gutenberg_header
13
+ S = "/workspace/spira_shared"; O = f"{S}/corpus_v3"; B = f"{S}/broad"; T = "/workspace/repos/toroidal-fractal-intelligence/data"
14
+ rd = lambda p: open(p, "rb").read(); log = lambda *a: print(*a, flush=True)
15
+ def lang(b):
16
+ m = re.search(rb"^Language: *(\w+)", b[:5000], re.M); return (m.group(1).decode().lower() if m else "?")
17
+ books = {}
18
+ for p in sorted(glob.glob(f"{O}/raw/pg*.txt")):
19
+ b = rd(p)
20
+ if b"*** START OF" not in b: log("skip (pas un ebook)", p); continue
21
+ l = lang(b)
22
+ if l not in ("english", "french"): log("skip langue", l, p); continue
23
+ books[os.path.basename(p)] = (l, strip_gutenberg_header(b))
24
+ for p in sorted(glob.glob(f"{S}/corpus_gutenberg/corpus/books/*.txt")):
25
+ b = rd(p); books["cg_" + os.path.basename(p)] = ("french" if "_fr" in p else "english", strip_gutenberg_header(b) if b"*** START" in b else b)
26
+ key = lambda n: hashlib.sha256(n.encode()).hexdigest()
27
+ ho = [min((n for n in books if n.startswith("pg") and books[n][0] == L), key=key) for L in ("english", "french")]
28
+ test = b"\n\n".join(books[n][1] for n in ho); oneshot = rd(f"{B}/broad_local_v1/oneshot.bin")
29
+ log("held-out", ho, len(test))
30
+ seen_par, parts, comp, dropped_dup = set(), [], {"broad_local_v1": 0, "english": 0, "french": 0}, 0
31
+ def add(src, b):
32
+ global dropped_dup
33
+ for par in re.split(rb"\n\s*\n", b):
34
+ if len(par.strip()) == 0: continue
35
+ h = hashlib.sha1(par.strip()).digest()
36
+ if h in seen_par: dropped_dup += len(par); continue
37
+ seen_par.add(h); parts.append(par); comp[src] += len(par) + 2
38
+ add("broad_local_v1", rd(f"{B}/broad_local_v1/train.bin"))
39
+ for n in sorted(books):
40
+ if n not in ho: add(books[n][0], books[n][1])
41
+ train = bytearray(b"\n\n".join(parts)); del parts, seen_par; books = {n: (v[0], b"") for n, v in books.items()}; log("train avant filtre", len(train), "doublons paragraphes", dropped_dup)
42
+ W = {os.path.basename(p)[:-4]: rd(p) for p in glob.glob(f"{S}/spira_m7/windows/*.bin") if "seen" not in p}
43
+ W.update({"freshcode_v2": freshcode_v2(), "contes_perrault": strip_gutenberg_header(rd(f"{T}/contes_perrault.txt")),
44
+ "kahnn_code": b"\n\n".join(rd(p) for p in sorted(glob.glob("/workspace/kahnn/data/code/*"))),
45
+ "chat_seed": rd(f"{T}/chat_seed_fr_en.txt"), "test_old": rd(f"{B}/old_local_v1/test.bin"), "test_broad": rd(f"{B}/broad_local_v1/test.bin"),
46
+ "oneshot_old": rd(f"{B}/old_local_v1/oneshot.bin"), "oneshot_broad": oneshot, "test_v3": test})
47
+ E = np.unique(np.concatenate([hashes(b) for b in W.values()])); log("fenêtres d'éval uniques", len(E))
48
+ def bad_mask(buf, CH=4_000_000):
49
+ a = bytes(buf); n = len(a); cut = np.zeros(n, bool); nb = 0
50
+ for s in range(0, max(1, n - 63), CH):
51
+ h = hashes(a[s:s + CH + 63]); hit = np.nonzero(np.isin(h, E))[0]; nb += len(hit)
52
+ for i in hit: cut[s + i:s + i + 64] = True
53
+ return cut, nb
54
+ it, removed = 0, 0
55
+ while True:
56
+ cut, nb = bad_mask(train); log("passe", it, "fenêtres en chevauchement", nb)
57
+ if nb == 0: break
58
+ a = np.frombuffer(bytes(train), np.uint8); keep = ~cut; removed += int(cut.sum())
59
+ # chaque segment coupé est remplacé par un saut de paragraphe
60
+ starts = np.nonzero(cut & ~np.r_[False, cut[:-1]])[0]; out = a[keep]
61
+ ins = np.searchsorted(np.nonzero(keep)[0], starts)
62
+ train = bytearray(np.insert(out, ins, np.uint8(10)).tobytes()); it += 1
63
+ train = bytes(train); del cut
64
+ # vérif finale par fenêtre
65
+ # SPIRA Relay 16 h 27 : vérif finale par blocs (l'ancienne ligne concaténait+np.unique tous les hashes du train -> OOM 6,3 Go)
66
+ HW = {k: np.unique(hashes(b)) for k, b in W.items()}; found = {k: np.zeros(len(v), bool) for k, v in HW.items()}
67
+ for s in range(0, len(train), 4_000_000):
68
+ hc = np.unique(hashes(train[s:s+4_000_063]))
69
+ for k in HW: found[k] |= np.isin(HW[k], hc)
70
+ del hc
71
+ ov = {k: int(found[k].sum()) for k in HW}; del HW, found
72
+ log("chevauchement final", ov); assert all(v == 0 for v in ov.values())
73
+ os.makedirs(O, exist_ok=True)
74
+ for n, b in (("train", train), ("test", test), ("oneshot", oneshot)): open(f"{O}/{n}.bin", "wb").write(b)
75
+ man = {"variant": "corpus_v3", "builder": "checker/v3/build_v3.py", "train_bytes": len(train), "test_bytes": len(test), "oneshot_bytes": len(oneshot),
76
+ "composition_pre_filter": comp, "dup_paragraph_bytes_dropped": dropped_dup, "overlap_bytes_removed": removed, "filter_passes": it,
77
+ "heldout_books": ho, "n_books": len(books), "books": {n: books[n][0] for n in sorted(books)}, "overlap64_final": ov,
78
+ "sha256": {n: hashlib.sha256(rd(f"{O}/{n}.bin")).hexdigest() for n in ("train", "test", "oneshot")},
79
+ "raw_sha256": {os.path.basename(p): hashlib.sha256(rd(p)).hexdigest() for p in sorted(glob.glob(f"{O}/raw/*.txt"))}}
80
+ for f in ("corpus_manifest.json", "manifest.json"): json.dump(man, open(f"{O}/{f}", "w"), indent=1, ensure_ascii=False)
81
+ log("OK", json.dumps({k: man[k] for k in ("train_bytes", "test_bytes", "overlap_bytes_removed", "filter_passes", "sha256")}))
v3_scale/code/v3verify/verify_v3.py ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """SPIRA Hashing - verification independante (lecture seule) d'un corpus: tailles, sha256, overlap 64 o.
2
+ Usage: verify_v3.py CORPUS_DIR OUT_JSON"""
3
+ import sys, os, json, hashlib, numpy as np
4
+ sys.path.insert(0, "/workspace/spira_shared/checker")
5
+ C, OUT = sys.argv[1:3]; K = 64; P = np.uint64(1099511628211)
6
+ def hashes(b):
7
+ a = np.frombuffer(b, np.uint8).astype(np.uint64); n = len(a) - K + 1
8
+ if n <= 0: return np.zeros(0, np.uint64)
9
+ h = np.zeros(n, np.uint64)
10
+ with np.errstate(over="ignore"):
11
+ for j in range(K): h = h * P + a[j:j+n] + np.uint64(1)
12
+ return h
13
+ W = {k: open(f"/workspace/spira_shared/spira_m7/windows/{k}.bin","rb").read() for k in ("fresh_en","fresh_fr","neither_code","test20k","queries")}
14
+ try:
15
+ from windows_v2 import freshcode_v2; W["fc2"] = freshcode_v2()
16
+ except Exception as e: W["fc2_ERROR"] = b""; print("fc2 indisponible", e)
17
+ F = {f: open(f"{C}/{f}.bin","rb").read() for f in ("train","test","oneshot") if os.path.exists(f"{C}/{f}.bin")}
18
+ for _k in ("test","oneshot"):
19
+ if _k in F: W["corpus_"+_k] = F[_k]
20
+ rep = {"corpus": C, "files": {f: {"bytes": len(b), "sha256": hashlib.sha256(b).hexdigest()} for f, b in F.items()}, "overlap64": {}}
21
+ ht = hashes(F["train"]); hs = np.sort(ht)
22
+ for k, b in W.items():
23
+ hw = hashes(b); u = np.unique(hw)
24
+ i = np.searchsorted(hs, u); i = np.minimum(i, len(hs)-1); cand = u[hs[i] == u]
25
+ # confirmation octet par octet (anti-collision)
26
+ tset = set(cand.tolist()); pos_t = {}
27
+ for p in np.nonzero(np.isin(ht, cand))[0]: pos_t.setdefault(int(ht[p]), []).append(int(p))
28
+ conf = 0
29
+ for p in np.nonzero(np.isin(hw, cand))[0]:
30
+ seg = b[p:p+K]
31
+ if any(F["train"][q:q+K] == seg for q in pos_t.get(int(hw[p]), ())): conf += 1
32
+ rep["overlap64"][k] = {"window_bytes": len(b), "n_64grams": int(len(hw)), "hash_hits": int(np.isin(hw, cand).sum()), "confirmed_positions": conf}
33
+ print(k, rep["overlap64"][k], flush=True)
34
+ rep["valid_zero_overlap"] = all(v["confirmed_positions"] == 0 for v in rep["overlap64"].values()) and "fc2_ERROR" not in W
35
+ m = f"{C}/corpus_manifest.json" if os.path.exists(f"{C}/corpus_manifest.json") else f"{C}/manifest.json"
36
+ if os.path.exists(m): rep["manifest_checker"] = json.load(open(m))
37
+ json.dump(rep, open(OUT, "w"), indent=1); print("VALID" if rep["valid_zero_overlap"] else "INVALID")
v3_scale/corpus_v3/corpus_manifest.json ADDED
@@ -0,0 +1,227 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "corpus_v3",
3
+ "builder": "checker/v3/build_v3.py",
4
+ "train_bytes": 93284487,
5
+ "test_bytes": 3486653,
6
+ "oneshot_bytes": 119763,
7
+ "composition_pre_filter": {
8
+ "broad_local_v1": 34443642,
9
+ "english": 41663184,
10
+ "french": 18630849
11
+ },
12
+ "dup_paragraph_bytes_dropped": 10514511,
13
+ "overlap_bytes_removed": 1463063,
14
+ "filter_passes": 1,
15
+ "heldout_books": [
16
+ "pg1184.txt",
17
+ "pg8712.txt"
18
+ ],
19
+ "n_books": 97,
20
+ "books": {
21
+ "cg_alice.txt": "english",
22
+ "cg_candide_fr.txt": "french",
23
+ "cg_dorian_gray.txt": "english",
24
+ "cg_dracula.txt": "english",
25
+ "cg_frankenstein.txt": "english",
26
+ "cg_great_expectations.txt": "english",
27
+ "cg_les_mis_fr.txt": "french",
28
+ "cg_metamorphosis.txt": "english",
29
+ "cg_moby_dick.txt": "english",
30
+ "cg_pride_prejudice.txt": "english",
31
+ "cg_sherlock.txt": "english",
32
+ "cg_treasure_island.txt": "english",
33
+ "cg_two_cities.txt": "english",
34
+ "pg1080.txt": "english",
35
+ "pg1184.txt": "english",
36
+ "pg120.txt": "english",
37
+ "pg1232.txt": "english",
38
+ "pg1257.txt": "english",
39
+ "pg1260.txt": "english",
40
+ "pg1342.txt": "english",
41
+ "pg135.txt": "english",
42
+ "pg13735.txt": "french",
43
+ "pg13846.txt": "french",
44
+ "pg13951.txt": "french",
45
+ "pg1400.txt": "english",
46
+ "pg14155.txt": "french",
47
+ "pg145.txt": "english",
48
+ "pg1497.txt": "english",
49
+ "pg15200.txt": "english",
50
+ "pg158.txt": "english",
51
+ "pg161.txt": "english",
52
+ "pg16328.txt": "english",
53
+ "pg1661.txt": "english",
54
+ "pg16901.txt": "french",
55
+ "pg1727.txt": "english",
56
+ "pg17489.txt": "french",
57
+ "pg17493.txt": "french",
58
+ "pg17494.txt": "french",
59
+ "pg17808.txt": "french",
60
+ "pg17989.txt": "french",
61
+ "pg17990.txt": "french",
62
+ "pg17991.txt": "french",
63
+ "pg17992.txt": "french",
64
+ "pg18143.txt": "french",
65
+ "pg1952.txt": "english",
66
+ "pg19657.txt": "french",
67
+ "pg205.txt": "english",
68
+ "pg215.txt": "english",
69
+ "pg219.txt": "english",
70
+ "pg23.txt": "english",
71
+ "pg2554.txt": "english",
72
+ "pg2591.txt": "english",
73
+ "pg2600.txt": "english",
74
+ "pg2650.txt": "french",
75
+ "pg26685.txt": "french",
76
+ "pg2701.txt": "english",
77
+ "pg28054.txt": "english",
78
+ "pg2814.txt": "english",
79
+ "pg2998.txt": "french",
80
+ "pg30254.txt": "english",
81
+ "pg3207.txt": "english",
82
+ "pg345.txt": "english",
83
+ "pg35.txt": "english",
84
+ "pg36.txt": "english",
85
+ "pg3600.txt": "english",
86
+ "pg37106.txt": "english",
87
+ "pg408.txt": "english",
88
+ "pg41211.txt": "french",
89
+ "pg42.txt": "english",
90
+ "pg42131.txt": "french",
91
+ "pg43.txt": "english",
92
+ "pg4300.txt": "english",
93
+ "pg4363.txt": "english",
94
+ "pg46.txt": "english",
95
+ "pg4650.txt": "french",
96
+ "pg4791.txt": "french",
97
+ "pg4968.txt": "french",
98
+ "pg5097.txt": "french",
99
+ "pg514.txt": "english",
100
+ "pg51516.txt": "french",
101
+ "pg5200.txt": "english",
102
+ "pg55.txt": "english",
103
+ "pg5711.txt": "french",
104
+ "pg6130.txt": "english",
105
+ "pg64317.txt": "english",
106
+ "pg6838.txt": "french",
107
+ "pg7196.txt": "english",
108
+ "pg74.txt": "english",
109
+ "pg76.txt": "english",
110
+ "pg768.txt": "english",
111
+ "pg799.txt": "french",
112
+ "pg844.txt": "english",
113
+ "pg8712.txt": "french",
114
+ "pg8800.txt": "english",
115
+ "pg9645.txt": "french",
116
+ "pg98.txt": "english",
117
+ "pg996.txt": "english"
118
+ },
119
+ "overlap64_final": {
120
+ "fresh_fr": 0,
121
+ "freshcode": 0,
122
+ "neither_code": 0,
123
+ "test20k": 0,
124
+ "fresh_en": 0,
125
+ "queries": 0,
126
+ "freshcode_v2": 0,
127
+ "contes_perrault": 0,
128
+ "kahnn_code": 0,
129
+ "chat_seed": 0,
130
+ "test_old": 0,
131
+ "test_broad": 0,
132
+ "oneshot_old": 0,
133
+ "oneshot_broad": 0,
134
+ "test_v3": 0
135
+ },
136
+ "sha256": {
137
+ "train": "aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c",
138
+ "test": "a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84",
139
+ "oneshot": "67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a"
140
+ },
141
+ "raw_sha256": {
142
+ "pg1080.txt": "331340f2ada84a7f79d756a344f6ce7d78026cd1d2359d33f79ca4a13d9f1519",
143
+ "pg1184.txt": "64f8d5cfa51fcecb904abf7312d395d512a71817e7359b91288beb50517c3836",
144
+ "pg120.txt": "5bc08275eacea8640b1aa185c645283bb8c564d7b23df6b03437d229b4fc6ebd",
145
+ "pg1232.txt": "33496a305f7a4b933784fb5aec95f920fbc3d4687794817e4cc444aa6578d2e4",
146
+ "pg1257.txt": "1682d060845717283d22462d1d884f79c2741ecdba3cfe5ceed700602ccf0e75",
147
+ "pg1260.txt": "13414dee2951c3ee731d76d2ffd822016b2479c892162760c5d0eb2aa5fa7631",
148
+ "pg1342.txt": "3f6bb9d6f78e0293b56acd4714dd68cb7d6d1d293402031ce9d5a216bcaf9d75",
149
+ "pg135.txt": "6cf3b9d6fc5e6f733737425267d3af296d7f99f9b9fd1d255a3ab7689a6346ab",
150
+ "pg13735.txt": "782a78ce6e52f1cadb894bc871ffd1330861e1a2db4bcf3da0fb325d62c72469",
151
+ "pg13846.txt": "4e3c7f3329d591408dca1ebe0818cd900622dcf6d090e16240bf0bbfac07370c",
152
+ "pg13951.txt": "46c01913219e6ff165256b8cbcaec7034b004f785a147341911b3caee70ccfcc",
153
+ "pg1400.txt": "9a637118af8e953e9764ec603d9b0a032883384d465acac2e27966a80cf1c6f8",
154
+ "pg14155.txt": "93f0a2259dbaf5fde6e7b96adc22346d52f80eb1e9319336c8381a045efe40cd",
155
+ "pg145.txt": "dc2e0107e6ae07e8e33da934b2ec4e8e600826a2e0cd48df265119cd02e5fb50",
156
+ "pg1497.txt": "3d2e397122881d7737a48fb80e71cddf4c78cda54ceac7e0905fe945dfab48fd",
157
+ "pg15200.txt": "7145bcdb997fa09b56acc750031dd7c5fecf26f198475041dac9c179b1890f96",
158
+ "pg158.txt": "532b122b4e6a76cc556d6fdcd729b5892f5c4ce4a1b7060b9f832adfad8680dc",
159
+ "pg161.txt": "7bcd77c45cb32fb8d9f89402561270cc6a911c0a4e20667176d5be779fc3974a",
160
+ "pg16328.txt": "8909085dc48daf123d29c40ecbb3837b16ba7f96bf0cbb9fe72de133a2d35348",
161
+ "pg1661.txt": "922e2a12ccb43a4c9544c260b2166c6ad2097aeb5957faeee113f173bb857cd0",
162
+ "pg16901.txt": "7d264a3faf62bc4e0e37da29d1d2b07c022bb2e0565ee0e6e6b534ff30422290",
163
+ "pg1727.txt": "ffbdb29c3dda284b65c11243db1a98167826a70c81bca2ed4b6232f86c905fb9",
164
+ "pg17489.txt": "a5de514ba7b9f2e1790e7e259c4e8b7a35ae1d29e4bf9a5f8767039c58b80503",
165
+ "pg17493.txt": "ffddbcee83e3c537b5586bfbbaf97ea6fab2b04250b9efaa63c027ab20aaffd0",
166
+ "pg17494.txt": "79894e327a20044dad5b93dd7d1fae0a8c06b74b86cd9f19d37bb54ffbacd44f",
167
+ "pg17808.txt": "401ad78cdcb606765f48f7b478c681d73b49d0c3666a1f6044d1a38564a01c4c",
168
+ "pg17989.txt": "9ea7a8b127a9dfa36de0b0cf9c2e26293beea5c37a0fc988fb9308b63160f282",
169
+ "pg17990.txt": "d3df81d687dfd2ba8e7d219287e767b659c189657c51a3f5433e1ba71f0d0613",
170
+ "pg17991.txt": "49cacb5455e93bebcd744a572447819a9a7f7161fb3516b5fdccbac7a4e2f571",
171
+ "pg17992.txt": "abaf80e7672d909e8adee9cf79ff18b6d79f785ac209cf44d7c06526ed627bd9",
172
+ "pg18143.txt": "dec49c928b81fddc0016e6be2def4045880142a272d299e5fe7acdcd11455fff",
173
+ "pg1952.txt": "20c7788eeae2fe3a0efd51a1f08b6723366a4be0778d65db219149a3522651d4",
174
+ "pg19657.txt": "91e3524d6e7816cda822f97fae3f49af204c7e32ad71ca23002de9e83e935f80",
175
+ "pg205.txt": "2d9a76a2e3e8195c69430516ebd33c4d0757a53ad432ff6186b7b794e6fe99f9",
176
+ "pg215.txt": "30f569c5671bba7bbc4a92ab6d8ee01a0d3be00f994819c77da62c28041eb8a2",
177
+ "pg219.txt": "1bcdf0a1ac611dd6b70a20d100a07085754da4e90a949d6031717b86cd6d020a",
178
+ "pg23.txt": "234c15348a66919bad1d534cdd48ee8ddf91f50115a706cbefaa8edd049672fd",
179
+ "pg2554.txt": "4a84a709a6e355f5fa555c7276c8a90125eb9d06f63bc8663410c3d1297eb688",
180
+ "pg2591.txt": "377ebb8ca06cd8bc0b1015ddb54766c44f50dece731c53bb030f1cc00f54092a",
181
+ "pg2600.txt": "2d5bb2ad5f422765e714617e21fa31bbaf8958aa79682c86fca6660fcc5d1b2b",
182
+ "pg2650.txt": "9ebdb837f733f41d6a822e010d1ccbf14b4ddafad0131482461edb72b55b0936",
183
+ "pg26685.txt": "b838eca71b85dd32b8ef0389d9b8a0a3fec41b85e060a1622f718da418760c46",
184
+ "pg2701.txt": "b5e88b71a1cda068e9ab1d94344c9d932338ba6da1a66def97a1aecd9cfcc87d",
185
+ "pg28054.txt": "2e75c187cb750df3f7c910fa90970b264b9a4fee212be2a73c6bb507d9ad22fe",
186
+ "pg2814.txt": "6be7d8400ac2549459307d1c1a20a25c86a736984269d3e9d989c817814be60c",
187
+ "pg2998.txt": "d23f9d231eed199a7ebab51af574a36dde00662b0310501b874b9f36db1bdd2f",
188
+ "pg30254.txt": "a0084df9930e6b7fb088a845a94fa8ae460aafa0f6a751a0f2d1b32ff3dccd2a",
189
+ "pg3207.txt": "3de1e492641d939567a8b0de827fb13e1ad992324f9ac8b204f60475009b7294",
190
+ "pg345.txt": "96cd16eacdbfebae8fdda5591f66e0cc8ee76be18e0cd1aca02bc00615782d28",
191
+ "pg35.txt": "2892e919000e17c83e1dac51b30f4675db50536b644d7579fe8a89bb399a9bdc",
192
+ "pg36.txt": "8417469e3ab664749f7b36ebb85799075818207b0380b54c27d07d592a44117d",
193
+ "pg3600.txt": "1b4c87312f0890e04cecee48e3a5fa263de65743230b819e16cb1bd72f7aee59",
194
+ "pg37106.txt": "494309c7c4b1a750f19bb69258760a602032ebb0a52955f5153818b3508d5fe0",
195
+ "pg408.txt": "87f8e832484792b761a510b953d67e3835fa46a9f4231a508bfeb914a98559ca",
196
+ "pg41211.txt": "c192ed74d4ffd4000497033b8a512e9f5ca805531281d5395259f954c12e30d2",
197
+ "pg42.txt": "6f5ebfb79a24bb24e88c8f7e5f35515321e545bf4f137fce6e406c9da0bb2b59",
198
+ "pg42131.txt": "bfb7cfb2ae429b4b408029accc0774ebae9875673f2b0130d51b14eb57008f3b",
199
+ "pg43.txt": "b43448a88391591f9cf82b25553df00faf47a2752a873185fcdd1156eebdb990",
200
+ "pg4300.txt": "e03094626f9528cf3fc287a49d6edbdbf47cd40483cb13ceb301e74e15ebbf9e",
201
+ "pg4363.txt": "6ee088f717d3760036086f108c03d57588036c9d344b7d71edbef4e631bd374a",
202
+ "pg46.txt": "8c36b93723d454982df045ec0efdf2664e5a48e6b5e2f670e883575f75ad0cc8",
203
+ "pg4650.txt": "1cc10ab3268a76aa01bd3f9722ae2ae0e5e36b869a18b0bb0832b75bc0327806",
204
+ "pg4791.txt": "cc2c572afc2b2965e252ed8626c4e023b54e272f3a83594f10bc8d7bc6a9f5d6",
205
+ "pg4968.txt": "22bbbb7f1a83c1fe457ef52ea1c00ff95461e15f62c0f5fd70130fbf82ae32e4",
206
+ "pg5097.txt": "53507b025cd3580f4fbfa546baabb36f25adfb48b581c5b5016d91ad69d36c68",
207
+ "pg514.txt": "36831477b339dbcd7e7206119d77e9e58b9727ef3c299d1a1dc4446e1db97850",
208
+ "pg51516.txt": "0c70c8e7e7ce5a9adda520495b074f486af3681c57cc21ba8556dc831aa0df04",
209
+ "pg5200.txt": "0933283a19a4daaef0d412e8f69c74fae126b0a946cb74b8f3c783a8e869830d",
210
+ "pg55.txt": "969bffab7740d4d8a0bac332d78bd0152ad4a40a2efc52f06c74a9bb6120be75",
211
+ "pg5711.txt": "7bbd0d116a119b0626b88d7f8e7e5196831e348395249df6d0ec61abc30e583b",
212
+ "pg6130.txt": "ca4f23115b384b0219427ef208e875f0351820b8f80f28259272997ea581c36b",
213
+ "pg64317.txt": "ce760ec377accd352b41bb8f64504a72d7aa18ab3afb42ded2b56cecacf29e35",
214
+ "pg6838.txt": "c5c049d44e8b44ca64c388396fb0ef270c14f7865554c6fb2cb2102c35f7e77b",
215
+ "pg7196.txt": "71e7f6fa3dd9c0f25008fe7cf176ce1b8d99126be50fdd61da6daf290540a8be",
216
+ "pg74.txt": "74d77384b123a6360db9ab58463cff8b38df8525fbdc6000c81ee388e1f3cf10",
217
+ "pg76.txt": "d617a37aa7ae1e1a93dcde2634db2bccb86824e31e29a55230bdbf77d6872d59",
218
+ "pg768.txt": "e533fe750589f0421d5d744576315f5c2b9b0d69e981179ea0551bbf134c5e02",
219
+ "pg799.txt": "b647a30294f786cc8b502748fd1d1dd17486fadc16ff4d91147b4a7b2402d270",
220
+ "pg844.txt": "1b8a58099bb1cdef6a845277a4bacf2f4a268702c165bde30124d4b5105d1851",
221
+ "pg8712.txt": "fbafaf23e91de6c540ac18e56613a0c8553fcff42842dad1c6ae02438f09d8a0",
222
+ "pg8800.txt": "3a7dd97b5fec82456c58237b33383a593480c2bfe088aeaaa4a519de2a10d39c",
223
+ "pg9645.txt": "1fd16ddf201decd5b376a27dc1576ff12fcfcf835ad344b26100d6d70071324c",
224
+ "pg98.txt": "d54c2b80d40a40b982cd88852c6180bb944d95acdb028af3d0e01a1750681784",
225
+ "pg996.txt": "2864143e9addf498c1c5fd585db4a1ff3e06651c344cddf0931342f1a0ce24d4"
226
+ }
227
+ }
v3_scale/corpus_v3/manifest.json ADDED
@@ -0,0 +1,227 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "corpus_v3",
3
+ "builder": "checker/v3/build_v3.py",
4
+ "train_bytes": 93284487,
5
+ "test_bytes": 3486653,
6
+ "oneshot_bytes": 119763,
7
+ "composition_pre_filter": {
8
+ "broad_local_v1": 34443642,
9
+ "english": 41663184,
10
+ "french": 18630849
11
+ },
12
+ "dup_paragraph_bytes_dropped": 10514511,
13
+ "overlap_bytes_removed": 1463063,
14
+ "filter_passes": 1,
15
+ "heldout_books": [
16
+ "pg1184.txt",
17
+ "pg8712.txt"
18
+ ],
19
+ "n_books": 97,
20
+ "books": {
21
+ "cg_alice.txt": "english",
22
+ "cg_candide_fr.txt": "french",
23
+ "cg_dorian_gray.txt": "english",
24
+ "cg_dracula.txt": "english",
25
+ "cg_frankenstein.txt": "english",
26
+ "cg_great_expectations.txt": "english",
27
+ "cg_les_mis_fr.txt": "french",
28
+ "cg_metamorphosis.txt": "english",
29
+ "cg_moby_dick.txt": "english",
30
+ "cg_pride_prejudice.txt": "english",
31
+ "cg_sherlock.txt": "english",
32
+ "cg_treasure_island.txt": "english",
33
+ "cg_two_cities.txt": "english",
34
+ "pg1080.txt": "english",
35
+ "pg1184.txt": "english",
36
+ "pg120.txt": "english",
37
+ "pg1232.txt": "english",
38
+ "pg1257.txt": "english",
39
+ "pg1260.txt": "english",
40
+ "pg1342.txt": "english",
41
+ "pg135.txt": "english",
42
+ "pg13735.txt": "french",
43
+ "pg13846.txt": "french",
44
+ "pg13951.txt": "french",
45
+ "pg1400.txt": "english",
46
+ "pg14155.txt": "french",
47
+ "pg145.txt": "english",
48
+ "pg1497.txt": "english",
49
+ "pg15200.txt": "english",
50
+ "pg158.txt": "english",
51
+ "pg161.txt": "english",
52
+ "pg16328.txt": "english",
53
+ "pg1661.txt": "english",
54
+ "pg16901.txt": "french",
55
+ "pg1727.txt": "english",
56
+ "pg17489.txt": "french",
57
+ "pg17493.txt": "french",
58
+ "pg17494.txt": "french",
59
+ "pg17808.txt": "french",
60
+ "pg17989.txt": "french",
61
+ "pg17990.txt": "french",
62
+ "pg17991.txt": "french",
63
+ "pg17992.txt": "french",
64
+ "pg18143.txt": "french",
65
+ "pg1952.txt": "english",
66
+ "pg19657.txt": "french",
67
+ "pg205.txt": "english",
68
+ "pg215.txt": "english",
69
+ "pg219.txt": "english",
70
+ "pg23.txt": "english",
71
+ "pg2554.txt": "english",
72
+ "pg2591.txt": "english",
73
+ "pg2600.txt": "english",
74
+ "pg2650.txt": "french",
75
+ "pg26685.txt": "french",
76
+ "pg2701.txt": "english",
77
+ "pg28054.txt": "english",
78
+ "pg2814.txt": "english",
79
+ "pg2998.txt": "french",
80
+ "pg30254.txt": "english",
81
+ "pg3207.txt": "english",
82
+ "pg345.txt": "english",
83
+ "pg35.txt": "english",
84
+ "pg36.txt": "english",
85
+ "pg3600.txt": "english",
86
+ "pg37106.txt": "english",
87
+ "pg408.txt": "english",
88
+ "pg41211.txt": "french",
89
+ "pg42.txt": "english",
90
+ "pg42131.txt": "french",
91
+ "pg43.txt": "english",
92
+ "pg4300.txt": "english",
93
+ "pg4363.txt": "english",
94
+ "pg46.txt": "english",
95
+ "pg4650.txt": "french",
96
+ "pg4791.txt": "french",
97
+ "pg4968.txt": "french",
98
+ "pg5097.txt": "french",
99
+ "pg514.txt": "english",
100
+ "pg51516.txt": "french",
101
+ "pg5200.txt": "english",
102
+ "pg55.txt": "english",
103
+ "pg5711.txt": "french",
104
+ "pg6130.txt": "english",
105
+ "pg64317.txt": "english",
106
+ "pg6838.txt": "french",
107
+ "pg7196.txt": "english",
108
+ "pg74.txt": "english",
109
+ "pg76.txt": "english",
110
+ "pg768.txt": "english",
111
+ "pg799.txt": "french",
112
+ "pg844.txt": "english",
113
+ "pg8712.txt": "french",
114
+ "pg8800.txt": "english",
115
+ "pg9645.txt": "french",
116
+ "pg98.txt": "english",
117
+ "pg996.txt": "english"
118
+ },
119
+ "overlap64_final": {
120
+ "fresh_fr": 0,
121
+ "freshcode": 0,
122
+ "neither_code": 0,
123
+ "test20k": 0,
124
+ "fresh_en": 0,
125
+ "queries": 0,
126
+ "freshcode_v2": 0,
127
+ "contes_perrault": 0,
128
+ "kahnn_code": 0,
129
+ "chat_seed": 0,
130
+ "test_old": 0,
131
+ "test_broad": 0,
132
+ "oneshot_old": 0,
133
+ "oneshot_broad": 0,
134
+ "test_v3": 0
135
+ },
136
+ "sha256": {
137
+ "train": "aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c",
138
+ "test": "a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84",
139
+ "oneshot": "67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a"
140
+ },
141
+ "raw_sha256": {
142
+ "pg1080.txt": "331340f2ada84a7f79d756a344f6ce7d78026cd1d2359d33f79ca4a13d9f1519",
143
+ "pg1184.txt": "64f8d5cfa51fcecb904abf7312d395d512a71817e7359b91288beb50517c3836",
144
+ "pg120.txt": "5bc08275eacea8640b1aa185c645283bb8c564d7b23df6b03437d229b4fc6ebd",
145
+ "pg1232.txt": "33496a305f7a4b933784fb5aec95f920fbc3d4687794817e4cc444aa6578d2e4",
146
+ "pg1257.txt": "1682d060845717283d22462d1d884f79c2741ecdba3cfe5ceed700602ccf0e75",
147
+ "pg1260.txt": "13414dee2951c3ee731d76d2ffd822016b2479c892162760c5d0eb2aa5fa7631",
148
+ "pg1342.txt": "3f6bb9d6f78e0293b56acd4714dd68cb7d6d1d293402031ce9d5a216bcaf9d75",
149
+ "pg135.txt": "6cf3b9d6fc5e6f733737425267d3af296d7f99f9b9fd1d255a3ab7689a6346ab",
150
+ "pg13735.txt": "782a78ce6e52f1cadb894bc871ffd1330861e1a2db4bcf3da0fb325d62c72469",
151
+ "pg13846.txt": "4e3c7f3329d591408dca1ebe0818cd900622dcf6d090e16240bf0bbfac07370c",
152
+ "pg13951.txt": "46c01913219e6ff165256b8cbcaec7034b004f785a147341911b3caee70ccfcc",
153
+ "pg1400.txt": "9a637118af8e953e9764ec603d9b0a032883384d465acac2e27966a80cf1c6f8",
154
+ "pg14155.txt": "93f0a2259dbaf5fde6e7b96adc22346d52f80eb1e9319336c8381a045efe40cd",
155
+ "pg145.txt": "dc2e0107e6ae07e8e33da934b2ec4e8e600826a2e0cd48df265119cd02e5fb50",
156
+ "pg1497.txt": "3d2e397122881d7737a48fb80e71cddf4c78cda54ceac7e0905fe945dfab48fd",
157
+ "pg15200.txt": "7145bcdb997fa09b56acc750031dd7c5fecf26f198475041dac9c179b1890f96",
158
+ "pg158.txt": "532b122b4e6a76cc556d6fdcd729b5892f5c4ce4a1b7060b9f832adfad8680dc",
159
+ "pg161.txt": "7bcd77c45cb32fb8d9f89402561270cc6a911c0a4e20667176d5be779fc3974a",
160
+ "pg16328.txt": "8909085dc48daf123d29c40ecbb3837b16ba7f96bf0cbb9fe72de133a2d35348",
161
+ "pg1661.txt": "922e2a12ccb43a4c9544c260b2166c6ad2097aeb5957faeee113f173bb857cd0",
162
+ "pg16901.txt": "7d264a3faf62bc4e0e37da29d1d2b07c022bb2e0565ee0e6e6b534ff30422290",
163
+ "pg1727.txt": "ffbdb29c3dda284b65c11243db1a98167826a70c81bca2ed4b6232f86c905fb9",
164
+ "pg17489.txt": "a5de514ba7b9f2e1790e7e259c4e8b7a35ae1d29e4bf9a5f8767039c58b80503",
165
+ "pg17493.txt": "ffddbcee83e3c537b5586bfbbaf97ea6fab2b04250b9efaa63c027ab20aaffd0",
166
+ "pg17494.txt": "79894e327a20044dad5b93dd7d1fae0a8c06b74b86cd9f19d37bb54ffbacd44f",
167
+ "pg17808.txt": "401ad78cdcb606765f48f7b478c681d73b49d0c3666a1f6044d1a38564a01c4c",
168
+ "pg17989.txt": "9ea7a8b127a9dfa36de0b0cf9c2e26293beea5c37a0fc988fb9308b63160f282",
169
+ "pg17990.txt": "d3df81d687dfd2ba8e7d219287e767b659c189657c51a3f5433e1ba71f0d0613",
170
+ "pg17991.txt": "49cacb5455e93bebcd744a572447819a9a7f7161fb3516b5fdccbac7a4e2f571",
171
+ "pg17992.txt": "abaf80e7672d909e8adee9cf79ff18b6d79f785ac209cf44d7c06526ed627bd9",
172
+ "pg18143.txt": "dec49c928b81fddc0016e6be2def4045880142a272d299e5fe7acdcd11455fff",
173
+ "pg1952.txt": "20c7788eeae2fe3a0efd51a1f08b6723366a4be0778d65db219149a3522651d4",
174
+ "pg19657.txt": "91e3524d6e7816cda822f97fae3f49af204c7e32ad71ca23002de9e83e935f80",
175
+ "pg205.txt": "2d9a76a2e3e8195c69430516ebd33c4d0757a53ad432ff6186b7b794e6fe99f9",
176
+ "pg215.txt": "30f569c5671bba7bbc4a92ab6d8ee01a0d3be00f994819c77da62c28041eb8a2",
177
+ "pg219.txt": "1bcdf0a1ac611dd6b70a20d100a07085754da4e90a949d6031717b86cd6d020a",
178
+ "pg23.txt": "234c15348a66919bad1d534cdd48ee8ddf91f50115a706cbefaa8edd049672fd",
179
+ "pg2554.txt": "4a84a709a6e355f5fa555c7276c8a90125eb9d06f63bc8663410c3d1297eb688",
180
+ "pg2591.txt": "377ebb8ca06cd8bc0b1015ddb54766c44f50dece731c53bb030f1cc00f54092a",
181
+ "pg2600.txt": "2d5bb2ad5f422765e714617e21fa31bbaf8958aa79682c86fca6660fcc5d1b2b",
182
+ "pg2650.txt": "9ebdb837f733f41d6a822e010d1ccbf14b4ddafad0131482461edb72b55b0936",
183
+ "pg26685.txt": "b838eca71b85dd32b8ef0389d9b8a0a3fec41b85e060a1622f718da418760c46",
184
+ "pg2701.txt": "b5e88b71a1cda068e9ab1d94344c9d932338ba6da1a66def97a1aecd9cfcc87d",
185
+ "pg28054.txt": "2e75c187cb750df3f7c910fa90970b264b9a4fee212be2a73c6bb507d9ad22fe",
186
+ "pg2814.txt": "6be7d8400ac2549459307d1c1a20a25c86a736984269d3e9d989c817814be60c",
187
+ "pg2998.txt": "d23f9d231eed199a7ebab51af574a36dde00662b0310501b874b9f36db1bdd2f",
188
+ "pg30254.txt": "a0084df9930e6b7fb088a845a94fa8ae460aafa0f6a751a0f2d1b32ff3dccd2a",
189
+ "pg3207.txt": "3de1e492641d939567a8b0de827fb13e1ad992324f9ac8b204f60475009b7294",
190
+ "pg345.txt": "96cd16eacdbfebae8fdda5591f66e0cc8ee76be18e0cd1aca02bc00615782d28",
191
+ "pg35.txt": "2892e919000e17c83e1dac51b30f4675db50536b644d7579fe8a89bb399a9bdc",
192
+ "pg36.txt": "8417469e3ab664749f7b36ebb85799075818207b0380b54c27d07d592a44117d",
193
+ "pg3600.txt": "1b4c87312f0890e04cecee48e3a5fa263de65743230b819e16cb1bd72f7aee59",
194
+ "pg37106.txt": "494309c7c4b1a750f19bb69258760a602032ebb0a52955f5153818b3508d5fe0",
195
+ "pg408.txt": "87f8e832484792b761a510b953d67e3835fa46a9f4231a508bfeb914a98559ca",
196
+ "pg41211.txt": "c192ed74d4ffd4000497033b8a512e9f5ca805531281d5395259f954c12e30d2",
197
+ "pg42.txt": "6f5ebfb79a24bb24e88c8f7e5f35515321e545bf4f137fce6e406c9da0bb2b59",
198
+ "pg42131.txt": "bfb7cfb2ae429b4b408029accc0774ebae9875673f2b0130d51b14eb57008f3b",
199
+ "pg43.txt": "b43448a88391591f9cf82b25553df00faf47a2752a873185fcdd1156eebdb990",
200
+ "pg4300.txt": "e03094626f9528cf3fc287a49d6edbdbf47cd40483cb13ceb301e74e15ebbf9e",
201
+ "pg4363.txt": "6ee088f717d3760036086f108c03d57588036c9d344b7d71edbef4e631bd374a",
202
+ "pg46.txt": "8c36b93723d454982df045ec0efdf2664e5a48e6b5e2f670e883575f75ad0cc8",
203
+ "pg4650.txt": "1cc10ab3268a76aa01bd3f9722ae2ae0e5e36b869a18b0bb0832b75bc0327806",
204
+ "pg4791.txt": "cc2c572afc2b2965e252ed8626c4e023b54e272f3a83594f10bc8d7bc6a9f5d6",
205
+ "pg4968.txt": "22bbbb7f1a83c1fe457ef52ea1c00ff95461e15f62c0f5fd70130fbf82ae32e4",
206
+ "pg5097.txt": "53507b025cd3580f4fbfa546baabb36f25adfb48b581c5b5016d91ad69d36c68",
207
+ "pg514.txt": "36831477b339dbcd7e7206119d77e9e58b9727ef3c299d1a1dc4446e1db97850",
208
+ "pg51516.txt": "0c70c8e7e7ce5a9adda520495b074f486af3681c57cc21ba8556dc831aa0df04",
209
+ "pg5200.txt": "0933283a19a4daaef0d412e8f69c74fae126b0a946cb74b8f3c783a8e869830d",
210
+ "pg55.txt": "969bffab7740d4d8a0bac332d78bd0152ad4a40a2efc52f06c74a9bb6120be75",
211
+ "pg5711.txt": "7bbd0d116a119b0626b88d7f8e7e5196831e348395249df6d0ec61abc30e583b",
212
+ "pg6130.txt": "ca4f23115b384b0219427ef208e875f0351820b8f80f28259272997ea581c36b",
213
+ "pg64317.txt": "ce760ec377accd352b41bb8f64504a72d7aa18ab3afb42ded2b56cecacf29e35",
214
+ "pg6838.txt": "c5c049d44e8b44ca64c388396fb0ef270c14f7865554c6fb2cb2102c35f7e77b",
215
+ "pg7196.txt": "71e7f6fa3dd9c0f25008fe7cf176ce1b8d99126be50fdd61da6daf290540a8be",
216
+ "pg74.txt": "74d77384b123a6360db9ab58463cff8b38df8525fbdc6000c81ee388e1f3cf10",
217
+ "pg76.txt": "d617a37aa7ae1e1a93dcde2634db2bccb86824e31e29a55230bdbf77d6872d59",
218
+ "pg768.txt": "e533fe750589f0421d5d744576315f5c2b9b0d69e981179ea0551bbf134c5e02",
219
+ "pg799.txt": "b647a30294f786cc8b502748fd1d1dd17486fadc16ff4d91147b4a7b2402d270",
220
+ "pg844.txt": "1b8a58099bb1cdef6a845277a4bacf2f4a268702c165bde30124d4b5105d1851",
221
+ "pg8712.txt": "fbafaf23e91de6c540ac18e56613a0c8553fcff42842dad1c6ae02438f09d8a0",
222
+ "pg8800.txt": "3a7dd97b5fec82456c58237b33383a593480c2bfe088aeaaa4a519de2a10d39c",
223
+ "pg9645.txt": "1fd16ddf201decd5b376a27dc1576ff12fcfcf835ad344b26100d6d70071324c",
224
+ "pg98.txt": "d54c2b80d40a40b982cd88852c6180bb944d95acdb028af3d0e01a1750681784",
225
+ "pg996.txt": "2864143e9addf498c1c5fd585db4a1ff3e06651c344cddf0931342f1a0ce24d4"
226
+ }
227
+ }
v3_scale/corpus_v3/oneshot.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a
3
+ size 119763
v3_scale/corpus_v3/test.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84
3
+ size 3486653
v3_scale/corpus_v3/train.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c
3
+ size 93284487
v3_scale/reports/audit_overlap_broad_local_v1.json ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "fresh_en": {
3
+ "windows64": 119700,
4
+ "windows64_overlap": 0,
5
+ "bytes_covered": 0,
6
+ "positions": 8000,
7
+ "positions_contaminated": 0,
8
+ "ref_fused_invocab_all": 5.166846071044134,
9
+ "ref_fused_invocab_clean": 5.166846071044134,
10
+ "ref_fused_invocab_contaminated": null
11
+ },
12
+ "fresh_fr": {
13
+ "windows64": 1088625,
14
+ "windows64_overlap": 0,
15
+ "bytes_covered": 0,
16
+ "positions": 8000,
17
+ "positions_contaminated": 0,
18
+ "ref_fused_invocab_all": 2.716933445661331,
19
+ "ref_fused_invocab_clean": 2.716933445661331,
20
+ "ref_fused_invocab_contaminated": null
21
+ },
22
+ "neither_code": {
23
+ "windows64": 545973,
24
+ "windows64_overlap": 545558,
25
+ "bytes_covered": 546036,
26
+ "positions": 8000,
27
+ "positions_contaminated": 8000,
28
+ "ref_fused_invocab_all": 7.669746297111013,
29
+ "ref_fused_invocab_clean": null,
30
+ "ref_fused_invocab_contaminated": 7.669746297111013
31
+ },
32
+ "test20k": {
33
+ "windows64": 894655,
34
+ "windows64_overlap": 33766,
35
+ "bytes_covered": 45852,
36
+ "positions": 20000,
37
+ "positions_contaminated": 136,
38
+ "ref_fused_invocab_all": 3.2328136979299766,
39
+ "ref_fused_invocab_clean": 3.2575806347775704,
40
+ "ref_fused_invocab_contaminated": 0.0
41
+ },
42
+ "queries": {
43
+ "windows64": 27737,
44
+ "windows64_overlap": 0,
45
+ "bytes_covered": 0,
46
+ "positions": 7750,
47
+ "positions_contaminated": 0,
48
+ "ref_fused_invocab_all": 8.02061855670103,
49
+ "ref_fused_invocab_clean": 8.02061855670103,
50
+ "ref_fused_invocab_contaminated": null
51
+ },
52
+ "freshcode_v2": {
53
+ "windows64": 199729,
54
+ "windows64_overlap": 0,
55
+ "bytes_covered": 0,
56
+ "positions": 8000,
57
+ "positions_contaminated": 0,
58
+ "ref_fused_invocab_all": 8.456316410861865,
59
+ "ref_fused_invocab_clean": 8.456316410861865,
60
+ "ref_fused_invocab_contaminated": null
61
+ }
62
+ }
v3_scale/reports/audit_overlap_old_local_v1.json ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "fresh_en": {
3
+ "windows64": 119700,
4
+ "windows64_overlap": 0,
5
+ "bytes_covered": 0,
6
+ "positions": 8000,
7
+ "positions_contaminated": 0,
8
+ "ref_fused_invocab_all": 3.3951697584879246,
9
+ "ref_fused_invocab_clean": 3.3951697584879246,
10
+ "ref_fused_invocab_contaminated": null
11
+ },
12
+ "fresh_fr": {
13
+ "windows64": 1088625,
14
+ "windows64_overlap": 0,
15
+ "bytes_covered": 0,
16
+ "positions": 8000,
17
+ "positions_contaminated": 0,
18
+ "ref_fused_invocab_all": 0.28310789556464294,
19
+ "ref_fused_invocab_clean": 0.28310789556464294,
20
+ "ref_fused_invocab_contaminated": null
21
+ },
22
+ "neither_code": {
23
+ "windows64": 545973,
24
+ "windows64_overlap": 224303,
25
+ "bytes_covered": 241788,
26
+ "positions": 8000,
27
+ "positions_contaminated": 7162,
28
+ "ref_fused_invocab_all": 6.951274027715691,
29
+ "ref_fused_invocab_clean": 8.47107438016529,
30
+ "ref_fused_invocab_contaminated": 6.7669172932330826
31
+ },
32
+ "test20k": {
33
+ "windows64": 319877,
34
+ "windows64_overlap": 1396,
35
+ "bytes_covered": 4254,
36
+ "positions": 20000,
37
+ "positions_contaminated": 0,
38
+ "ref_fused_invocab_all": 3.238982443568613,
39
+ "ref_fused_invocab_clean": 3.238982443568613,
40
+ "ref_fused_invocab_contaminated": null
41
+ },
42
+ "queries": {
43
+ "windows64": 27737,
44
+ "windows64_overlap": 0,
45
+ "bytes_covered": 0,
46
+ "positions": 7099,
47
+ "positions_contaminated": 0,
48
+ "ref_fused_invocab_all": 0.0,
49
+ "ref_fused_invocab_clean": 0.0,
50
+ "ref_fused_invocab_contaminated": null
51
+ },
52
+ "freshcode_v2": {
53
+ "windows64": 199729,
54
+ "windows64_overlap": 0,
55
+ "bytes_covered": 0,
56
+ "positions": 8000,
57
+ "positions_contaminated": 0,
58
+ "ref_fused_invocab_all": 11.504241671839436,
59
+ "ref_fused_invocab_clean": 11.504241671839436,
60
+ "ref_fused_invocab_contaminated": null
61
+ }
62
+ }
v3_scale/reports/corpus_v3.sha256 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c corpus_v3/train.bin
2
+ a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84 corpus_v3/test.bin
3
+ 67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a corpus_v3/oneshot.bin
v3_scale/reports/recheck_bench_v4_oldref_1722.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "seen": {
3
+ "n": 8000,
4
+ "invocab": 5868,
5
+ "fused": 5770,
6
+ "snn": 1937,
7
+ "fused_pct": 98.33,
8
+ "snn_pct": 33.01,
9
+ "cov": 0.7335,
10
+ "reported": [
11
+ 98.33,
12
+ 33.01,
13
+ 0.7335
14
+ ]
15
+ },
16
+ "fc2": {
17
+ "n": 8000,
18
+ "invocab": 4833,
19
+ "fused": 556,
20
+ "snn": 403,
21
+ "fused_pct": 11.504,
22
+ "snn_pct": 8.339,
23
+ "cov": 0.604125,
24
+ "reported": [
25
+ 11.504,
26
+ 8.339,
27
+ 0.604125
28
+ ]
29
+ }
30
+ }
v3_scale/reports/recheck_corpus_v3_1700.json ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sizes": {
3
+ "train.bin": 93284487,
4
+ "test.bin": 3486653,
5
+ "oneshot.bin": 119763
6
+ },
7
+ "sha_ok": {
8
+ "train": true,
9
+ "test": true,
10
+ "oneshot": true
11
+ },
12
+ "overlap64_final": {
13
+ "fresh_fr": 0,
14
+ "freshcode": 0,
15
+ "neither_code": 0,
16
+ "test20k": 0,
17
+ "fresh_en": 0,
18
+ "queries": 0,
19
+ "freshcode_v2": 0,
20
+ "contes_perrault": 0,
21
+ "kahnn_code": 0,
22
+ "chat_seed": 0,
23
+ "test_old": 0,
24
+ "test_broad": 0,
25
+ "oneshot_old": 0,
26
+ "oneshot_broad": 0,
27
+ "test_v3": 0
28
+ },
29
+ "heldout": [
30
+ "pg1184.txt",
31
+ "pg8712.txt"
32
+ ],
33
+ "n_books": 97,
34
+ "comp": {
35
+ "broad_local_v1": 34443642,
36
+ "english": 41663184,
37
+ "french": 18630849
38
+ },
39
+ "manifests_identical": true
40
+ }
v3_scale/reports/recheck_scale_smoke_1706.json ADDED
@@ -0,0 +1,198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "acc": {
3
+ "old_ref": {
4
+ "win": {
5
+ "seen": {
6
+ "n": 300,
7
+ "invocab": 235,
8
+ "hits_in": 228,
9
+ "hits_all": 290,
10
+ "pct": 97.02127659574468
11
+ },
12
+ "fc2": {
13
+ "n": 300,
14
+ "invocab": 187,
15
+ "hits_in": 31,
16
+ "hits_all": 35,
17
+ "pct": 16.577540106951872
18
+ },
19
+ "fresh_en": {
20
+ "n": 300,
21
+ "invocab": 202,
22
+ "hits_in": 5,
23
+ "hits_all": 5,
24
+ "pct": 2.4752475247524752
25
+ },
26
+ "fresh_fr": {
27
+ "n": 300,
28
+ "invocab": 102,
29
+ "hits_in": 1,
30
+ "hits_all": 1,
31
+ "pct": 0.9803921568627451
32
+ },
33
+ "neither_code": {
34
+ "n": 300,
35
+ "invocab": 179,
36
+ "hits_in": 14,
37
+ "hits_all": 14,
38
+ "pct": 7.82122905027933
39
+ },
40
+ "test20k": {
41
+ "n": 300,
42
+ "invocab": 201,
43
+ "hits_in": 4,
44
+ "hits_all": 4,
45
+ "pct": 1.9900497512437811
46
+ },
47
+ "queries": {
48
+ "n": 300,
49
+ "invocab": 141,
50
+ "hits_in": 0,
51
+ "hits_all": 0,
52
+ "pct": 0.0
53
+ },
54
+ "test_v3": {
55
+ "n": 300,
56
+ "invocab": 120,
57
+ "hits_in": 17,
58
+ "hits_all": 17,
59
+ "pct": 14.166666666666666
60
+ }
61
+ },
62
+ "fresh_mean_excl_nc": 6.031649367746257,
63
+ "gap_excl_nc": 90.98962722799843,
64
+ "fresh_mean_incl_nc": 6.287303608108124,
65
+ "gap_incl_nc": 90.73397298763656
66
+ },
67
+ "broad14_ref": {
68
+ "win": {
69
+ "seen": {
70
+ "n": 300,
71
+ "invocab": 255,
72
+ "hits_in": 232,
73
+ "hits_all": 272,
74
+ "pct": 90.98039215686275
75
+ },
76
+ "fc2": {
77
+ "n": 300,
78
+ "invocab": 246,
79
+ "hits_in": 2,
80
+ "hits_all": 2,
81
+ "pct": 0.8130081300813008
82
+ },
83
+ "fresh_en": {
84
+ "n": 300,
85
+ "invocab": 234,
86
+ "hits_in": 9,
87
+ "hits_all": 9,
88
+ "pct": 3.8461538461538463
89
+ },
90
+ "fresh_fr": {
91
+ "n": 300,
92
+ "invocab": 247,
93
+ "hits_in": 79,
94
+ "hits_all": 79,
95
+ "pct": 31.983805668016196
96
+ },
97
+ "neither_code": {
98
+ "n": 300,
99
+ "invocab": 244,
100
+ "hits_in": 7,
101
+ "hits_all": 7,
102
+ "pct": 2.8688524590163933
103
+ },
104
+ "test20k": {
105
+ "n": 300,
106
+ "invocab": 233,
107
+ "hits_in": 5,
108
+ "hits_all": 5,
109
+ "pct": 2.1459227467811157
110
+ },
111
+ "queries": {
112
+ "n": 300,
113
+ "invocab": 186,
114
+ "hits_in": 15,
115
+ "hits_all": 15,
116
+ "pct": 8.064516129032258
117
+ },
118
+ "test_v3": {
119
+ "n": 300,
120
+ "invocab": 171,
121
+ "hits_in": 1,
122
+ "hits_all": 1,
123
+ "pct": 0.5847953216374269
124
+ }
125
+ },
126
+ "fresh_mean_excl_nc": 7.906366973617025,
127
+ "gap_excl_nc": 83.07402518324572,
128
+ "fresh_mean_incl_nc": 7.186722042959792,
129
+ "gap_incl_nc": 83.79367011390296
130
+ }
131
+ },
132
+ "pairs": {
133
+ "seen": {
134
+ "A_only": 50,
135
+ "B_only": 54,
136
+ "p": 0.7687918101386728,
137
+ "raw_A_only": 25,
138
+ "raw_B_only": 7,
139
+ "raw_p": 0.0021024015732109547
140
+ },
141
+ "fc2": {
142
+ "A_only": 31,
143
+ "B_only": 2,
144
+ "p": 1.3085082173347473e-07,
145
+ "raw_A_only": 35,
146
+ "raw_B_only": 2,
147
+ "raw_p": 1.0244548320770264e-08
148
+ },
149
+ "fresh_en": {
150
+ "A_only": 5,
151
+ "B_only": 9,
152
+ "p": 0.4239501953125,
153
+ "raw_A_only": 5,
154
+ "raw_B_only": 9,
155
+ "raw_p": 0.4239501953125
156
+ },
157
+ "fresh_fr": {
158
+ "A_only": 1,
159
+ "B_only": 79,
160
+ "p": 1.3400325923359048e-22,
161
+ "raw_A_only": 1,
162
+ "raw_B_only": 79,
163
+ "raw_p": 1.3400325923359048e-22
164
+ },
165
+ "neither_code": {
166
+ "A_only": 12,
167
+ "B_only": 5,
168
+ "p": 0.143463134765625,
169
+ "raw_A_only": 12,
170
+ "raw_B_only": 5,
171
+ "raw_p": 0.143463134765625
172
+ },
173
+ "test20k": {
174
+ "A_only": 4,
175
+ "B_only": 5,
176
+ "p": 1.0,
177
+ "raw_A_only": 4,
178
+ "raw_B_only": 5,
179
+ "raw_p": 1.0
180
+ },
181
+ "queries": {
182
+ "A_only": 0,
183
+ "B_only": 15,
184
+ "p": 6.103515625e-05,
185
+ "raw_A_only": 0,
186
+ "raw_B_only": 15,
187
+ "raw_p": 6.103515625e-05
188
+ },
189
+ "test_v3": {
190
+ "A_only": 17,
191
+ "B_only": 1,
192
+ "p": 0.00014495849609375,
193
+ "raw_A_only": 17,
194
+ "raw_B_only": 1,
195
+ "raw_p": 0.00014495849609375
196
+ }
197
+ }
198
+ }
v3_scale/reports/recheck_v4broad14_1748.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "broad14_ref/seen": {
3
+ "n": 8000,
4
+ "invocab": 6980,
5
+ "fused": 6417,
6
+ "snn": 3571,
7
+ "fused_pct": 91.934,
8
+ "snn_pct": 51.16,
9
+ "cov": 0.8725,
10
+ "reported": [
11
+ 91.934,
12
+ 51.16,
13
+ 0.8725
14
+ ]
15
+ },
16
+ "broad14_ref/fc2": {
17
+ "n": 8000,
18
+ "invocab": 6776,
19
+ "fused": 573,
20
+ "snn": 207,
21
+ "fused_pct": 8.456,
22
+ "snn_pct": 3.055,
23
+ "cov": 0.847,
24
+ "reported": [
25
+ 8.456,
26
+ 3.055,
27
+ 0.847
28
+ ]
29
+ },
30
+ "broad14_ref/fresh_en": {
31
+ "n": 8000,
32
+ "invocab": 6503,
33
+ "fused": 336,
34
+ "snn": 325,
35
+ "fused_pct": 5.167,
36
+ "snn_pct": 4.998,
37
+ "cov": 0.812875,
38
+ "reported": [
39
+ 5.167,
40
+ 4.998,
41
+ 0.812875
42
+ ]
43
+ }
44
+ }
v3_scale/reports/recheck_v4broad14_1755.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "broad14_ref/seen": {
3
+ "n": 8000,
4
+ "invocab": 6980,
5
+ "fused": 6417,
6
+ "snn": 3571,
7
+ "fused_pct": 91.934,
8
+ "snn_pct": 51.16,
9
+ "cov": 0.8725,
10
+ "reported": [
11
+ 91.934,
12
+ 51.16,
13
+ 0.8725
14
+ ]
15
+ },
16
+ "broad14_ref/fc2": {
17
+ "n": 8000,
18
+ "invocab": 6776,
19
+ "fused": 573,
20
+ "snn": 207,
21
+ "fused_pct": 8.456,
22
+ "snn_pct": 3.055,
23
+ "cov": 0.847,
24
+ "reported": [
25
+ 8.456,
26
+ 3.055,
27
+ 0.847
28
+ ]
29
+ },
30
+ "broad14_ref/fresh_en": {
31
+ "n": 8000,
32
+ "invocab": 6503,
33
+ "fused": 336,
34
+ "snn": 325,
35
+ "fused_pct": 5.167,
36
+ "snn_pct": 4.998,
37
+ "cov": 0.812875,
38
+ "reported": [
39
+ 5.167,
40
+ 4.998,
41
+ 0.812875
42
+ ]
43
+ },
44
+ "broad14_ref/fresh_fr": {
45
+ "n": 8000,
46
+ "invocab": 4748,
47
+ "fused": 129,
48
+ "snn": 20,
49
+ "fused_pct": 2.717,
50
+ "snn_pct": 0.421,
51
+ "cov": 0.5935,
52
+ "reported": [
53
+ 2.717,
54
+ 0.421,
55
+ 0.5935
56
+ ]
57
+ },
58
+ "broad14_ref/neither_code": {
59
+ "n": 8000,
60
+ "invocab": 6819,
61
+ "fused": 523,
62
+ "snn": 182,
63
+ "fused_pct": 7.67,
64
+ "snn_pct": 2.669,
65
+ "cov": 0.852375,
66
+ "reported": [
67
+ 7.67,
68
+ 2.669,
69
+ 0.852375
70
+ ]
71
+ }
72
+ }
v3_scale/reports/recheck_v4broad14_1758.json ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "broad14_ref/seen": {
3
+ "n": 8000,
4
+ "invocab": 6980,
5
+ "fused": 6417,
6
+ "snn": 3571,
7
+ "fused_pct": 91.934,
8
+ "snn_pct": 51.16,
9
+ "cov": 0.8725,
10
+ "reported": [
11
+ 91.934,
12
+ 51.16,
13
+ 0.8725
14
+ ]
15
+ },
16
+ "broad14_ref/fc2": {
17
+ "n": 8000,
18
+ "invocab": 6776,
19
+ "fused": 573,
20
+ "snn": 207,
21
+ "fused_pct": 8.456,
22
+ "snn_pct": 3.055,
23
+ "cov": 0.847,
24
+ "reported": [
25
+ 8.456,
26
+ 3.055,
27
+ 0.847
28
+ ]
29
+ },
30
+ "broad14_ref/fresh_en": {
31
+ "n": 8000,
32
+ "invocab": 6503,
33
+ "fused": 336,
34
+ "snn": 325,
35
+ "fused_pct": 5.167,
36
+ "snn_pct": 4.998,
37
+ "cov": 0.812875,
38
+ "reported": [
39
+ 5.167,
40
+ 4.998,
41
+ 0.812875
42
+ ]
43
+ },
44
+ "broad14_ref/fresh_fr": {
45
+ "n": 8000,
46
+ "invocab": 4748,
47
+ "fused": 129,
48
+ "snn": 20,
49
+ "fused_pct": 2.717,
50
+ "snn_pct": 0.421,
51
+ "cov": 0.5935,
52
+ "reported": [
53
+ 2.717,
54
+ 0.421,
55
+ 0.5935
56
+ ]
57
+ },
58
+ "broad14_ref/neither_code": {
59
+ "n": 8000,
60
+ "invocab": 6819,
61
+ "fused": 523,
62
+ "snn": 182,
63
+ "fused_pct": 7.67,
64
+ "snn_pct": 2.669,
65
+ "cov": 0.852375,
66
+ "reported": [
67
+ 7.67,
68
+ 2.669,
69
+ 0.852375
70
+ ]
71
+ },
72
+ "broad14_ref/test20k": {
73
+ "n": 20000,
74
+ "invocab": 15652,
75
+ "fused": 506,
76
+ "snn": 501,
77
+ "fused_pct": 3.233,
78
+ "snn_pct": 3.201,
79
+ "cov": 0.7826,
80
+ "reported": [
81
+ 3.233,
82
+ 3.201,
83
+ 0.7826
84
+ ]
85
+ }
86
+ }
v3_scale/reports/recheck_v4code_1733.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "old_ref/seen": {
3
+ "n": 8000,
4
+ "invocab": 5868,
5
+ "fused": 5770,
6
+ "snn": 1937,
7
+ "fused_pct": 98.33,
8
+ "snn_pct": 33.01,
9
+ "cov": 0.7335,
10
+ "reported": [
11
+ 98.33,
12
+ 33.01,
13
+ 0.7335
14
+ ]
15
+ },
16
+ "old_ref/fc2": {
17
+ "n": 8000,
18
+ "invocab": 4833,
19
+ "fused": 556,
20
+ "snn": 403,
21
+ "fused_pct": 11.504,
22
+ "snn_pct": 8.339,
23
+ "cov": 0.604125,
24
+ "reported": [
25
+ 11.504,
26
+ 8.339,
27
+ 0.604125
28
+ ]
29
+ },
30
+ "old_ref/fresh_en": {
31
+ "n": 8000,
32
+ "invocab": 5714,
33
+ "fused": 194,
34
+ "snn": 175,
35
+ "fused_pct": 3.395,
36
+ "snn_pct": 3.063,
37
+ "cov": 0.71425,
38
+ "reported": [
39
+ 3.395,
40
+ 3.063,
41
+ 0.71425
42
+ ]
43
+ },
44
+ "old_ref/fresh_fr": {
45
+ "n": 8000,
46
+ "invocab": 3179,
47
+ "fused": 9,
48
+ "snn": 9,
49
+ "fused_pct": 0.283,
50
+ "snn_pct": 0.283,
51
+ "cov": 0.397375,
52
+ "reported": [
53
+ 0.283,
54
+ 0.283,
55
+ 0.397375
56
+ ]
57
+ },
58
+ "old_ref/neither_code": {
59
+ "n": 8000,
60
+ "invocab": 4474,
61
+ "fused": 311,
62
+ "snn": 229,
63
+ "fused_pct": 6.951,
64
+ "snn_pct": 5.118,
65
+ "cov": 0.55925,
66
+ "reported": [
67
+ 6.951,
68
+ 5.118,
69
+ 0.55925
70
+ ]
71
+ }
72
+ }
v3_scale/reports/recheck_v4fresh_gen_1728.json ADDED
@@ -0,0 +1,368 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bench": {
3
+ "old_ref/seen": {
4
+ "n": 8000,
5
+ "invocab": 5868,
6
+ "fused": 5770,
7
+ "snn": 1937,
8
+ "fused_pct": 98.33,
9
+ "snn_pct": 33.01,
10
+ "cov": 0.7335,
11
+ "reported": [
12
+ 98.33,
13
+ 33.01,
14
+ 0.7335
15
+ ]
16
+ },
17
+ "old_ref/fc2": {
18
+ "n": 8000,
19
+ "invocab": 4833,
20
+ "fused": 556,
21
+ "snn": 403,
22
+ "fused_pct": 11.504,
23
+ "snn_pct": 8.339,
24
+ "cov": 0.604125,
25
+ "reported": [
26
+ 11.504,
27
+ 8.339,
28
+ 0.604125
29
+ ]
30
+ },
31
+ "old_ref/fresh_en": {
32
+ "n": 8000,
33
+ "invocab": 5714,
34
+ "fused": 194,
35
+ "snn": 175,
36
+ "fused_pct": 3.395,
37
+ "snn_pct": 3.063,
38
+ "cov": 0.71425,
39
+ "reported": [
40
+ 3.395,
41
+ 3.063,
42
+ 0.71425
43
+ ]
44
+ },
45
+ "old_ref/fresh_fr": {
46
+ "n": 8000,
47
+ "invocab": 3179,
48
+ "fused": 9,
49
+ "snn": 9,
50
+ "fused_pct": 0.283,
51
+ "snn_pct": 0.283,
52
+ "cov": 0.397375,
53
+ "reported": [
54
+ 0.283,
55
+ 0.283,
56
+ 0.397375
57
+ ]
58
+ }
59
+ },
60
+ "gen": {
61
+ "old_ref/fused/en1": {
62
+ "byte_acc": 0.06,
63
+ "rep_byte_acc": 0.055,
64
+ "exact_bytes": false,
65
+ "vw": 0.4762,
66
+ "rep_vw": 0.4762,
67
+ "nw": 21,
68
+ "rep_nw": 21
69
+ },
70
+ "old_ref/fused/en2": {
71
+ "byte_acc": 0.085,
72
+ "rep_byte_acc": 0.085,
73
+ "exact_bytes": true,
74
+ "vw": 0.3158,
75
+ "rep_vw": 0.3158,
76
+ "nw": 19,
77
+ "rep_nw": 19
78
+ },
79
+ "old_ref/fused/fr1": {
80
+ "byte_acc": 0.025,
81
+ "rep_byte_acc": 0.02,
82
+ "exact_bytes": false,
83
+ "vw": 0.64,
84
+ "rep_vw": 0.64,
85
+ "nw": 25,
86
+ "rep_nw": 25
87
+ },
88
+ "old_ref/fused/fr2": {
89
+ "byte_acc": 0.09,
90
+ "rep_byte_acc": 0.09,
91
+ "exact_bytes": true,
92
+ "vw": 0.7333,
93
+ "rep_vw": 0.7333,
94
+ "nw": 15,
95
+ "rep_nw": 15
96
+ },
97
+ "old_ref/fused/chat": {
98
+ "byte_acc": 0.08,
99
+ "rep_byte_acc": 0.08,
100
+ "exact_bytes": true,
101
+ "vw": 0.5,
102
+ "rep_vw": 0.5,
103
+ "nw": 26,
104
+ "rep_nw": 26
105
+ },
106
+ "old_ref/fused/code": {
107
+ "byte_acc": 0.095,
108
+ "rep_byte_acc": 0.095,
109
+ "exact_bytes": true,
110
+ "vw": 0.4118,
111
+ "rep_vw": 0.4118,
112
+ "nw": 17,
113
+ "rep_nw": 17
114
+ },
115
+ "old_ref/fused/v3test": {
116
+ "byte_acc": 0.08,
117
+ "rep_byte_acc": 0.08,
118
+ "exact_bytes": true,
119
+ "vw": 0.52,
120
+ "rep_vw": 0.52,
121
+ "nw": 25,
122
+ "rep_nw": 25
123
+ },
124
+ "old_ref/v2b/en1": {
125
+ "byte_acc": 0.065,
126
+ "rep_byte_acc": 0.065,
127
+ "exact_bytes": true,
128
+ "vw": 0.9545,
129
+ "rep_vw": 0.9545,
130
+ "nw": 22,
131
+ "rep_nw": 22
132
+ },
133
+ "old_ref/v2b/en2": {
134
+ "byte_acc": 0.085,
135
+ "rep_byte_acc": 0.085,
136
+ "exact_bytes": true,
137
+ "vw": 0.9744,
138
+ "rep_vw": 0.9744,
139
+ "nw": 39,
140
+ "rep_nw": 39
141
+ },
142
+ "old_ref/v2b/fr1": {
143
+ "byte_acc": 0.06,
144
+ "rep_byte_acc": 0.06,
145
+ "exact_bytes": false,
146
+ "vw": 0.931,
147
+ "rep_vw": 0.931,
148
+ "nw": 29,
149
+ "rep_nw": 29
150
+ },
151
+ "old_ref/v2b/fr2": {
152
+ "byte_acc": 0.06,
153
+ "rep_byte_acc": 0.06,
154
+ "exact_bytes": true,
155
+ "vw": 1.0,
156
+ "rep_vw": 1.0,
157
+ "nw": 27,
158
+ "rep_nw": 27
159
+ },
160
+ "old_ref/v2b/chat": {
161
+ "byte_acc": 0.09,
162
+ "rep_byte_acc": 0.09,
163
+ "exact_bytes": true,
164
+ "vw": 0.9714,
165
+ "rep_vw": 0.9714,
166
+ "nw": 35,
167
+ "rep_nw": 35
168
+ },
169
+ "old_ref/v2b/code": {
170
+ "byte_acc": 0.11,
171
+ "rep_byte_acc": 0.11,
172
+ "exact_bytes": true,
173
+ "vw": 0.9412,
174
+ "rep_vw": 0.9412,
175
+ "nw": 17,
176
+ "rep_nw": 17
177
+ },
178
+ "old_ref/v2b/v3test": {
179
+ "byte_acc": 0.1,
180
+ "rep_byte_acc": 0.1,
181
+ "exact_bytes": true,
182
+ "vw": 0.913,
183
+ "rep_vw": 0.913,
184
+ "nw": 23,
185
+ "rep_nw": 23
186
+ },
187
+ "broad14_ref/fused/en1": {
188
+ "byte_acc": 0.065,
189
+ "rep_byte_acc": 0.065,
190
+ "exact_bytes": true,
191
+ "vw": 0.9211,
192
+ "rep_vw": 0.9211,
193
+ "nw": 38,
194
+ "rep_nw": 38
195
+ },
196
+ "broad14_ref/fused/en2": {
197
+ "byte_acc": 0.05,
198
+ "rep_byte_acc": 0.05,
199
+ "exact_bytes": true,
200
+ "vw": 0.5938,
201
+ "rep_vw": 0.5938,
202
+ "nw": 32,
203
+ "rep_nw": 32
204
+ },
205
+ "broad14_ref/fused/fr1": {
206
+ "byte_acc": 0.04,
207
+ "rep_byte_acc": 0.04,
208
+ "exact_bytes": true,
209
+ "vw": 0.6286,
210
+ "rep_vw": 0.6286,
211
+ "nw": 35,
212
+ "rep_nw": 35
213
+ },
214
+ "broad14_ref/fused/fr2": {
215
+ "byte_acc": 0.06,
216
+ "rep_byte_acc": 0.06,
217
+ "exact_bytes": true,
218
+ "vw": 0.8205,
219
+ "rep_vw": 0.8205,
220
+ "nw": 39,
221
+ "rep_nw": 39
222
+ },
223
+ "broad14_ref/fused/chat": {
224
+ "byte_acc": 0.065,
225
+ "rep_byte_acc": 0.065,
226
+ "exact_bytes": true,
227
+ "vw": 0.7895,
228
+ "rep_vw": 0.7895,
229
+ "nw": 38,
230
+ "rep_nw": 38
231
+ },
232
+ "broad14_ref/fused/code": {
233
+ "byte_acc": 0.065,
234
+ "rep_byte_acc": 0.065,
235
+ "exact_bytes": true,
236
+ "vw": 0.5429,
237
+ "rep_vw": 0.5429,
238
+ "nw": 35,
239
+ "rep_nw": 35
240
+ },
241
+ "broad14_ref/fused/v3test": {
242
+ "byte_acc": 0.075,
243
+ "rep_byte_acc": 0.075,
244
+ "exact_bytes": true,
245
+ "vw": 0.8974,
246
+ "rep_vw": 0.8974,
247
+ "nw": 39,
248
+ "rep_nw": 39
249
+ },
250
+ "broad14_ref/v2b/en1": {
251
+ "byte_acc": 0.055,
252
+ "rep_byte_acc": 0.055,
253
+ "exact_bytes": true,
254
+ "vw": 1.0,
255
+ "rep_vw": 1.0,
256
+ "nw": 34,
257
+ "rep_nw": 34
258
+ },
259
+ "broad14_ref/v2b/en2": {
260
+ "byte_acc": 0.08,
261
+ "rep_byte_acc": 0.08,
262
+ "exact_bytes": true,
263
+ "vw": 0.9737,
264
+ "rep_vw": 0.9737,
265
+ "nw": 38,
266
+ "rep_nw": 38
267
+ },
268
+ "broad14_ref/v2b/fr1": {
269
+ "byte_acc": 0.125,
270
+ "rep_byte_acc": 0.125,
271
+ "exact_bytes": true,
272
+ "vw": 0.5,
273
+ "rep_vw": 0.5,
274
+ "nw": 4,
275
+ "rep_nw": 4
276
+ },
277
+ "broad14_ref/v2b/fr2": {
278
+ "byte_acc": 0.05,
279
+ "rep_byte_acc": 0.05,
280
+ "exact_bytes": true,
281
+ "vw": 0.913,
282
+ "rep_vw": 0.913,
283
+ "nw": 23,
284
+ "rep_nw": 23
285
+ },
286
+ "broad14_ref/v2b/chat": {
287
+ "byte_acc": 0.12,
288
+ "rep_byte_acc": 0.12,
289
+ "exact_bytes": true,
290
+ "vw": 0.6667,
291
+ "rep_vw": 0.6667,
292
+ "nw": 3,
293
+ "rep_nw": 3
294
+ },
295
+ "broad14_ref/v2b/code": {
296
+ "byte_acc": 0.065,
297
+ "rep_byte_acc": 0.065,
298
+ "exact_bytes": true,
299
+ "vw": 0.8889,
300
+ "rep_vw": 0.8889,
301
+ "nw": 18,
302
+ "rep_nw": 18
303
+ },
304
+ "broad14_ref/v2b/v3test": {
305
+ "byte_acc": 0.11,
306
+ "rep_byte_acc": 0.11,
307
+ "exact_bytes": true,
308
+ "vw": 0.95,
309
+ "rep_vw": 0.95,
310
+ "nw": 40,
311
+ "rep_nw": 40
312
+ }
313
+ },
314
+ "gen_agg": {
315
+ "old_ref/fused": {
316
+ "n": 7,
317
+ "byte_acc": 0.0721,
318
+ "vw": 0.5139,
319
+ "rep8": 0.0555,
320
+ "lang_drift": 4,
321
+ "dom_drift": 1,
322
+ "copy_mean": 56.7
323
+ },
324
+ "old_ref/v2b": {
325
+ "n": 7,
326
+ "byte_acc": 0.0814,
327
+ "vw": 0.9551,
328
+ "rep8": 0.1813,
329
+ "lang_drift": 4,
330
+ "dom_drift": 1,
331
+ "copy_mean": 126.0
332
+ },
333
+ "broad14_ref/fused": {
334
+ "n": 7,
335
+ "byte_acc": 0.06,
336
+ "vw": 0.742,
337
+ "rep8": 0.0052,
338
+ "lang_drift": 3,
339
+ "dom_drift": 1,
340
+ "copy_mean": 70.0
341
+ },
342
+ "broad14_ref/v2b": {
343
+ "n": 7,
344
+ "byte_acc": 0.0864,
345
+ "vw": 0.8418,
346
+ "rep8": 0.2694,
347
+ "lang_drift": 2,
348
+ "dom_drift": 1,
349
+ "copy_mean": 101.4
350
+ }
351
+ },
352
+ "mcnemar_gen_bytes": {
353
+ "fused": {
354
+ "old_hits": 103,
355
+ "broad_hits": 84,
356
+ "old_only": 87,
357
+ "broad_only": 68,
358
+ "p": 0.14799763102341487
359
+ },
360
+ "v2b": {
361
+ "old_hits": 114,
362
+ "broad_hits": 121,
363
+ "old_only": 92,
364
+ "broad_only": 99,
365
+ "p": 0.6642959893958231
366
+ }
367
+ }
368
+ }
v3_scale/reports/recheck_v4t20q_1737.json ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "old_ref/seen": {
3
+ "n": 8000,
4
+ "invocab": 5868,
5
+ "fused": 5770,
6
+ "snn": 1937,
7
+ "fused_pct": 98.33,
8
+ "snn_pct": 33.01,
9
+ "cov": 0.7335,
10
+ "reported": [
11
+ 98.33,
12
+ 33.01,
13
+ 0.7335
14
+ ]
15
+ },
16
+ "old_ref/fc2": {
17
+ "n": 8000,
18
+ "invocab": 4833,
19
+ "fused": 556,
20
+ "snn": 403,
21
+ "fused_pct": 11.504,
22
+ "snn_pct": 8.339,
23
+ "cov": 0.604125,
24
+ "reported": [
25
+ 11.504,
26
+ 8.339,
27
+ 0.604125
28
+ ]
29
+ },
30
+ "old_ref/fresh_en": {
31
+ "n": 8000,
32
+ "invocab": 5714,
33
+ "fused": 194,
34
+ "snn": 175,
35
+ "fused_pct": 3.395,
36
+ "snn_pct": 3.063,
37
+ "cov": 0.71425,
38
+ "reported": [
39
+ 3.395,
40
+ 3.063,
41
+ 0.71425
42
+ ]
43
+ },
44
+ "old_ref/fresh_fr": {
45
+ "n": 8000,
46
+ "invocab": 3179,
47
+ "fused": 9,
48
+ "snn": 9,
49
+ "fused_pct": 0.283,
50
+ "snn_pct": 0.283,
51
+ "cov": 0.397375,
52
+ "reported": [
53
+ 0.283,
54
+ 0.283,
55
+ 0.397375
56
+ ]
57
+ },
58
+ "old_ref/neither_code": {
59
+ "n": 8000,
60
+ "invocab": 4474,
61
+ "fused": 311,
62
+ "snn": 229,
63
+ "fused_pct": 6.951,
64
+ "snn_pct": 5.118,
65
+ "cov": 0.55925,
66
+ "reported": [
67
+ 6.951,
68
+ 5.118,
69
+ 0.55925
70
+ ]
71
+ },
72
+ "old_ref/test20k": {
73
+ "n": 20000,
74
+ "invocab": 13379,
75
+ "fused": 388,
76
+ "snn": 374,
77
+ "fused_pct": 2.9,
78
+ "snn_pct": 2.795,
79
+ "cov": 0.66895,
80
+ "reported": [
81
+ 2.9,
82
+ 2.795,
83
+ 0.66895
84
+ ]
85
+ },
86
+ "old_ref/queries": {
87
+ "n": 7099,
88
+ "invocab": 3349,
89
+ "fused": 0,
90
+ "snn": 0,
91
+ "fused_pct": 0.0,
92
+ "snn_pct": 0.0,
93
+ "cov": 0.4717565854345683,
94
+ "reported": [
95
+ 0.0,
96
+ 0.0,
97
+ 0.4717565854345683
98
+ ]
99
+ }
100
+ }
v3_scale/reports/recheck_v4testv3_1743.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "old_ref/seen": {
3
+ "n": 8000,
4
+ "invocab": 5868,
5
+ "fused": 5770,
6
+ "snn": 1937,
7
+ "fused_pct": 98.33,
8
+ "snn_pct": 33.01,
9
+ "cov": 0.7335,
10
+ "reported": [
11
+ 98.33,
12
+ 33.01,
13
+ 0.7335
14
+ ]
15
+ },
16
+ "old_ref/fc2": {
17
+ "n": 8000,
18
+ "invocab": 4833,
19
+ "fused": 556,
20
+ "snn": 403,
21
+ "fused_pct": 11.504,
22
+ "snn_pct": 8.339,
23
+ "cov": 0.604125,
24
+ "reported": [
25
+ 11.504,
26
+ 8.339,
27
+ 0.604125
28
+ ]
29
+ },
30
+ "old_ref/fresh_en": {
31
+ "n": 8000,
32
+ "invocab": 5714,
33
+ "fused": 194,
34
+ "snn": 175,
35
+ "fused_pct": 3.395,
36
+ "snn_pct": 3.063,
37
+ "cov": 0.71425,
38
+ "reported": [
39
+ 3.395,
40
+ 3.063,
41
+ 0.71425
42
+ ]
43
+ },
44
+ "old_ref/fresh_fr": {
45
+ "n": 8000,
46
+ "invocab": 3179,
47
+ "fused": 9,
48
+ "snn": 9,
49
+ "fused_pct": 0.283,
50
+ "snn_pct": 0.283,
51
+ "cov": 0.397375,
52
+ "reported": [
53
+ 0.283,
54
+ 0.283,
55
+ 0.397375
56
+ ]
57
+ },
58
+ "old_ref/neither_code": {
59
+ "n": 8000,
60
+ "invocab": 4474,
61
+ "fused": 311,
62
+ "snn": 229,
63
+ "fused_pct": 6.951,
64
+ "snn_pct": 5.118,
65
+ "cov": 0.55925,
66
+ "reported": [
67
+ 6.951,
68
+ 5.118,
69
+ 0.55925
70
+ ]
71
+ },
72
+ "old_ref/test20k": {
73
+ "n": 20000,
74
+ "invocab": 13379,
75
+ "fused": 388,
76
+ "snn": 374,
77
+ "fused_pct": 2.9,
78
+ "snn_pct": 2.795,
79
+ "cov": 0.66895,
80
+ "reported": [
81
+ 2.9,
82
+ 2.795,
83
+ 0.66895
84
+ ]
85
+ },
86
+ "old_ref/queries": {
87
+ "n": 7099,
88
+ "invocab": 3349,
89
+ "fused": 0,
90
+ "snn": 0,
91
+ "fused_pct": 0.0,
92
+ "snn_pct": 0.0,
93
+ "cov": 0.4717565854345683,
94
+ "reported": [
95
+ 0.0,
96
+ 0.0,
97
+ 0.4717565854345683
98
+ ]
99
+ },
100
+ "old_ref/test_v3": {
101
+ "n": 20000,
102
+ "invocab": 12922,
103
+ "fused": 532,
104
+ "snn": 510,
105
+ "fused_pct": 4.117,
106
+ "snn_pct": 3.947,
107
+ "cov": 0.6461,
108
+ "reported": [
109
+ 4.117,
110
+ 3.947,
111
+ 0.6461
112
+ ]
113
+ }
114
+ }
v3_scale/reports/results_v4.jsonl ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "seen", "n_positions": 8000, "fused_invocab_pct": 98.33, "snn_invocab_pct": 33.01, "vocab_coverage": 0.7335, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 128.8, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
2
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "fc2", "n_positions": 8000, "fused_invocab_pct": 11.504, "snn_invocab_pct": 8.339, "vocab_coverage": 0.604125, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 114.0, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
3
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "fresh_en", "n_positions": 8000, "fused_invocab_pct": 3.395, "snn_invocab_pct": 3.063, "vocab_coverage": 0.71425, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 239.1, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
4
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "fresh_fr", "n_positions": 8000, "fused_invocab_pct": 0.283, "snn_invocab_pct": 0.283, "vocab_coverage": 0.397375, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 187.0, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
5
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "neither_code", "n_positions": 8000, "fused_invocab_pct": 6.951, "snn_invocab_pct": 5.118, "vocab_coverage": 0.55925, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 155.4, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
6
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "test20k", "n_positions": 20000, "fused_invocab_pct": 2.9, "snn_invocab_pct": 2.795, "vocab_coverage": 0.66895, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 288.0, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
7
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "queries", "n_positions": 7099, "fused_invocab_pct": 0.0, "snn_invocab_pct": 0.0, "vocab_coverage": 0.4717565854345683, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 106.7, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
8
+ {"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "test_v3", "n_positions": 20000, "fused_invocab_pct": 4.117, "snn_invocab_pct": 3.947, "vocab_coverage": 0.6461, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 288.2, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
9
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "seen", "n_positions": 8000, "fused_invocab_pct": 91.934, "snn_invocab_pct": 51.16, "vocab_coverage": 0.8725, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 113.5, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
10
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "fc2", "n_positions": 8000, "fused_invocab_pct": 8.456, "snn_invocab_pct": 3.055, "vocab_coverage": 0.847, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 114.7, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
11
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "fresh_en", "n_positions": 8000, "fused_invocab_pct": 5.167, "snn_invocab_pct": 4.998, "vocab_coverage": 0.812875, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 115.1, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
12
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "fresh_fr", "n_positions": 8000, "fused_invocab_pct": 2.717, "snn_invocab_pct": 0.421, "vocab_coverage": 0.5935, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 109.3, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
13
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "neither_code", "n_positions": 8000, "fused_invocab_pct": 7.67, "snn_invocab_pct": 2.669, "vocab_coverage": 0.852375, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 98.6, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
14
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "test20k", "n_positions": 20000, "fused_invocab_pct": 3.233, "snn_invocab_pct": 3.201, "vocab_coverage": 0.7826, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 241.0, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
15
+ {"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "queries", "n_positions": 7750, "fused_invocab_pct": 8.021, "snn_invocab_pct": 8.021, "vocab_coverage": 0.6258064516129033, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 84.6, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
v3_scale/reports/scale_bench_v4.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "models": {
3
+ "old (r\u00e9f)": {
4
+ "tag": "old_ref",
5
+ "acc_invocab": {
6
+ "seen": 98.32992501704159,
7
+ "fc2": 11.504241671839436,
8
+ "fresh_en": 3.3951697584879246,
9
+ "fresh_fr": 0.28310789556464294,
10
+ "neither_code": 6.951274027715691,
11
+ "test20k": 2.9000672696016148,
12
+ "queries": 0.0,
13
+ "test_v3": 4.117009750812568
14
+ },
15
+ "fresh_mean": 3.699932724384364,
16
+ "gap": 94.62999229265722,
17
+ "wall_s": 1576.3,
18
+ "peak_rss_mb": 710.1,
19
+ "scale": 0.03498957642842646,
20
+ "scale_ok": true
21
+ }
22
+ },
23
+ "pairs": []
24
+ }
v3_scale/reports/scale_bench_v4.md ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Banc scale v4 — fused_acc_invocab (%) — fenêtres fixes identiques (seen = corpus propre)
2
+
3
+ | modèle | corpus | seen | fc2 | fresh_en | fresh_fr | neither_code | test20k | queries | test_v3 | fresh moy. | écart seen−fresh | wall s | pic RAM Mo | scale cal |
4
+ |---|---|---|---|---|---|---|---|---|---|---|---|---|---|
5
+ | old (réf) | old_local_v1 | 98.33 | 11.50 | 3.40 | 0.28 | 6.95 | 2.90 | 0.00 | 4.12 | 3.70 | 94.63 | 1576 | 710 | 0.035 |
6
+ | broad 14 % (réf) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
7
+ | broad 45 % (cov b) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
8
+ | broad FULL (M8, AM 100 %) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
9
+ | broad_local_v1 cov a | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
10
+ | M8 broad CM 14 % | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
11
+ | corpus_v3 cov b | corpus_v3 | … | … | … | … | … | … | … | … | | en attente | | | |
12
+
13
+ ## McNemar exact par paires (fused_hit, toutes positions)
14
+ | A | B | fenêtre | A seul | B seul | p | signe |
15
+ |---|---|---|---|---|---|---|
v3_scale/reports/scale_bench_v4_smoke.json ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "models": {
3
+ "old (r\u00e9f)": {
4
+ "tag": "old_ref",
5
+ "acc_invocab": {
6
+ "seen": 97.02127659574468,
7
+ "fc2": 16.577540106951872,
8
+ "fresh_en": 2.4752475247524752,
9
+ "fresh_fr": 0.9803921568627451,
10
+ "neither_code": 7.82122905027933,
11
+ "test20k": 1.9900497512437811,
12
+ "queries": 0.0,
13
+ "test_v3": 14.166666666666666
14
+ },
15
+ "fresh_mean": 6.031649367746257,
16
+ "gap": 90.98962722799843,
17
+ "wall_s": 93.3,
18
+ "peak_rss_mb": 710.7,
19
+ "scale": 0.03498957642842646,
20
+ "scale_ok": true
21
+ },
22
+ "broad 14 % (r\u00e9f)": {
23
+ "tag": "broad14_ref",
24
+ "acc_invocab": {
25
+ "seen": 90.98039215686275,
26
+ "fc2": 0.8130081300813009,
27
+ "fresh_en": 3.8461538461538463,
28
+ "fresh_fr": 31.983805668016196,
29
+ "neither_code": 2.8688524590163933,
30
+ "test20k": 2.1459227467811157,
31
+ "queries": 8.064516129032258,
32
+ "test_v3": 0.5847953216374269
33
+ },
34
+ "fresh_mean": 7.906366973617025,
35
+ "gap": 83.07402518324572,
36
+ "wall_s": 105.9,
37
+ "peak_rss_mb": 1280.0,
38
+ "scale": 0.034140001563555386,
39
+ "scale_ok": true
40
+ }
41
+ },
42
+ "pairs": [
43
+ {
44
+ "A": "old (r\u00e9f)",
45
+ "B": "broad 14 % (r\u00e9f)",
46
+ "window": "seen",
47
+ "n": 300,
48
+ "A_only": 25,
49
+ "B_only": 7,
50
+ "p": 0.0021024015732109547
51
+ },
52
+ {
53
+ "A": "old (r\u00e9f)",
54
+ "B": "broad 14 % (r\u00e9f)",
55
+ "window": "fc2",
56
+ "n": 300,
57
+ "A_only": 35,
58
+ "B_only": 2,
59
+ "p": 1.0244548320770264e-08
60
+ },
61
+ {
62
+ "A": "old (r\u00e9f)",
63
+ "B": "broad 14 % (r\u00e9f)",
64
+ "window": "fresh_en",
65
+ "n": 300,
66
+ "A_only": 5,
67
+ "B_only": 9,
68
+ "p": 0.4239501953125
69
+ },
70
+ {
71
+ "A": "old (r\u00e9f)",
72
+ "B": "broad 14 % (r\u00e9f)",
73
+ "window": "fresh_fr",
74
+ "n": 300,
75
+ "A_only": 1,
76
+ "B_only": 79,
77
+ "p": 1.3400325923359048e-22
78
+ },
79
+ {
80
+ "A": "old (r\u00e9f)",
81
+ "B": "broad 14 % (r\u00e9f)",
82
+ "window": "neither_code",
83
+ "n": 300,
84
+ "A_only": 12,
85
+ "B_only": 5,
86
+ "p": 0.143463134765625
87
+ },
88
+ {
89
+ "A": "old (r\u00e9f)",
90
+ "B": "broad 14 % (r\u00e9f)",
91
+ "window": "test20k",
92
+ "n": 300,
93
+ "A_only": 4,
94
+ "B_only": 5,
95
+ "p": 1.0
96
+ },
97
+ {
98
+ "A": "old (r\u00e9f)",
99
+ "B": "broad 14 % (r\u00e9f)",
100
+ "window": "queries",
101
+ "n": 300,
102
+ "A_only": 0,
103
+ "B_only": 15,
104
+ "p": 6.103515625e-05
105
+ },
106
+ {
107
+ "A": "old (r\u00e9f)",
108
+ "B": "broad 14 % (r\u00e9f)",
109
+ "window": "test_v3",
110
+ "n": 300,
111
+ "A_only": 17,
112
+ "B_only": 1,
113
+ "p": 0.00014495849609375
114
+ }
115
+ ]
116
+ }
v3_scale/reports/scale_bench_v4_smoke.md ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Banc scale v4 — fused_acc_invocab (%) — fenêtres fixes identiques (seen = corpus propre)
2
+
3
+ | modèle | corpus | seen | fc2 | fresh_en | fresh_fr | neither_code | test20k | queries | test_v3 | fresh moy. | écart seen−fresh | wall s | pic RAM Mo | scale cal |
4
+ |---|---|---|---|---|---|---|---|---|---|---|---|---|---|
5
+ | old (réf) | old_local_v1 | 97.02 | 16.58 | 2.48 | 0.98 | 7.82 | 1.99 | 0.00 | 14.17 | 6.03 | 90.99 | 93 | 711 | 0.035 |
6
+ | broad 14 % (réf) | broad_local_v1 | 90.98 | 0.81 | 3.85 | 31.98 | 2.87 | 2.15 | 8.06 | 0.58 | 7.91 | 83.07 | 106 | 1280 | 0.034 |
7
+ | broad 45 % (cov b) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
8
+ | broad FULL (M8, AM 100 %) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
9
+ | broad_local_v1 cov a | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
10
+ | M8 broad CM 14 % | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
11
+ | corpus_v3 cov b | corpus_v3 | … | … | … | … | … | … | … | … | | en attente | | | |
12
+
13
+ ## McNemar exact par paires (fused_hit, toutes positions)
14
+ | A | B | fenêtre | A seul | B seul | p | signe |
15
+ |---|---|---|---|---|---|---|
16
+ | old (réf) | broad 14 % (réf) | seen | 25 | 7 | 0.0021 | A>B * |
17
+ | old (réf) | broad 14 % (réf) | fc2 | 35 | 2 | 1e-08 | A>B * |
18
+ | old (réf) | broad 14 % (réf) | fresh_en | 5 | 9 | 0.42 | B>A |
19
+ | old (réf) | broad 14 % (réf) | fresh_fr | 1 | 79 | 1.3e-22 | B>A * |
20
+ | old (réf) | broad 14 % (réf) | neither_code | 12 | 5 | 0.14 | A>B |
21
+ | old (réf) | broad 14 % (réf) | test20k | 4 | 5 | 1 | B>A |
22
+ | old (réf) | broad 14 % (réf) | queries | 0 | 15 | 6.1e-05 | B>A * |
23
+ | old (réf) | broad 14 % (réf) | test_v3 | 17 | 1 | 0.00014 | A>B * |
v3_scale/reports/spira_relay.md ADDED
The diff for this file is too large to render. See raw diff