v3_scale: code banc v4/gen/v3, rapports validés, corpus_v3 (sha256 vérifié), README
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- v3_scale/README.md +22 -0
- v3_scale/code/gen/build_lexicon.py +5 -0
- v3_scale/code/gen/build_prompts.py +30 -0
- v3_scale/code/gen/gen_eval.py +81 -0
- v3_scale/code/recount/recheck_bench_v4_oldref_1722.py +11 -0
- v3_scale/code/recount/recheck_corpus_v3_1700.py +8 -0
- v3_scale/code/recount/recheck_scale_smoke_1706.py +29 -0
- v3_scale/code/recount/recheck_v4broad14_1748.py +10 -0
- v3_scale/code/recount/recheck_v4broad14_1755.py +10 -0
- v3_scale/code/recount/recheck_v4broad14_1758.py +10 -0
- v3_scale/code/recount/recheck_v4code_1733.py +10 -0
- v3_scale/code/recount/recheck_v4fresh_gen_1728.py +52 -0
- v3_scale/code/recount/recheck_v4t20q_1737.py +10 -0
- v3_scale/code/recount/recheck_v4testv3_1743.py +10 -0
- v3_scale/code/scale/bench_common.py +30 -0
- v3_scale/code/scale/eval_bench_v4.py +27 -0
- v3_scale/code/scale/heavy_checker.sh +3 -0
- v3_scale/code/scale/hf_push_new.py +26 -0
- v3_scale/code/scale/jobs_v4.tsv +7 -0
- v3_scale/code/scale/registry_v4.tsv +8 -0
- v3_scale/code/scale/scale_bench_v4.py +39 -0
- v3_scale/code/scale/train_cm_m8_full.py +26 -0
- v3_scale/code/scale/watch_scale.sh +27 -0
- v3_scale/code/v3/audit_overlap.py +31 -0
- v3_scale/code/v3/build_v3_lowmem.py +81 -0
- v3_scale/code/v3verify/verify_v3.py +37 -0
- v3_scale/corpus_v3/corpus_manifest.json +227 -0
- v3_scale/corpus_v3/manifest.json +227 -0
- v3_scale/corpus_v3/oneshot.bin +3 -0
- v3_scale/corpus_v3/test.bin +3 -0
- v3_scale/corpus_v3/train.bin +3 -0
- v3_scale/reports/audit_overlap_broad_local_v1.json +62 -0
- v3_scale/reports/audit_overlap_old_local_v1.json +62 -0
- v3_scale/reports/corpus_v3.sha256 +3 -0
- v3_scale/reports/recheck_bench_v4_oldref_1722.json +30 -0
- v3_scale/reports/recheck_corpus_v3_1700.json +40 -0
- v3_scale/reports/recheck_scale_smoke_1706.json +198 -0
- v3_scale/reports/recheck_v4broad14_1748.json +44 -0
- v3_scale/reports/recheck_v4broad14_1755.json +72 -0
- v3_scale/reports/recheck_v4broad14_1758.json +86 -0
- v3_scale/reports/recheck_v4code_1733.json +72 -0
- v3_scale/reports/recheck_v4fresh_gen_1728.json +368 -0
- v3_scale/reports/recheck_v4t20q_1737.json +100 -0
- v3_scale/reports/recheck_v4testv3_1743.json +114 -0
- v3_scale/reports/results_v4.jsonl +15 -0
- v3_scale/reports/scale_bench_v4.json +24 -0
- v3_scale/reports/scale_bench_v4.md +15 -0
- v3_scale/reports/scale_bench_v4_smoke.json +116 -0
- v3_scale/reports/scale_bench_v4_smoke.md +23 -0
- v3_scale/reports/spira_relay.md +0 -0
v3_scale/README.md
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# SPIRA — v3_scale (Checker, 23 oct. 2026)
|
| 2 |
+
|
| 3 |
+
Ajout sous `v3_scale/` uniquement ; les fichiers d'origine à la racine du dépôt ne sont pas modifiés.
|
| 4 |
+
|
| 5 |
+
## Contenu
|
| 6 |
+
- `code/scale/` : banc v4 (bench_common, eval_bench_v4, scale_bench_v4, registry/jobs_v4, watch_scale.sh, heavy_checker.sh, train_cm_m8_full.py, hf_push_new.*).
|
| 7 |
+
- `code/gen/` : scripts d'éval génération (sans prompts ni échantillons : ce sont des données d'éval, non publiées).
|
| 8 |
+
- `code/v3/` : build_v3_lowmem.py, audit_overlap.py ; `code/v3verify/verify_v3.py` (version corrigée).
|
| 9 |
+
- `code/recount/` : scripts de recompte du banc v4 et de corpus_v3.
|
| 10 |
+
- `reports/` : scale_bench_v4*.md/.json, results_v4.jsonl, audit_overlap_*.json, recomptes recheck_*.json, spira_relay.md.
|
| 11 |
+
- `corpus_v3/` : manifest.json, corpus_manifest.json, train.bin, test.bin, oneshot.bin (sha256 vérifiés contre le manifeste ; sans recouvrement avec les fenêtres d'éval). raw/ non inclus.
|
| 12 |
+
- `checkpoints/` : seulement les checkpoints terminés et calibrés (fusion.scale ∈ [0,03 ; 0,11]).
|
| 13 |
+
|
| 14 |
+
## Chiffres validés (banc v4, fused_acc_invocab %, n = 8 000 par fenêtre)
|
| 15 |
+
- old réf (old_local_v1, scale 0,034) : seen 98,33 ; fc2 11,50 ; fresh_en 3,40 ; fresh_fr 0,28 ; neither_code 6,95 ; test20k 2,90 ; queries 0,00 ; test_v3 4,12.
|
| 16 |
+
- broad14 réf (broad_local_v1, scale 0,035), recompté à ce jour : seen 91,93 ; fc2 8,46 ; fresh_en 5,17 ; fresh_fr 2,72 ; neither_code 7,67 ; test20k 3,23. queries et test_v3 en cours.
|
| 17 |
+
|
| 18 |
+
## Verdicts
|
| 19 |
+
- broad14 comme remplaçant de old : **rejeté** (pertes significatives sur fc2, test_v3, seen).
|
| 20 |
+
- Génération broad14 vs old : aucun gain significatif (fused p = 0,148 ; v2b p = 0,664) ; génération encore incohérente (byte_acc ≤ 9 %).
|
| 21 |
+
- V1m (décodage) : retenu, aucune perte sur 6 fenêtres.
|
| 22 |
+
- M8 FULL, cov a/b, M8 14 %, corpus_v3 cov b : pas encore de verdict ; poussés automatiquement quand terminés et calibrés.
|
v3_scale/code/gen/build_lexicon.py
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Lexique FR+EN : mots (minuscules, ≥2 lettres) de fréquence ≥3 dans corpus_v3/train.bin. → lexicon_v3.json"""
|
| 2 |
+
import re, json, collections
|
| 3 |
+
WRE = re.compile(r"[a-zà-ÿœæç']+"); c = collections.Counter(); f = open("/workspace/spira_shared/corpus_v3/train.bin", "rb")
|
| 4 |
+
while (b := f.read(4_000_000)): c.update(w.strip("'") for w in WRE.findall(b.decode("utf-8", "ignore").lower()))
|
| 5 |
+
L = sorted(w for w, n in c.items() if n >= 3 and len(w) >= 2); json.dump(L, open("lexicon_v3.json", "w"), ensure_ascii=False); print(len(L))
|
v3_scale/code/gen/build_prompts.py
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — prompts de génération (≥5, frais). Prompt 192 o + continuation vraie 200 o, coupés sur espace.
|
| 3 |
+
Vérif : 0 fenêtre de 64 o (prompt+continuation) présente dans old_local_v1, broad_local_v1, corpus_v3 train.bin (scan exhaustif par blocs, set de petite taille).
|
| 4 |
+
Si chevauchement : décale de 4 Ko et réessaie. Sortie prompts_v1.json (+ sha256)."""
|
| 5 |
+
import sys, json, hashlib, numpy as np
|
| 6 |
+
sys.path.insert(0, "/workspace/spira_shared/checker/scale"); sys.path.insert(0, "/workspace/spira_shared/checker/fc2")
|
| 7 |
+
from bench_common import fixed_windows, CORPORA, rd
|
| 8 |
+
from rollhash import hashes
|
| 9 |
+
W = fixed_windows(); SRC = [("en1", "fresh_en", 20_000), ("en2", "fresh_en", 70_000), ("fr1", "fresh_fr", 30_000), ("fr2", "fresh_fr", 90_000),
|
| 10 |
+
("chat", "queries", 5_000), ("code", "fc2", 40_000), ("v3test", "test_v3", 1_500_000)]
|
| 11 |
+
def cut(b, off):
|
| 12 |
+
s = b.index(b" ", off) + 1; return b[s:s + 192], b[s + 192:s + 392]
|
| 13 |
+
C = {k: rd(f"{p}/train.bin") for k, p in CORPORA.items()}
|
| 14 |
+
def overlap(seg):
|
| 15 |
+
h = np.unique(hashes(seg)); r = {}
|
| 16 |
+
for k, tr in C.items():
|
| 17 |
+
n = 0
|
| 18 |
+
for s in range(0, len(tr), 8_000_000): n += int(np.isin(hashes(tr[s:s + 8_000_063]), h).sum())
|
| 19 |
+
r[k] = n
|
| 20 |
+
return r
|
| 21 |
+
out = []
|
| 22 |
+
for name, w, off in SRC:
|
| 23 |
+
b = W[w][0]
|
| 24 |
+
for tries in range(6):
|
| 25 |
+
p, c = cut(b, off); ov = overlap(p + c)
|
| 26 |
+
if sum(ov.values()) == 0: break
|
| 27 |
+
off += 4096
|
| 28 |
+
out.append({"name": name, "window": w, "offset": off, "prompt": p.decode("utf-8", "replace"), "cont": c.decode("utf-8", "replace"),
|
| 29 |
+
"prompt_b64": p.hex(), "cont_hex": c.hex(), "overlap64": ov, "fresh": sum(ov.values()) == 0}); print(name, off, ov, flush=True)
|
| 30 |
+
json.dump(out, open("prompts_v1.json", "w"), ensure_ascii=False, indent=1); print(hashlib.sha256(open("prompts_v1.json", "rb").read()).hexdigest())
|
v3_scale/code/gen/gen_eval.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — éval de génération v1. Usage : gen_eval.py CKPT CORPUS_NAME TAG (PYTHONHASHSEED=0, OMP_NUM_THREADS=1)
|
| 3 |
+
Prompts : prompts_v1.json (7 prompts frais, 64-o overlap = 0 avec old/broad/v3 train). 200 octets générés par prompt.
|
| 4 |
+
Décodages (aucune variante nouvelle) :
|
| 5 |
+
État : snapshot SNN (core.snapshot + W_rec) restauré avant CHAQUE génération (SPIRA.generate seul n'est pas déterministe d'un appel à l'autre : plasticité).
|
| 6 |
+
fused : boucle = copie de SPIRA.generate du repo (top_k=cfg.top_k, T=cfg.temperature, seed=0) ; vérifiée == m.generate sur le 1er prompt.
|
| 7 |
+
v2b : référence m9g v2b — même boucle, mais si c_AM=1-exp(-am_w/m)<0.5 : scores = logsoftmax(fused) + 2·log p_JM(col|p2,p1)
|
| 8 |
+
(NGram JM 0.2/0.5/0.3 copiée de m9g/eval_m9g_v2.py, comptée sur le train du modèle).
|
| 9 |
+
Métriques vs vraie continuation : byte_acc (octet à octet sur 200), valid_word_rate (lexique corpus_v3 freq≥3, FR+EN), rep_rate (8-grammes d'octets
|
| 10 |
+
déjà vus dans la génération), lang/domain (stopwords FR/EN ; code si ≥4 % de []{}()=;_#<>) et drift vs continuation, longest_copy (plus long extrait
|
| 11 |
+
de la génération présent dans le train du modèle). Sortie results_gen_v1.jsonl + samples/."""
|
| 12 |
+
import os, sys, json, re, time, resource
|
| 13 |
+
sys.path.insert(0, "/workspace/spira_shared/checker/scale")
|
| 14 |
+
import numpy as np
|
| 15 |
+
from bench_common import load_model, CORPORA, rd, fnv1a64
|
| 16 |
+
D = os.path.dirname(os.path.abspath(__file__)); ckpt, cname, tag = sys.argv[1:4]; NB = 200; GATE, LAMB = 0.5, 2.0; JM = (0.2, 0.5, 0.3)
|
| 17 |
+
OUT = os.environ.get("OUT", f"{D}/results_gen_v1.jsonl")
|
| 18 |
+
P = json.load(open(f"{D}/prompts_v1.json")); LEX = set(json.load(open(f"{D}/lexicon_v3.json")))
|
| 19 |
+
t0 = time.time(); train = rd(f"{CORPORA[cname]}/train.bin"); m, kind, saved, cal = load_model(ckpt, train)
|
| 20 |
+
V = m.cfg.vocab_size; K = V + 1
|
| 21 |
+
src = open("/workspace/spira_shared/checker/m9g/eval_m9g_v2.py").read(); g = {"np": np, "K": K, "JM": JM}
|
| 22 |
+
exec(src[src.index("class NGram"):src.index("ng = NGram")], g); ng = g["NGram"](m.cols_of(m._ids_train))
|
| 23 |
+
S0 = (m.core.snapshot(), m.core.W_rec.copy())
|
| 24 |
+
def pristine(): m.core.restore(S0[0]); m.core.W_rec = S0[1].copy()
|
| 25 |
+
def lsm(u): z = u - u.max(); return z - np.log(np.exp(z).sum())
|
| 26 |
+
def gen(prompt, n_tokens, v2b, seed=0):
|
| 27 |
+
pristine(); cfg = m.cfg; rng = np.random.default_rng(seed); m.core.reset_dynamics(); m._win_count = 0
|
| 28 |
+
ids = [fnv1a64(p) for p in m.ept.segment(prompt)]; out = bytearray(prompt); x = np.zeros(m.core.n_res, dtype=np.float32); ng_used = 0
|
| 29 |
+
for pid in ids: x = m._feed(pid)
|
| 30 |
+
for _ in range(n_tokens):
|
| 31 |
+
fused, am_val, am_w, bypass_id = m.fuse_scores(x, ids)
|
| 32 |
+
if bypass_id is not None: chosen_id = bypass_id
|
| 33 |
+
else:
|
| 34 |
+
scores = np.asarray(fused, dtype=np.float64).copy()
|
| 35 |
+
if v2b and 1.0 - np.exp(-am_w / m.am.m) < GATE:
|
| 36 |
+
cc = m.cols_of(np.array(ids[-2:], dtype=np.uint64)); p1 = int(cc[-1]); p2 = int(cc[0]) if len(cc) > 1 else None
|
| 37 |
+
scores = lsm(scores[:K]) + LAMB * ng.logp(p2, p1); ng_used += 1
|
| 38 |
+
scores[V] = -np.inf; k = max(1, min(cfg.top_k, V)); top = np.argpartition(scores, -k)[-k:]
|
| 39 |
+
w = np.exp((scores[top] - scores[top].max()) / max(cfg.temperature, 1e-6)); w /= w.sum()
|
| 40 |
+
chosen_id = int(m.vocab_ids[int(top[rng.choice(len(top), p=w)])])
|
| 41 |
+
b = m.registry.bytes_of(chosen_id)
|
| 42 |
+
if b is None: break
|
| 43 |
+
out.extend(b); ids.append(chosen_id); x = m._feed(chosen_id)
|
| 44 |
+
if len(out) - len(prompt) >= NB: break
|
| 45 |
+
return bytes(out[len(prompt):len(prompt) + NB]), ng_used
|
| 46 |
+
FR = set("le la les de des du un une et est que qui dans pour pas sur au aux il elle ne se ce son sa ses avec par plus mais".split())
|
| 47 |
+
EN = set("the of and to a in is that it was for on with as his her he she they not but at by be this had".split())
|
| 48 |
+
WRE = re.compile(r"[a-zà-ÿœæç']+")
|
| 49 |
+
def words(t): return [w.strip("'") for w in WRE.findall(t.lower()) if len(w.strip("'")) >= 2]
|
| 50 |
+
def lang(t):
|
| 51 |
+
ws = words(t); f = sum(w in FR for w in ws); e = sum(w in EN for w in ws); return "fr" if f > e else ("en" if e > f else "?")
|
| 52 |
+
def dom(t): return "code" if t and sum(c in "[]{}()=;_#<>" for c in t) / len(t) >= 0.04 else "prose"
|
| 53 |
+
def longest_copy(gb):
|
| 54 |
+
L, best = 0, b""
|
| 55 |
+
for i in range(len(gb)):
|
| 56 |
+
while i + L < len(gb) and gb[i:i + L + 1] in train: L += 1; best = gb[i:i + L]
|
| 57 |
+
return L, best
|
| 58 |
+
def rep(gb, n=8):
|
| 59 |
+
seen, r, tot = set(), 0, 0
|
| 60 |
+
for i in range(len(gb) - n + 1):
|
| 61 |
+
s = gb[i:i + n]; tot += 1; r += s in seen; seen.add(s)
|
| 62 |
+
return r / max(tot, 1)
|
| 63 |
+
os.makedirs(f"{D}/samples", exist_ok=True); tl = time.time() - t0
|
| 64 |
+
p0 = bytes.fromhex(P[0]["prompt_b64"]); pristine(); ref = m.generate(p0, 120, seed=0)[len(p0):][:NB]; mine, _ = gen(p0, 120, False)
|
| 65 |
+
mine2, _ = gen(p0, 120, False)
|
| 66 |
+
assert mine == mine2, "non déterministe"
|
| 67 |
+
assert ref[:len(mine)] == mine[:len(ref)], "boucle fused != SPIRA.generate"
|
| 68 |
+
print("CHECK determinism OK, loop==SPIRA.generate OK", flush=True)
|
| 69 |
+
for dec in ("fused", "v2b"):
|
| 70 |
+
for p in P:
|
| 71 |
+
pb, cb = bytes.fromhex(p["prompt_b64"]), bytes.fromhex(p["cont_hex"]); t1 = time.time(); gb, nu = gen(pb, 400, dec == "v2b")
|
| 72 |
+
gt, ct = gb.decode("utf-8", "replace"), cb.decode("utf-8", "replace"); ws = words(gt); L, ex = longest_copy(gb)
|
| 73 |
+
r = {"tag": tag, "corpus": cname, "kind": kind, "decoding": dec, "prompt": p["name"], "n_bytes": len(gb),
|
| 74 |
+
"byte_acc": sum(a == b for a, b in zip(gb, cb)) / NB, "valid_word_rate": sum(w in LEX for w in ws) / max(len(ws), 1), "n_words": len(ws),
|
| 75 |
+
"rep_rate8": rep(gb), "lang_true": lang(ct), "lang_gen": lang(gt), "dom_true": dom(ct), "dom_gen": dom(gt),
|
| 76 |
+
"lang_drift": lang(gt) != lang(ct), "dom_drift": dom(gt) != dom(ct), "longest_copy": L, "copy_excerpt": ex.decode("utf-8", "replace"),
|
| 77 |
+
"ngram_steps": nu, "gen_s": round(time.time() - t1, 1), "load_s": round(tl, 1), "fusion_cal": cal,
|
| 78 |
+
"peak_rss_mb": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024), "seed": 0}
|
| 79 |
+
open(OUT, "a").write(json.dumps(r, ensure_ascii=False) + "\n")
|
| 80 |
+
open(f"{D}/samples/{tag}__{dec}__{p['name']}.txt", "w").write(f"PROMPT:\n{p['prompt']}\n\nVRAI:\n{ct}\n\nGÉNÉRÉ:\n{gt}\n")
|
| 81 |
+
print(dec, p["name"], f"acc={r['byte_acc']:.3f} vw={r['valid_word_rate']:.2f} rep={r['rep_rate8']:.2f} {r['lang_gen']}/{r['lang_true']} {r['dom_gen']}/{r['dom_true']} copy={L}", flush=True)
|
v3_scale/code/recount/recheck_bench_v4_oldref_1722.py
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/scale/'
|
| 3 |
+
rep=[json.loads(l) for l in open(B+'results_v4.jsonl')]
|
| 4 |
+
out={}
|
| 5 |
+
for r in rep:
|
| 6 |
+
w=r['window']; d=np.load(B+'hits_v4/%s__%s.npz'%(r['tag'],w))
|
| 7 |
+
iv=d['invocab']; n=len(iv); ni=int(iv.sum())
|
| 8 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 9 |
+
out[w]=dict(n=n,invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/n,
|
| 10 |
+
reported=(r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']))
|
| 11 |
+
json.dump(out,open('recheck_bench_v4_oldref_1722.json','w'),indent=1); print(json.dumps(out,indent=1))
|
v3_scale/code/recount/recheck_corpus_v3_1700.py
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import json,hashlib,os
|
| 2 |
+
d='/workspace/spira_shared/corpus_v3/'
|
| 3 |
+
m=json.load(open(d+'manifest.json'))
|
| 4 |
+
out={'sizes':{f:os.path.getsize(d+f) for f in ['train.bin','test.bin','oneshot.bin']}}
|
| 5 |
+
out['sha_ok']={k:hashlib.sha256(open(d+k if k.endswith('.bin') else d+k+'.bin','rb').read()).hexdigest()==v for k,v in m['sha256'].items()}
|
| 6 |
+
out['overlap64_final']=m['overlap64_final'];out['heldout']=m['heldout_books'];out['n_books']=len(m['books'])
|
| 7 |
+
out['comp']=m['composition_pre_filter'];out['manifests_identical']=open(d+'manifest.json').read()==open(d+'corpus_manifest.json').read()
|
| 8 |
+
json.dump(out,open('/workspace/spira_shared/checker/recount/recheck_corpus_v3_1700.json','w'),indent=1);print(json.dumps(out))
|
v3_scale/code/recount/recheck_scale_smoke_1706.py
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
from math import comb
|
| 3 |
+
H='/workspace/spira_shared/checker/scale/smoke/hits/'
|
| 4 |
+
W=['seen','fc2','fresh_en','fresh_fr','neither_code','test20k','queries','test_v3']
|
| 5 |
+
def mcn(b,c):
|
| 6 |
+
n=b+c
|
| 7 |
+
if n==0: return 1.0
|
| 8 |
+
k=min(b,c); p=sum(comb(n,i) for i in range(k+1))/2**n*2
|
| 9 |
+
return min(1.0,p)
|
| 10 |
+
out={'acc':{},'pairs':{}}
|
| 11 |
+
for t in ['old_ref','broad14_ref']:
|
| 12 |
+
a={}
|
| 13 |
+
for w in W:
|
| 14 |
+
d=np.load(H+f'{t}__{w}.npz'); iv=d['invocab']; h=d['fused_hit']
|
| 15 |
+
a[w]=dict(n=int(len(h)),invocab=int(iv.sum()),hits_in=int((h&iv).sum()),hits_all=int(h.sum()),pct=100*(h&iv).sum()/max(1,iv.sum()))
|
| 16 |
+
fr=[a[w]['pct'] for w in W if w not in('seen','neither_code')]
|
| 17 |
+
fr7=[a[w]['pct'] for w in W if w!='seen']
|
| 18 |
+
out['acc'][t]=dict(win=a,fresh_mean_excl_nc=float(np.mean(fr)),gap_excl_nc=a['seen']['pct']-float(np.mean(fr)),fresh_mean_incl_nc=float(np.mean(fr7)),gap_incl_nc=a['seen']['pct']-float(np.mean(fr7)))
|
| 19 |
+
for w in W:
|
| 20 |
+
A=np.load(H+f'old_ref__{w}.npz');B=np.load(H+f'broad14_ref__{w}.npz')
|
| 21 |
+
ha=A['fused_hit']&A['invocab']; hb=B['fused_hit']&B['invocab']
|
| 22 |
+
b=int((ha&~hb).sum()); c=int((~ha&hb).sum())
|
| 23 |
+
ha2=A['fused_hit'];hb2=B['fused_hit']
|
| 24 |
+
b2=int((ha2&~hb2).sum()); c2=int((~ha2&hb2).sum())
|
| 25 |
+
out['pairs'][w]=dict(A_only=b,B_only=c,p=mcn(b,c),raw_A_only=b2,raw_B_only=c2,raw_p=mcn(b2,c2))
|
| 26 |
+
json.dump(out,open('recheck_scale_smoke_1706.json','w'),indent=1,default=float)
|
| 27 |
+
for t,v in out['acc'].items():
|
| 28 |
+
print(t,{w:(x['hits_in'],x['invocab'],round(x['pct'],2)) for w,x in v['win'].items()},'freshNoNC',round(v['fresh_mean_excl_nc'],2),'gap',round(v['gap_excl_nc'],2),'freshNC',round(v['fresh_mean_incl_nc'],2),'gapNC',round(v['gap_incl_nc'],2))
|
| 29 |
+
for w,v in out['pairs'].items(): print(w,v)
|
v3_scale/code/recount/recheck_v4broad14_1748.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/'
|
| 3 |
+
out={}
|
| 4 |
+
for r in [x for x in (json.loads(l) for l in open(B+'scale/results_v4.jsonl')) if x['tag']=='broad14_ref']:
|
| 5 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 6 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 7 |
+
out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 8 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 9 |
+
json.dump(out,open('recheck_v4broad14_1748.json','w'),indent=1)
|
| 10 |
+
print(json.dumps(out,indent=1))
|
v3_scale/code/recount/recheck_v4broad14_1755.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/'
|
| 3 |
+
out={}
|
| 4 |
+
for r in [x for x in (json.loads(l) for l in open(B+'scale/results_v4.jsonl')) if x['tag']=='broad14_ref']:
|
| 5 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 6 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 7 |
+
out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 8 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 9 |
+
json.dump(out,open('recheck_v4broad14_1755.json','w'),indent=1)
|
| 10 |
+
print(json.dumps(out,indent=1))
|
v3_scale/code/recount/recheck_v4broad14_1758.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/'
|
| 3 |
+
out={}
|
| 4 |
+
for r in [x for x in (json.loads(l) for l in open(B+'scale/results_v4.jsonl')) if x['tag']=='broad14_ref']:
|
| 5 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 6 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 7 |
+
out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 8 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 9 |
+
json.dump(out,open('recheck_v4broad14_1758.json','w'),indent=1)
|
| 10 |
+
print(json.dumps(out,indent=1))
|
v3_scale/code/recount/recheck_v4code_1733.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/'
|
| 3 |
+
out={}
|
| 4 |
+
for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
|
| 5 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 6 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 7 |
+
out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 8 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 9 |
+
json.dump(out,open('recheck_v4code_1733.json','w'),indent=1)
|
| 10 |
+
print(json.dumps(out,indent=1))
|
v3_scale/code/recount/recheck_v4fresh_gen_1728.py
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json, re
|
| 2 |
+
from math import comb
|
| 3 |
+
B='/workspace/spira_shared/checker/'
|
| 4 |
+
out={'bench':{}, 'gen':{}}
|
| 5 |
+
for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
|
| 6 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 7 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 8 |
+
out['bench'][r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 9 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 10 |
+
# gen: recount from samples (text decoded with 'replace' -> byte recount exact only if re-encoded length/bytes match)
|
| 11 |
+
LEX=set(json.load(open(B+'gen/lexicon_v3.json'))) if True else None
|
| 12 |
+
P={p['name']:p for p in json.load(open(B+'gen/prompts_v1.json'))}
|
| 13 |
+
WRE=re.compile(r"[a-zà-ÿœæç']+")
|
| 14 |
+
def words(t): return [w.strip("'") for w in WRE.findall(t.lower()) if len(w.strip("'"))>=2]
|
| 15 |
+
def mcn(b,c):
|
| 16 |
+
n=b+c
|
| 17 |
+
if n==0: return 1.0
|
| 18 |
+
k=min(b,c); return min(1.0,2*sum(comb(n,i) for i in range(k+1))/2**n)
|
| 19 |
+
rows=[json.loads(l) for l in open(B+'gen/results_gen_v1.jsonl')]
|
| 20 |
+
hits={}
|
| 21 |
+
for r in rows:
|
| 22 |
+
t=open(B+'gen/samples/%s__%s__%s.txt'%(r['tag'],r['decoding'],r['prompt'])).read()
|
| 23 |
+
gt=t.split('\nGÉNÉRÉ:\n',1)[1][:-1]; gb=gt.encode(); cb=bytes.fromhex(P[r['prompt']]['cont_hex'])
|
| 24 |
+
h=[a==b for a,b in zip(gb[:200],cb[:200])]+[False]*max(0,200-len(gb[:200]))
|
| 25 |
+
ws=words(gt); vw=sum(w in LEX for w in ws)/max(len(ws),1)
|
| 26 |
+
exact=('\ufffd' not in gt)
|
| 27 |
+
hits[(r['tag'],r['decoding'],r['prompt'])]=h
|
| 28 |
+
out['gen']['%s/%s/%s'%(r['tag'],r['decoding'],r['prompt'])]=dict(byte_acc=sum(h)/200,rep_byte_acc=r['byte_acc'],exact_bytes=exact,vw=round(vw,4),rep_vw=round(r['valid_word_rate'],4),nw=len(ws),rep_nw=r['n_words'])
|
| 29 |
+
agg={}
|
| 30 |
+
for tag in ('old_ref','broad14_ref'):
|
| 31 |
+
for dec in ('fused','v2b'):
|
| 32 |
+
rs=[r for r in rows if r['tag']==tag and r['decoding']==dec]
|
| 33 |
+
agg[tag+'/'+dec]=dict(n=len(rs),byte_acc=round(np.mean([r['byte_acc'] for r in rs]),4),vw=round(np.mean([r['valid_word_rate'] for r in rs]),4),
|
| 34 |
+
rep8=round(np.mean([r['rep_rate8'] for r in rs]),4),lang_drift=sum(r['lang_drift'] for r in rs),dom_drift=sum(r['dom_drift'] for r in rs),
|
| 35 |
+
copy_mean=round(np.mean([r['longest_copy'] for r in rs]),1))
|
| 36 |
+
out['gen_agg']=agg
|
| 37 |
+
# paired McNemar broad14 vs old on per-byte hits (1400 positions per decoding)
|
| 38 |
+
mc={}
|
| 39 |
+
for dec in ('fused','v2b'):
|
| 40 |
+
b=c=0;ho=hb=0
|
| 41 |
+
for p in P:
|
| 42 |
+
if ('old_ref',dec,p) not in hits: continue
|
| 43 |
+
for x,y in zip(hits[('old_ref',dec,p)],hits[('broad14_ref',dec,p)]):
|
| 44 |
+
ho+=x;hb+=y;b+=x and not y;c+=y and not x
|
| 45 |
+
mc[dec]=dict(old_hits=ho,broad_hits=hb,old_only=b,broad_only=c,p=mcn(b,c))
|
| 46 |
+
out['mcnemar_gen_bytes']=mc
|
| 47 |
+
json.dump(out,open('recheck_v4fresh_gen_1728.json','w'),indent=1,ensure_ascii=False)
|
| 48 |
+
print(json.dumps({k:out[k] for k in ('bench','gen_agg','mcnemar_gen_bytes')},indent=1))
|
| 49 |
+
bad=[k for k,v in out['gen'].items() if v['exact_bytes'] and (abs(v['byte_acc']-v['rep_byte_acc'])>1e-9)]
|
| 50 |
+
vwbad=[k for k,v in out['gen'].items() if abs(v['vw']-v['rep_vw'])>1e-3]
|
| 51 |
+
print('byte_acc mismatches (exact samples):',bad); print('vw mismatches:',vwbad)
|
| 52 |
+
print('non-exact samples:',[k for k,v in out['gen'].items() if not v['exact_bytes']])
|
v3_scale/code/recount/recheck_v4t20q_1737.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/'
|
| 3 |
+
out={}
|
| 4 |
+
for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
|
| 5 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 6 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 7 |
+
out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 8 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 9 |
+
json.dump(out,open('recheck_v4t20q_1737.json','w'),indent=1)
|
| 10 |
+
print(json.dumps(out,indent=1))
|
v3_scale/code/recount/recheck_v4testv3_1743.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np, json
|
| 2 |
+
B='/workspace/spira_shared/checker/'
|
| 3 |
+
out={}
|
| 4 |
+
for r in [json.loads(l) for l in open(B+'scale/results_v4.jsonl')]:
|
| 5 |
+
w=r['window']; d=np.load(B+'scale/hits_v4/%s__%s.npz'%(r['tag'],w)); iv=d['invocab']; ni=int(iv.sum())
|
| 6 |
+
f=int((d['fused_hit']&iv).sum()); s=int((d['snn_hit']&iv).sum())
|
| 7 |
+
out[r['tag']+'/'+w]=dict(n=len(iv),invocab=ni,fused=f,snn=s,fused_pct=round(100*f/ni,3),snn_pct=round(100*s/ni,3),cov=ni/len(iv),
|
| 8 |
+
reported=[r['fused_invocab_pct'],r['snn_invocab_pct'],r['vocab_coverage']])
|
| 9 |
+
json.dump(out,open('recheck_v4testv3_1743.json','w'),indent=1)
|
| 10 |
+
print(json.dumps(out,indent=1))
|
v3_scale/code/scale/bench_common.py
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Checker — commun banc v4 + génération : chargement modèle (auto cm/plain), fenêtres FIXES identiques pour tous les modèles.
|
| 2 |
+
seen = train[2_600_000:2_800_000] du corpus propre du modèle (seule fenêtre dépendante du corpus) ; toutes les autres sont des octets fixes."""
|
| 3 |
+
import os, sys, glob, json
|
| 4 |
+
sys.path.insert(0, "/workspace/SPIRA/python"); sys.path.insert(0, "/workspace/spira_shared/semhash"); sys.path.insert(0, "/workspace/spira_shared/checker")
|
| 5 |
+
import numpy as np
|
| 6 |
+
from spira.model import SPIRA
|
| 7 |
+
from spira.hashing import fnv1a64
|
| 8 |
+
S = "/workspace/spira_shared"; T = "/workspace/repos/toroidal-fractal-intelligence/data"; B = f"{S}/broad/broad_local_v1"
|
| 9 |
+
CORPORA = {"old_local_v1": f"{S}/broad/old_local_v1", "broad_local_v1": B, "corpus_v3": f"{S}/corpus_v3"}
|
| 10 |
+
rd = lambda p: open(p, "rb").read()
|
| 11 |
+
def fixed_windows():
|
| 12 |
+
from windows_v2 import freshcode_v2
|
| 13 |
+
from build_corpus_local import strip_gutenberg_header
|
| 14 |
+
return {"fc2": (freshcode_v2(), 8000), "fresh_en": (rd(f"{B}/oneshot.bin"), 8000),
|
| 15 |
+
"fresh_fr": (strip_gutenberg_header(rd(f"{T}/contes_perrault.txt")), 8000),
|
| 16 |
+
"neither_code": (b"\n\n".join(rd(p) for p in sorted(glob.glob("/workspace/kahnn/data/code/*"))), 8000),
|
| 17 |
+
"test20k": (rd(f"{B}/test.bin"), 20000), "queries": (rd(f"{T}/chat_seed_fr_en.txt"), 8000),
|
| 18 |
+
"test_v3": (rd(f"{S}/corpus_v3/test.bin"), 20000)}
|
| 19 |
+
def kind_of(ckpt):
|
| 20 |
+
return "cm" if "cm_params" in np.load(ckpt, allow_pickle=False).files else "plain"
|
| 21 |
+
def load_model(ckpt, train):
|
| 22 |
+
k = kind_of(ckpt)
|
| 23 |
+
if k == "cm":
|
| 24 |
+
import countmin
|
| 25 |
+
m = SPIRA.load.__func__(countmin.CMSPIRA, ckpt); z = np.load(ckpt, allow_pickle=False)
|
| 26 |
+
m.cm = json.loads(str(z["cm_params"])); m.cm_sem = z["cm_sem"].astype(np.int32); m._row_cache = {}
|
| 27 |
+
else: m = SPIRA.load(ckpt)
|
| 28 |
+
ps = m.ept.segment(train); m._ids_train = np.fromiter((fnv1a64(p) for p in ps), dtype=np.uint64, count=len(ps))
|
| 29 |
+
saved = dict(m.fusion); cal = m.calibrate_fusion(train[:60_000])
|
| 30 |
+
return m, k, saved, cal
|
v3_scale/code/scale/eval_bench_v4.py
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — banc scale v4. Usage : eval_bench_v4.py CKPT CORPUS_NAME TAG env MAXPOS (cap/fenêtre, smoke) WINDOWS HITS OUT
|
| 3 |
+
Fenêtres : seen (corpus propre) + fc2, fresh_en, fresh_fr, neither_code, test20k, queries, test_v3 (octets fixes, identiques pour tous).
|
| 4 |
+
Boucle par position = eval_hits() de checker/eval_ref_v3.py (copie ligne à ligne de SPIRA.eval_stream), importée telle quelle.
|
| 5 |
+
Enregistre wall-time et pic RSS (ru_maxrss) par fenêtre."""
|
| 6 |
+
import os, sys, json, time, resource
|
| 7 |
+
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
| 8 |
+
import numpy as np
|
| 9 |
+
from bench_common import *
|
| 10 |
+
ckpt, cname, tag = sys.argv[1:4]; D = os.path.dirname(os.path.abspath(__file__))
|
| 11 |
+
HITS = os.environ.get("HITS", f"{D}/hits_v4"); OUT = os.environ.get("OUT", f"{D}/results_v4.jsonl"); MAXPOS = int(os.environ.get("MAXPOS", "0")); SEL = os.environ.get("WINDOWS")
|
| 12 |
+
src = open(f"{S}/checker/eval_ref_v3.py").read(); g = {"np": np, "fnv1a64": fnv1a64}
|
| 13 |
+
exec(src[src.index("def eval_hits"):src.index("m = SPIRA.load(ckpt)")], g); eval_hits = g["eval_hits"]
|
| 14 |
+
t0 = time.perf_counter(); train = rd(f"{CORPORA[cname]}/train.bin")
|
| 15 |
+
m, kind, saved, cal = load_model(ckpt, train); tload = time.perf_counter() - t0
|
| 16 |
+
W = {"seen": (train[2_600_000:2_800_000], 8000), **fixed_windows()}; del train
|
| 17 |
+
os.makedirs(HITS, exist_ok=True)
|
| 18 |
+
for name, (data, mp) in W.items():
|
| 19 |
+
if SEL and name not in SEL.split(","): continue
|
| 20 |
+
mp = min(mp, MAXPOS) if MAXPOS else mp; t1 = time.perf_counter(); r, H = eval_hits(m, data, max_positions=mp)
|
| 21 |
+
np.savez_compressed(f"{HITS}/{tag}__{name}.npz", **H)
|
| 22 |
+
rec = {"tag": tag, "corpus": cname, "ckpt": ckpt, "kind": kind, "window": name, "n_positions": r["n_positions"],
|
| 23 |
+
"fused_invocab_pct": round(100 * r["fused_acc_invocab"], 3), "snn_invocab_pct": round(100 * r["snn_acc_invocab"], 3),
|
| 24 |
+
"vocab_coverage": r["vocab_coverage"], "fusion_saved": saved, "fusion_cal": cal, "load_s": round(tload, 1),
|
| 25 |
+
"eval_s": round(time.perf_counter() - t1, 1), "peak_rss_mb": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 1),
|
| 26 |
+
"maxpos": MAXPOS, "script": "eval_bench_v4"}
|
| 27 |
+
open(OUT, "a").write(json.dumps(rec) + "\n"); print(name, rec["fused_invocab_pct"], rec["eval_s"], rec["peak_rss_mb"], flush=True)
|
v3_scale/code/scale/heavy_checker.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# nombre de process lourds Checker (python uniquement)
|
| 3 |
+
pgrep -f "^/workspace/SPIRA-venv/bin/python .*(checker/m10c/tf_v6|checker/gen/|gen_eval.py|eval_bench_v4|checker/scale/|checker/v3/|build_v3|train_cm_m8.py broad_local_v1|eval_ref_v2_cm.py .*checker/m8)" | wc -l
|
v3_scale/code/scale/hf_push_new.py
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — push HF sous v3_scale/ uniquement (thefinalboss/SPIRA). Jamais de delete, jamais d'écriture hors v3_scale/.
|
| 3 |
+
Usage : hf_push_new.py "message" local1=repo/path1 local2=repo/path2 ...
|
| 4 |
+
- refuse tout chemin repo hors v3_scale/ ; refuse d'écraser un checkpoint (.npz) déjà présent sur le repo.
|
| 5 |
+
- refuse les données d'éval (hits*.npz, samples/, prompts_v1.json, results_gen*, corpus_gutenberg).
|
| 6 |
+
- log : checker/scale/hf_push.log (heure Toronto, URL commit, fichiers+tailles)."""
|
| 7 |
+
import os, sys, datetime, zoneinfo
|
| 8 |
+
from huggingface_hub import HfApi, CommitOperationAdd
|
| 9 |
+
REPO = "thefinalboss/SPIRA"; LOG = "/workspace/spira_shared/checker/scale/hf_push.log"
|
| 10 |
+
BAN = ("hits", "/samples/", "prompts_v1.json", "results_gen", "corpus_gutenberg", ".env", "token")
|
| 11 |
+
def main():
|
| 12 |
+
msg, pairs = sys.argv[1], [a.split("=", 1) for a in sys.argv[2:]]
|
| 13 |
+
api = HfApi(); remote = set(api.list_repo_files(REPO)); ops, lst = [], []
|
| 14 |
+
for loc, rp in pairs:
|
| 15 |
+
assert rp.startswith("v3_scale/") and ".." not in rp, rp
|
| 16 |
+
assert not any(b in loc or b in rp for b in BAN), f"interdit: {loc}"
|
| 17 |
+
if rp.endswith(".npz") and rp in remote: print("déjà présent, ignoré:", rp); continue
|
| 18 |
+
ops.append(CommitOperationAdd(path_in_repo=rp, path_or_fileobj=loc)); lst.append((rp, os.path.getsize(loc)))
|
| 19 |
+
if not ops: print("rien à pousser"); return
|
| 20 |
+
ci = api.create_commit(REPO, operations=ops, commit_message=msg)
|
| 21 |
+
t = datetime.datetime.now(zoneinfo.ZoneInfo("America/Toronto")).strftime("%F %H:%M:%S %Z")
|
| 22 |
+
with open(LOG, "a") as f:
|
| 23 |
+
f.write(f"{t}\t{ci.oid}\t{ci.commit_url}\t{msg}\n")
|
| 24 |
+
for rp, s in lst: f.write(f" {rp}\t{s}\n")
|
| 25 |
+
print(t, ci.oid, ci.commit_url); [print(" ", rp, s) for rp, s in lst]
|
| 26 |
+
if __name__ == "__main__": main()
|
v3_scale/code/scale/jobs_v4.tsv
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
old_ref /workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz old_local_v1
|
| 2 |
+
broad14_ref /workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz broad_local_v1
|
| 3 |
+
broad45_covb /workspace/spira_shared/spira_cov/cm_broad_cov_b.npz broad_local_v1
|
| 4 |
+
broad_full /workspace/spira_shared/checker/scale/cm_broad_full.npz broad_local_v1
|
| 5 |
+
broad_cova /workspace/spira_shared/spira_cov/cm_broad_cov_a.npz broad_local_v1
|
| 6 |
+
m8_broad14 /workspace/spira_shared/spira_m8/cm_broad_local_v1.npz broad_local_v1
|
| 7 |
+
v3_covb /workspace/spira_shared/spira_v3verify/cm_v3_cov_b.npz corpus_v3
|
v3_scale/code/scale/registry_v4.tsv
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# tag corpus étiquette
|
| 2 |
+
old_ref old_local_v1 old (réf)
|
| 3 |
+
broad14_ref broad_local_v1 broad 14 % (réf)
|
| 4 |
+
broad45_covb broad_local_v1 broad 45 % (cov b)
|
| 5 |
+
broad_full broad_local_v1 broad FULL (M8, AM 100 %)
|
| 6 |
+
broad_cova broad_local_v1 broad_local_v1 cov a
|
| 7 |
+
m8_broad14 broad_local_v1 M8 broad CM 14 %
|
| 8 |
+
v3_covb corpus_v3 corpus_v3 cov b
|
v3_scale/code/scale/scale_bench_v4.py
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — rapport banc v4 (registry_v4.tsv : tag corpus étiquette). Lit hits_v4/<tag>__<w>.npz + results_v4.jsonl (dernière ligne non-smoke).
|
| 3 |
+
fused_acc_invocab ; moyenne fresh = 6 fenêtres (fc2, fresh_en, fresh_fr, test20k, queries, test_v3) ; écart = seen − fresh ; neither_code rapporté à part
|
| 4 |
+
(neither_code exclu : contaminé 100 % broad / 90 % old). McNemar exact par paires sur fused_hit, toutes fenêtres fixes (seen inclus même si corpus différents : octets différents, à lire comme comparaison de mémorisation).
|
| 5 |
+
Sorties scale_bench_v4.md / .json. Env HITS, OUT, SUFFIX (smoke)."""
|
| 6 |
+
import os, json, itertools, numpy as np
|
| 7 |
+
from math import comb
|
| 8 |
+
D = os.path.dirname(os.path.abspath(__file__)); HITS = os.environ.get("HITS", f"{D}/hits_v4"); OUT = os.environ.get("OUT", f"{D}/results_v4.jsonl"); SUF = os.environ.get("SUFFIX", "")
|
| 9 |
+
WIN = ["seen", "fc2", "fresh_en", "fresh_fr", "neither_code", "test20k", "queries", "test_v3"]; FRESH = ["fc2", "fresh_en", "fresh_fr", "test20k", "queries", "test_v3"] # 6 fenêtres fraîches propres (Relay 17:10)
|
| 10 |
+
def mcn(b, c):
|
| 11 |
+
n = b + c; k = min(b, c)
|
| 12 |
+
return 1.0 if n == 0 else min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2 ** n)
|
| 13 |
+
R = {}
|
| 14 |
+
if os.path.exists(OUT):
|
| 15 |
+
for l in open(OUT):
|
| 16 |
+
r = json.loads(l); R.setdefault(r["tag"], {})[r["window"]] = r
|
| 17 |
+
E = []
|
| 18 |
+
for l in open(f"{D}/registry_v4.tsv"):
|
| 19 |
+
if not l.strip() or l.startswith("#"): continue
|
| 20 |
+
tag, c, lab = l.rstrip("\n").split("\t"); P = {w: f"{HITS}/{tag}__{w}.npz" for w in WIN}
|
| 21 |
+
E.append((lab, tag, c, {w: dict(np.load(p)) for w, p in P.items()} if all(map(os.path.exists, P.values())) else None))
|
| 22 |
+
J = {"models": {}, "pairs": []}
|
| 23 |
+
rows = ["| modèle | corpus | " + " | ".join(WIN) + " | fresh moy. | écart seen−fresh | wall s | pic RAM Mo | scale cal |", "|---|---|" + "---|" * (len(WIN) + 4)]
|
| 24 |
+
for lab, tag, c, h in E:
|
| 25 |
+
if h is None: rows.append(f"| {lab} | {c} | " + " | ".join("…" for _ in WIN) + " | | en attente | | | |"); continue
|
| 26 |
+
a = {w: 100 * float(h[w]["fused_hit"][h[w]["invocab"]].mean()) for w in WIN}; fm = float(np.mean([a[w] for w in FRESH])); rr = R.get(tag, {})
|
| 27 |
+
ws = sum(x["eval_s"] for x in rr.values()) + (next(iter(rr.values()))["load_s"] if rr else 0); pk = max([x["peak_rss_mb"] for x in rr.values()] or [0])
|
| 28 |
+
sc = next(iter(rr.values()))["fusion_cal"]["scale"] if rr else float("nan")
|
| 29 |
+
J["models"][lab] = {"tag": tag, "acc_invocab": a, "fresh_mean": fm, "gap": a["seen"] - fm, "wall_s": ws, "peak_rss_mb": pk, "scale": sc, "scale_ok": 0.03 <= sc <= 0.11}
|
| 30 |
+
rows.append(f"| {lab} | {c} | " + " | ".join(f"{a[w]:.2f}" for w in WIN) + f" | {fm:.2f} | {a['seen'] - fm:.2f} | {ws:.0f} | {pk:.0f} | {sc:.3f}{'' if 0.03 <= sc <= 0.11 else ' HORS'} |")
|
| 31 |
+
pr = ["| A | B | fenêtre | A seul | B seul | p | signe |", "|---|---|---|---|---|---|---|"]
|
| 32 |
+
for (la, ta, ca, ha), (lb, tb, cb, hb) in itertools.combinations([e for e in E if e[3] is not None], 2):
|
| 33 |
+
for w in WIN:
|
| 34 |
+
x, y = ha[w]["fused_hit"], hb[w]["fused_hit"]; n = min(len(x), len(y)); iv = ha[w]["invocab"][:n] | hb[w]["invocab"][:n]; x, y = x[:n], y[:n]
|
| 35 |
+
b, cc = int((x & ~y).sum()), int((~x & y).sum()); p = mcn(b, cc)
|
| 36 |
+
J["pairs"].append({"A": la, "B": lb, "window": w, "n": n, "A_only": b, "B_only": cc, "p": p})
|
| 37 |
+
pr.append(f"| {la} | {lb} | {w} | {b} | {cc} | {p:.2g} | {'B>A' if cc > b else ('A>B' if b > cc else '=')}{' *' if p < 0.05 else ''} |")
|
| 38 |
+
md = "# Banc scale v4 — fused_acc_invocab (%) — fenêtres fixes identiques (seen = corpus propre)\n\n" + "\n".join(rows) + "\n\n## McNemar exact par paires (fused_hit, toutes positions)\n" + "\n".join(pr) + "\n"
|
| 39 |
+
open(f"{D}/scale_bench_v4{SUF}.md", "w").write(md); json.dump(J, open(f"{D}/scale_bench_v4{SUF}.json", "w"), indent=1); print(md)
|
v3_scale/code/scale/train_cm_m8_full.py
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""M8 broad FULL (Checker) — copie de spira_m8/train_cm_m8.py ; m.prepare() remplacé par spira_cov.prepare_cov (tranches stratifiées)
|
| 2 |
+
et m.train() par spira_cov.train_cov. AM = 100 % des patches de train (couverture complète). SNN : budget SNN_TOKENS (défaut = N, 100 %)
|
| 3 |
+
avec EPOCHS (défaut 1). calibrate (save_with_calibration) AVANT save, vérif du scale rechargé.
|
| 4 |
+
Usage : CORPUS LAYOUT OUT.npz env SNN_TOKENS, AM_TOKENS (défaut N), EPOCHS"""
|
| 5 |
+
import sys, json, time, os
|
| 6 |
+
sys.path.insert(0, "/workspace/spira_shared/semhash"); sys.path.insert(0, "/workspace/spira_shared/spira_cov")
|
| 7 |
+
from countmin import CMSPIRA, save_with_calibration, SPIRAConfig
|
| 8 |
+
from resonly import CMResOnly
|
| 9 |
+
from cov_prepare import prepare_cov, train_cov
|
| 10 |
+
corpus, layout, out = sys.argv[1:4]
|
| 11 |
+
Cls = {"resonly": CMResOnly, "concat": CMSPIRA}[layout]
|
| 12 |
+
log = lambda *a: print(time.strftime("%H:%M:%S"), *a, flush=True)
|
| 13 |
+
data = open(f"/workspace/spira_shared/broad/{corpus}/train.bin", "rb").read()
|
| 14 |
+
ep = int(os.environ.get("EPOCHS", "1"))
|
| 15 |
+
cfg = SPIRAConfig(vocab_size=16384, d_in=1024, snn_tokens=600_000, am_tokens=1_200_000, epochs=ep)
|
| 16 |
+
t0 = time.time(); m = Cls.build_cm(data, cfg, log=log)
|
| 17 |
+
N = len(m._ids_train); snn = int(os.environ.get("SNN_TOKENS", N)); am = int(os.environ.get("AM_TOKENS", N))
|
| 18 |
+
cov = prepare_cov(m, arm="a", snn_tokens=snn, am_tokens=am, log=log); m._patches_train = None
|
| 19 |
+
log("cfg", cfg, layout, "N", N, "snn", snn, "am", am, "n_state", m.core.n_state)
|
| 20 |
+
h = train_cov(m, log_fn=log, ckpt_path=None)
|
| 21 |
+
saved = save_with_calibration(m, out, data[:60_000])
|
| 22 |
+
assert Cls.load(out).fusion["scale"] == saved["scale"]
|
| 23 |
+
json.dump({"corpus": corpus, "layout": layout, "history": h, "fusion": saved, "secs": time.time() - t0, "n_state": m.core.n_state,
|
| 24 |
+
"cov": {k: cov[k] for k in ("arm", "N", "seed", "k")}, "snn_tokens": snn, "am_tokens": am, "epochs": ep},
|
| 25 |
+
open(out.replace(".npz", ".summary.json"), "w"), default=str, indent=1)
|
| 26 |
+
log("CALIBRATED+SAVED", saved)
|
v3_scale/code/scale/watch_scale.sh
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# Checker — watcher scale v2 (remplace v1, sauvegardé logs/watch_scale.v1.sh.bak). 1 cœur, nice 10, OMP=1, PYTHONHASHSEED=0.
|
| 3 |
+
# Boucle : pour chaque ligne de jobs_v4.tsv (tag ckpt corpus), quand le .npz existe, est stable ≥120 s et se charge :
|
| 4 |
+
# vérifie fusion.scale ∈ [0,03 ; 0,11] (enregistré), puis eval_bench_v4.py (8 fenêtres fixes) → scale_bench_v4.py, puis gen/gen_eval.py.
|
| 5 |
+
# Un seul job à la fois ; seulement si < 4 lourds Checker ET MemAvailable ≥ 4 Go. (Bloc M8 FULL de v1 conservé, déjà déclenché.)
|
| 6 |
+
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONHASHSEED=0
|
| 7 |
+
D=/workspace/spira_shared/checker/scale; G=/workspace/spira_shared/checker/gen; PY=/workspace/SPIRA-venv/bin/python; cd $D
|
| 8 |
+
log(){ echo "$(TZ=America/Toronto date '+%F %H:%M') $*" >> $D/watch_scale.log; }
|
| 9 |
+
slot(){ [ $(bash $D/heavy_checker.sh) -lt 4 ] && [ $(awk '/MemAvailable/{print int($2/1048576)}' /proc/meminfo) -ge 4 ]; }
|
| 10 |
+
log "démarrage watcher v2 pid $$"
|
| 11 |
+
while true; do
|
| 12 |
+
while IFS=$'\t' read -r tag ck corp; do
|
| 13 |
+
[ -z "$tag" ] && continue; [ -f logs/v4_$tag.done ] && continue
|
| 14 |
+
[ -f "$ck" ] && [ $(( $(date +%s) - $(stat -c %Y "$ck") )) -ge 120 ] || continue
|
| 15 |
+
slot || continue
|
| 16 |
+
sc=$($PY -c "import numpy as np,json,sys;z=np.load(sys.argv[1],allow_pickle=False);print(json.loads(str(z['fusion']))['scale'] if 'fusion' in z.files else 'NA')" "$ck" 2>/dev/null) || continue
|
| 17 |
+
ok=$($PY -c "import sys;s=sys.argv[1];print('OK' if s!='NA' and 0.03<=float(s)<=0.11 else 'HORS_PLAGE')" "$sc" 2>/dev/null)
|
| 18 |
+
log "$tag ckpt prêt, fusion.scale(sauvé)=$sc → $ok ; banc v4"
|
| 19 |
+
nice -n 10 timeout 10800 $PY $D/eval_bench_v4.py "$ck" "$corp" $tag > logs/v4_$tag.log 2>&1; rc=$?; log "$tag banc v4 EXIT $rc"
|
| 20 |
+
$PY $D/scale_bench_v4.py > logs/bench_v4_last.log 2>&1; log "banc v4 mis à jour"
|
| 21 |
+
if [ $rc = 0 ]; then
|
| 22 |
+
[ -f $G/logs/gen_$tag.log ] || (cd $G && nice -n 10 timeout 7200 $PY -u $G/gen_eval.py "$ck" "$corp" $tag > $G/logs/gen_$tag.log 2>&1); log "$tag génération EXIT $?"
|
| 23 |
+
echo "$sc $ok" > logs/v4_$tag.done
|
| 24 |
+
fi
|
| 25 |
+
done < jobs_v4.tsv
|
| 26 |
+
sleep 120
|
| 27 |
+
done
|
v3_scale/code/v3/audit_overlap.py
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — chevauchement 64 o des fenêtres d'éval (définitions eval_ref_v3) avec old/broad train.bin, par position d'éval.
|
| 3 |
+
Position t (cible = patch t+1) « contaminée » si un octet du patch cible est couvert par une fenêtre de 64 o présente dans train.
|
| 4 |
+
Recalcule fused_acc_invocab des réfs (scale/hits) sur positions propres. Usage : audit_overlap.py CORPUS (old_local_v1|broad_local_v1)"""
|
| 5 |
+
import sys, os, glob, json, numpy as np
|
| 6 |
+
sys.path.insert(0, "/workspace/SPIRA/python"); sys.path.insert(0, "/workspace/spira_shared/checker/fc2"); sys.path.insert(0, "/workspace/spira_shared/checker")
|
| 7 |
+
from rollhash import hashes
|
| 8 |
+
from spira.model import SPIRA
|
| 9 |
+
from windows_v2 import freshcode_v2
|
| 10 |
+
from build_corpus_local import strip_gutenberg_header
|
| 11 |
+
c = sys.argv[1]; C = f"/workspace/spira_shared/broad/{c}"; T = "/workspace/repos/toroidal-fractal-intelligence/data"; rd = lambda p: open(p, "rb").read()
|
| 12 |
+
tr = rd(f"{C}/train.bin"); th = np.unique(np.concatenate([hashes(tr[s:s+4_000_063]) for s in range(0, len(tr), 4_000_000)]))
|
| 13 |
+
W = {"fresh_en": (rd(f"{C}/oneshot.bin"), 8000), "fresh_fr": (strip_gutenberg_header(rd(f"{T}/contes_perrault.txt")), 8000),
|
| 14 |
+
"neither_code": (b"\n\n".join(rd(p) for p in sorted(glob.glob("/workspace/kahnn/data/code/*"))), 8000),
|
| 15 |
+
"test20k": (rd(f"{C}/test.bin"), 20000), "queries": (rd(f"{T}/chat_seed_fr_en.txt"), 8000), "freshcode_v2": (freshcode_v2(), 8000)}
|
| 16 |
+
m = SPIRA.load(f"/workspace/spira_shared/checker/ckpt/ref_fixed2cal_{c}.npz")
|
| 17 |
+
D = "/workspace/spira_shared/checker/scale/hits"; out = {}
|
| 18 |
+
for k, (b, mp) in W.items():
|
| 19 |
+
h = hashes(b); bad = np.isin(h, th); cov = np.zeros(len(b) + 1, np.int32)
|
| 20 |
+
for i in np.nonzero(bad)[0]: cov[i] += 1; cov[i + 64] -= 1
|
| 21 |
+
covered = np.cumsum(cov)[:len(b)] > 0
|
| 22 |
+
pats = m.ept.segment(b); off = np.cumsum([0] + [len(p) for p in pats])
|
| 23 |
+
n = min(len(pats) - 1, mp); cont = np.array([covered[off[t+1]:off[t+2]].any() for t in range(n)])
|
| 24 |
+
r = {"windows64": int(len(h)), "windows64_overlap": int(bad.sum()), "bytes_covered": int(covered.sum()), "positions": n, "positions_contaminated": int(cont.sum())}
|
| 25 |
+
hp = f"{D}/ref_fixed2cal_{c}__{c}__{k}.npz"
|
| 26 |
+
if os.path.exists(hp):
|
| 27 |
+
z = np.load(hp); fh, iv = z["fused_hit"][:n], z["invocab"][:n]; cl = ~cont
|
| 28 |
+
r["ref_fused_invocab_all"] = 100 * float(fh[iv].mean()); r["ref_fused_invocab_clean"] = 100 * float(fh[iv & cl].mean()) if (iv & cl).any() else None
|
| 29 |
+
r["ref_fused_invocab_contaminated"] = 100 * float(fh[iv & cont].mean()) if (iv & cont).any() else None
|
| 30 |
+
np.save(f"{os.path.dirname(os.path.abspath(__file__))}/contam_{c}__{k}.npy", cont); out[k] = r; print(k, json.dumps(r), flush=True)
|
| 31 |
+
json.dump(out, open(f"{os.path.dirname(os.path.abspath(__file__))}/audit_overlap_{c}.json", "w"), indent=1)
|
v3_scale/code/v3/build_v3_lowmem.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Checker — corpus_v3 = broad_local_v1/train.bin + livres corpus_gutenberg + livres Project Gutenberg téléchargés (corpus_v3/raw, EN+FR).
|
| 3 |
+
1) en-têtes Gutenberg retirés (strip_gutenberg_header) ; 2) dédup exacte par paragraphe (sha1, \n\n) ; 3) held-out : 2 livres entiers
|
| 4 |
+
(1 EN, 1 FR, choisis par sha256 de l'id) → test.bin ; oneshot.bin = broad_local_v1/oneshot.bin (fresh_en inchangé) ;
|
| 5 |
+
4) suppression dans train de TOUTE fenêtre de 64 o présente dans une fenêtre d'éval (spira_m7/windows/*, freshcode_v2, contes_perrault,
|
| 6 |
+
kahnn code, chat_seed, test old/broad, oneshot old/broad, test/oneshot v3) — hash exact fc2/rollhash, octets couverts coupés, itéré jusqu'à 0.
|
| 7 |
+
Sorties /workspace/spira_shared/corpus_v3/{train,test,oneshot}.bin + corpus_manifest.json (= manifest.json)."""
|
| 8 |
+
import sys, os, glob, json, hashlib, re, numpy as np
|
| 9 |
+
sys.path.insert(0, "/workspace/spira_shared/checker/fc2"); sys.path.insert(0, "/workspace/spira_shared/checker")
|
| 10 |
+
from rollhash import hashes
|
| 11 |
+
from windows_v2 import freshcode_v2
|
| 12 |
+
from build_corpus_local import strip_gutenberg_header
|
| 13 |
+
S = "/workspace/spira_shared"; O = f"{S}/corpus_v3"; B = f"{S}/broad"; T = "/workspace/repos/toroidal-fractal-intelligence/data"
|
| 14 |
+
rd = lambda p: open(p, "rb").read(); log = lambda *a: print(*a, flush=True)
|
| 15 |
+
def lang(b):
|
| 16 |
+
m = re.search(rb"^Language: *(\w+)", b[:5000], re.M); return (m.group(1).decode().lower() if m else "?")
|
| 17 |
+
books = {}
|
| 18 |
+
for p in sorted(glob.glob(f"{O}/raw/pg*.txt")):
|
| 19 |
+
b = rd(p)
|
| 20 |
+
if b"*** START OF" not in b: log("skip (pas un ebook)", p); continue
|
| 21 |
+
l = lang(b)
|
| 22 |
+
if l not in ("english", "french"): log("skip langue", l, p); continue
|
| 23 |
+
books[os.path.basename(p)] = (l, strip_gutenberg_header(b))
|
| 24 |
+
for p in sorted(glob.glob(f"{S}/corpus_gutenberg/corpus/books/*.txt")):
|
| 25 |
+
b = rd(p); books["cg_" + os.path.basename(p)] = ("french" if "_fr" in p else "english", strip_gutenberg_header(b) if b"*** START" in b else b)
|
| 26 |
+
key = lambda n: hashlib.sha256(n.encode()).hexdigest()
|
| 27 |
+
ho = [min((n for n in books if n.startswith("pg") and books[n][0] == L), key=key) for L in ("english", "french")]
|
| 28 |
+
test = b"\n\n".join(books[n][1] for n in ho); oneshot = rd(f"{B}/broad_local_v1/oneshot.bin")
|
| 29 |
+
log("held-out", ho, len(test))
|
| 30 |
+
seen_par, parts, comp, dropped_dup = set(), [], {"broad_local_v1": 0, "english": 0, "french": 0}, 0
|
| 31 |
+
def add(src, b):
|
| 32 |
+
global dropped_dup
|
| 33 |
+
for par in re.split(rb"\n\s*\n", b):
|
| 34 |
+
if len(par.strip()) == 0: continue
|
| 35 |
+
h = hashlib.sha1(par.strip()).digest()
|
| 36 |
+
if h in seen_par: dropped_dup += len(par); continue
|
| 37 |
+
seen_par.add(h); parts.append(par); comp[src] += len(par) + 2
|
| 38 |
+
add("broad_local_v1", rd(f"{B}/broad_local_v1/train.bin"))
|
| 39 |
+
for n in sorted(books):
|
| 40 |
+
if n not in ho: add(books[n][0], books[n][1])
|
| 41 |
+
train = bytearray(b"\n\n".join(parts)); del parts, seen_par; books = {n: (v[0], b"") for n, v in books.items()}; log("train avant filtre", len(train), "doublons paragraphes", dropped_dup)
|
| 42 |
+
W = {os.path.basename(p)[:-4]: rd(p) for p in glob.glob(f"{S}/spira_m7/windows/*.bin") if "seen" not in p}
|
| 43 |
+
W.update({"freshcode_v2": freshcode_v2(), "contes_perrault": strip_gutenberg_header(rd(f"{T}/contes_perrault.txt")),
|
| 44 |
+
"kahnn_code": b"\n\n".join(rd(p) for p in sorted(glob.glob("/workspace/kahnn/data/code/*"))),
|
| 45 |
+
"chat_seed": rd(f"{T}/chat_seed_fr_en.txt"), "test_old": rd(f"{B}/old_local_v1/test.bin"), "test_broad": rd(f"{B}/broad_local_v1/test.bin"),
|
| 46 |
+
"oneshot_old": rd(f"{B}/old_local_v1/oneshot.bin"), "oneshot_broad": oneshot, "test_v3": test})
|
| 47 |
+
E = np.unique(np.concatenate([hashes(b) for b in W.values()])); log("fenêtres d'éval uniques", len(E))
|
| 48 |
+
def bad_mask(buf, CH=4_000_000):
|
| 49 |
+
a = bytes(buf); n = len(a); cut = np.zeros(n, bool); nb = 0
|
| 50 |
+
for s in range(0, max(1, n - 63), CH):
|
| 51 |
+
h = hashes(a[s:s + CH + 63]); hit = np.nonzero(np.isin(h, E))[0]; nb += len(hit)
|
| 52 |
+
for i in hit: cut[s + i:s + i + 64] = True
|
| 53 |
+
return cut, nb
|
| 54 |
+
it, removed = 0, 0
|
| 55 |
+
while True:
|
| 56 |
+
cut, nb = bad_mask(train); log("passe", it, "fenêtres en chevauchement", nb)
|
| 57 |
+
if nb == 0: break
|
| 58 |
+
a = np.frombuffer(bytes(train), np.uint8); keep = ~cut; removed += int(cut.sum())
|
| 59 |
+
# chaque segment coupé est remplacé par un saut de paragraphe
|
| 60 |
+
starts = np.nonzero(cut & ~np.r_[False, cut[:-1]])[0]; out = a[keep]
|
| 61 |
+
ins = np.searchsorted(np.nonzero(keep)[0], starts)
|
| 62 |
+
train = bytearray(np.insert(out, ins, np.uint8(10)).tobytes()); it += 1
|
| 63 |
+
train = bytes(train); del cut
|
| 64 |
+
# vérif finale par fenêtre
|
| 65 |
+
# SPIRA Relay 16 h 27 : vérif finale par blocs (l'ancienne ligne concaténait+np.unique tous les hashes du train -> OOM 6,3 Go)
|
| 66 |
+
HW = {k: np.unique(hashes(b)) for k, b in W.items()}; found = {k: np.zeros(len(v), bool) for k, v in HW.items()}
|
| 67 |
+
for s in range(0, len(train), 4_000_000):
|
| 68 |
+
hc = np.unique(hashes(train[s:s+4_000_063]))
|
| 69 |
+
for k in HW: found[k] |= np.isin(HW[k], hc)
|
| 70 |
+
del hc
|
| 71 |
+
ov = {k: int(found[k].sum()) for k in HW}; del HW, found
|
| 72 |
+
log("chevauchement final", ov); assert all(v == 0 for v in ov.values())
|
| 73 |
+
os.makedirs(O, exist_ok=True)
|
| 74 |
+
for n, b in (("train", train), ("test", test), ("oneshot", oneshot)): open(f"{O}/{n}.bin", "wb").write(b)
|
| 75 |
+
man = {"variant": "corpus_v3", "builder": "checker/v3/build_v3.py", "train_bytes": len(train), "test_bytes": len(test), "oneshot_bytes": len(oneshot),
|
| 76 |
+
"composition_pre_filter": comp, "dup_paragraph_bytes_dropped": dropped_dup, "overlap_bytes_removed": removed, "filter_passes": it,
|
| 77 |
+
"heldout_books": ho, "n_books": len(books), "books": {n: books[n][0] for n in sorted(books)}, "overlap64_final": ov,
|
| 78 |
+
"sha256": {n: hashlib.sha256(rd(f"{O}/{n}.bin")).hexdigest() for n in ("train", "test", "oneshot")},
|
| 79 |
+
"raw_sha256": {os.path.basename(p): hashlib.sha256(rd(p)).hexdigest() for p in sorted(glob.glob(f"{O}/raw/*.txt"))}}
|
| 80 |
+
for f in ("corpus_manifest.json", "manifest.json"): json.dump(man, open(f"{O}/{f}", "w"), indent=1, ensure_ascii=False)
|
| 81 |
+
log("OK", json.dumps({k: man[k] for k in ("train_bytes", "test_bytes", "overlap_bytes_removed", "filter_passes", "sha256")}))
|
v3_scale/code/v3verify/verify_v3.py
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""SPIRA Hashing - verification independante (lecture seule) d'un corpus: tailles, sha256, overlap 64 o.
|
| 2 |
+
Usage: verify_v3.py CORPUS_DIR OUT_JSON"""
|
| 3 |
+
import sys, os, json, hashlib, numpy as np
|
| 4 |
+
sys.path.insert(0, "/workspace/spira_shared/checker")
|
| 5 |
+
C, OUT = sys.argv[1:3]; K = 64; P = np.uint64(1099511628211)
|
| 6 |
+
def hashes(b):
|
| 7 |
+
a = np.frombuffer(b, np.uint8).astype(np.uint64); n = len(a) - K + 1
|
| 8 |
+
if n <= 0: return np.zeros(0, np.uint64)
|
| 9 |
+
h = np.zeros(n, np.uint64)
|
| 10 |
+
with np.errstate(over="ignore"):
|
| 11 |
+
for j in range(K): h = h * P + a[j:j+n] + np.uint64(1)
|
| 12 |
+
return h
|
| 13 |
+
W = {k: open(f"/workspace/spira_shared/spira_m7/windows/{k}.bin","rb").read() for k in ("fresh_en","fresh_fr","neither_code","test20k","queries")}
|
| 14 |
+
try:
|
| 15 |
+
from windows_v2 import freshcode_v2; W["fc2"] = freshcode_v2()
|
| 16 |
+
except Exception as e: W["fc2_ERROR"] = b""; print("fc2 indisponible", e)
|
| 17 |
+
F = {f: open(f"{C}/{f}.bin","rb").read() for f in ("train","test","oneshot") if os.path.exists(f"{C}/{f}.bin")}
|
| 18 |
+
for _k in ("test","oneshot"):
|
| 19 |
+
if _k in F: W["corpus_"+_k] = F[_k]
|
| 20 |
+
rep = {"corpus": C, "files": {f: {"bytes": len(b), "sha256": hashlib.sha256(b).hexdigest()} for f, b in F.items()}, "overlap64": {}}
|
| 21 |
+
ht = hashes(F["train"]); hs = np.sort(ht)
|
| 22 |
+
for k, b in W.items():
|
| 23 |
+
hw = hashes(b); u = np.unique(hw)
|
| 24 |
+
i = np.searchsorted(hs, u); i = np.minimum(i, len(hs)-1); cand = u[hs[i] == u]
|
| 25 |
+
# confirmation octet par octet (anti-collision)
|
| 26 |
+
tset = set(cand.tolist()); pos_t = {}
|
| 27 |
+
for p in np.nonzero(np.isin(ht, cand))[0]: pos_t.setdefault(int(ht[p]), []).append(int(p))
|
| 28 |
+
conf = 0
|
| 29 |
+
for p in np.nonzero(np.isin(hw, cand))[0]:
|
| 30 |
+
seg = b[p:p+K]
|
| 31 |
+
if any(F["train"][q:q+K] == seg for q in pos_t.get(int(hw[p]), ())): conf += 1
|
| 32 |
+
rep["overlap64"][k] = {"window_bytes": len(b), "n_64grams": int(len(hw)), "hash_hits": int(np.isin(hw, cand).sum()), "confirmed_positions": conf}
|
| 33 |
+
print(k, rep["overlap64"][k], flush=True)
|
| 34 |
+
rep["valid_zero_overlap"] = all(v["confirmed_positions"] == 0 for v in rep["overlap64"].values()) and "fc2_ERROR" not in W
|
| 35 |
+
m = f"{C}/corpus_manifest.json" if os.path.exists(f"{C}/corpus_manifest.json") else f"{C}/manifest.json"
|
| 36 |
+
if os.path.exists(m): rep["manifest_checker"] = json.load(open(m))
|
| 37 |
+
json.dump(rep, open(OUT, "w"), indent=1); print("VALID" if rep["valid_zero_overlap"] else "INVALID")
|
v3_scale/corpus_v3/corpus_manifest.json
ADDED
|
@@ -0,0 +1,227 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"variant": "corpus_v3",
|
| 3 |
+
"builder": "checker/v3/build_v3.py",
|
| 4 |
+
"train_bytes": 93284487,
|
| 5 |
+
"test_bytes": 3486653,
|
| 6 |
+
"oneshot_bytes": 119763,
|
| 7 |
+
"composition_pre_filter": {
|
| 8 |
+
"broad_local_v1": 34443642,
|
| 9 |
+
"english": 41663184,
|
| 10 |
+
"french": 18630849
|
| 11 |
+
},
|
| 12 |
+
"dup_paragraph_bytes_dropped": 10514511,
|
| 13 |
+
"overlap_bytes_removed": 1463063,
|
| 14 |
+
"filter_passes": 1,
|
| 15 |
+
"heldout_books": [
|
| 16 |
+
"pg1184.txt",
|
| 17 |
+
"pg8712.txt"
|
| 18 |
+
],
|
| 19 |
+
"n_books": 97,
|
| 20 |
+
"books": {
|
| 21 |
+
"cg_alice.txt": "english",
|
| 22 |
+
"cg_candide_fr.txt": "french",
|
| 23 |
+
"cg_dorian_gray.txt": "english",
|
| 24 |
+
"cg_dracula.txt": "english",
|
| 25 |
+
"cg_frankenstein.txt": "english",
|
| 26 |
+
"cg_great_expectations.txt": "english",
|
| 27 |
+
"cg_les_mis_fr.txt": "french",
|
| 28 |
+
"cg_metamorphosis.txt": "english",
|
| 29 |
+
"cg_moby_dick.txt": "english",
|
| 30 |
+
"cg_pride_prejudice.txt": "english",
|
| 31 |
+
"cg_sherlock.txt": "english",
|
| 32 |
+
"cg_treasure_island.txt": "english",
|
| 33 |
+
"cg_two_cities.txt": "english",
|
| 34 |
+
"pg1080.txt": "english",
|
| 35 |
+
"pg1184.txt": "english",
|
| 36 |
+
"pg120.txt": "english",
|
| 37 |
+
"pg1232.txt": "english",
|
| 38 |
+
"pg1257.txt": "english",
|
| 39 |
+
"pg1260.txt": "english",
|
| 40 |
+
"pg1342.txt": "english",
|
| 41 |
+
"pg135.txt": "english",
|
| 42 |
+
"pg13735.txt": "french",
|
| 43 |
+
"pg13846.txt": "french",
|
| 44 |
+
"pg13951.txt": "french",
|
| 45 |
+
"pg1400.txt": "english",
|
| 46 |
+
"pg14155.txt": "french",
|
| 47 |
+
"pg145.txt": "english",
|
| 48 |
+
"pg1497.txt": "english",
|
| 49 |
+
"pg15200.txt": "english",
|
| 50 |
+
"pg158.txt": "english",
|
| 51 |
+
"pg161.txt": "english",
|
| 52 |
+
"pg16328.txt": "english",
|
| 53 |
+
"pg1661.txt": "english",
|
| 54 |
+
"pg16901.txt": "french",
|
| 55 |
+
"pg1727.txt": "english",
|
| 56 |
+
"pg17489.txt": "french",
|
| 57 |
+
"pg17493.txt": "french",
|
| 58 |
+
"pg17494.txt": "french",
|
| 59 |
+
"pg17808.txt": "french",
|
| 60 |
+
"pg17989.txt": "french",
|
| 61 |
+
"pg17990.txt": "french",
|
| 62 |
+
"pg17991.txt": "french",
|
| 63 |
+
"pg17992.txt": "french",
|
| 64 |
+
"pg18143.txt": "french",
|
| 65 |
+
"pg1952.txt": "english",
|
| 66 |
+
"pg19657.txt": "french",
|
| 67 |
+
"pg205.txt": "english",
|
| 68 |
+
"pg215.txt": "english",
|
| 69 |
+
"pg219.txt": "english",
|
| 70 |
+
"pg23.txt": "english",
|
| 71 |
+
"pg2554.txt": "english",
|
| 72 |
+
"pg2591.txt": "english",
|
| 73 |
+
"pg2600.txt": "english",
|
| 74 |
+
"pg2650.txt": "french",
|
| 75 |
+
"pg26685.txt": "french",
|
| 76 |
+
"pg2701.txt": "english",
|
| 77 |
+
"pg28054.txt": "english",
|
| 78 |
+
"pg2814.txt": "english",
|
| 79 |
+
"pg2998.txt": "french",
|
| 80 |
+
"pg30254.txt": "english",
|
| 81 |
+
"pg3207.txt": "english",
|
| 82 |
+
"pg345.txt": "english",
|
| 83 |
+
"pg35.txt": "english",
|
| 84 |
+
"pg36.txt": "english",
|
| 85 |
+
"pg3600.txt": "english",
|
| 86 |
+
"pg37106.txt": "english",
|
| 87 |
+
"pg408.txt": "english",
|
| 88 |
+
"pg41211.txt": "french",
|
| 89 |
+
"pg42.txt": "english",
|
| 90 |
+
"pg42131.txt": "french",
|
| 91 |
+
"pg43.txt": "english",
|
| 92 |
+
"pg4300.txt": "english",
|
| 93 |
+
"pg4363.txt": "english",
|
| 94 |
+
"pg46.txt": "english",
|
| 95 |
+
"pg4650.txt": "french",
|
| 96 |
+
"pg4791.txt": "french",
|
| 97 |
+
"pg4968.txt": "french",
|
| 98 |
+
"pg5097.txt": "french",
|
| 99 |
+
"pg514.txt": "english",
|
| 100 |
+
"pg51516.txt": "french",
|
| 101 |
+
"pg5200.txt": "english",
|
| 102 |
+
"pg55.txt": "english",
|
| 103 |
+
"pg5711.txt": "french",
|
| 104 |
+
"pg6130.txt": "english",
|
| 105 |
+
"pg64317.txt": "english",
|
| 106 |
+
"pg6838.txt": "french",
|
| 107 |
+
"pg7196.txt": "english",
|
| 108 |
+
"pg74.txt": "english",
|
| 109 |
+
"pg76.txt": "english",
|
| 110 |
+
"pg768.txt": "english",
|
| 111 |
+
"pg799.txt": "french",
|
| 112 |
+
"pg844.txt": "english",
|
| 113 |
+
"pg8712.txt": "french",
|
| 114 |
+
"pg8800.txt": "english",
|
| 115 |
+
"pg9645.txt": "french",
|
| 116 |
+
"pg98.txt": "english",
|
| 117 |
+
"pg996.txt": "english"
|
| 118 |
+
},
|
| 119 |
+
"overlap64_final": {
|
| 120 |
+
"fresh_fr": 0,
|
| 121 |
+
"freshcode": 0,
|
| 122 |
+
"neither_code": 0,
|
| 123 |
+
"test20k": 0,
|
| 124 |
+
"fresh_en": 0,
|
| 125 |
+
"queries": 0,
|
| 126 |
+
"freshcode_v2": 0,
|
| 127 |
+
"contes_perrault": 0,
|
| 128 |
+
"kahnn_code": 0,
|
| 129 |
+
"chat_seed": 0,
|
| 130 |
+
"test_old": 0,
|
| 131 |
+
"test_broad": 0,
|
| 132 |
+
"oneshot_old": 0,
|
| 133 |
+
"oneshot_broad": 0,
|
| 134 |
+
"test_v3": 0
|
| 135 |
+
},
|
| 136 |
+
"sha256": {
|
| 137 |
+
"train": "aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c",
|
| 138 |
+
"test": "a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84",
|
| 139 |
+
"oneshot": "67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a"
|
| 140 |
+
},
|
| 141 |
+
"raw_sha256": {
|
| 142 |
+
"pg1080.txt": "331340f2ada84a7f79d756a344f6ce7d78026cd1d2359d33f79ca4a13d9f1519",
|
| 143 |
+
"pg1184.txt": "64f8d5cfa51fcecb904abf7312d395d512a71817e7359b91288beb50517c3836",
|
| 144 |
+
"pg120.txt": "5bc08275eacea8640b1aa185c645283bb8c564d7b23df6b03437d229b4fc6ebd",
|
| 145 |
+
"pg1232.txt": "33496a305f7a4b933784fb5aec95f920fbc3d4687794817e4cc444aa6578d2e4",
|
| 146 |
+
"pg1257.txt": "1682d060845717283d22462d1d884f79c2741ecdba3cfe5ceed700602ccf0e75",
|
| 147 |
+
"pg1260.txt": "13414dee2951c3ee731d76d2ffd822016b2479c892162760c5d0eb2aa5fa7631",
|
| 148 |
+
"pg1342.txt": "3f6bb9d6f78e0293b56acd4714dd68cb7d6d1d293402031ce9d5a216bcaf9d75",
|
| 149 |
+
"pg135.txt": "6cf3b9d6fc5e6f733737425267d3af296d7f99f9b9fd1d255a3ab7689a6346ab",
|
| 150 |
+
"pg13735.txt": "782a78ce6e52f1cadb894bc871ffd1330861e1a2db4bcf3da0fb325d62c72469",
|
| 151 |
+
"pg13846.txt": "4e3c7f3329d591408dca1ebe0818cd900622dcf6d090e16240bf0bbfac07370c",
|
| 152 |
+
"pg13951.txt": "46c01913219e6ff165256b8cbcaec7034b004f785a147341911b3caee70ccfcc",
|
| 153 |
+
"pg1400.txt": "9a637118af8e953e9764ec603d9b0a032883384d465acac2e27966a80cf1c6f8",
|
| 154 |
+
"pg14155.txt": "93f0a2259dbaf5fde6e7b96adc22346d52f80eb1e9319336c8381a045efe40cd",
|
| 155 |
+
"pg145.txt": "dc2e0107e6ae07e8e33da934b2ec4e8e600826a2e0cd48df265119cd02e5fb50",
|
| 156 |
+
"pg1497.txt": "3d2e397122881d7737a48fb80e71cddf4c78cda54ceac7e0905fe945dfab48fd",
|
| 157 |
+
"pg15200.txt": "7145bcdb997fa09b56acc750031dd7c5fecf26f198475041dac9c179b1890f96",
|
| 158 |
+
"pg158.txt": "532b122b4e6a76cc556d6fdcd729b5892f5c4ce4a1b7060b9f832adfad8680dc",
|
| 159 |
+
"pg161.txt": "7bcd77c45cb32fb8d9f89402561270cc6a911c0a4e20667176d5be779fc3974a",
|
| 160 |
+
"pg16328.txt": "8909085dc48daf123d29c40ecbb3837b16ba7f96bf0cbb9fe72de133a2d35348",
|
| 161 |
+
"pg1661.txt": "922e2a12ccb43a4c9544c260b2166c6ad2097aeb5957faeee113f173bb857cd0",
|
| 162 |
+
"pg16901.txt": "7d264a3faf62bc4e0e37da29d1d2b07c022bb2e0565ee0e6e6b534ff30422290",
|
| 163 |
+
"pg1727.txt": "ffbdb29c3dda284b65c11243db1a98167826a70c81bca2ed4b6232f86c905fb9",
|
| 164 |
+
"pg17489.txt": "a5de514ba7b9f2e1790e7e259c4e8b7a35ae1d29e4bf9a5f8767039c58b80503",
|
| 165 |
+
"pg17493.txt": "ffddbcee83e3c537b5586bfbbaf97ea6fab2b04250b9efaa63c027ab20aaffd0",
|
| 166 |
+
"pg17494.txt": "79894e327a20044dad5b93dd7d1fae0a8c06b74b86cd9f19d37bb54ffbacd44f",
|
| 167 |
+
"pg17808.txt": "401ad78cdcb606765f48f7b478c681d73b49d0c3666a1f6044d1a38564a01c4c",
|
| 168 |
+
"pg17989.txt": "9ea7a8b127a9dfa36de0b0cf9c2e26293beea5c37a0fc988fb9308b63160f282",
|
| 169 |
+
"pg17990.txt": "d3df81d687dfd2ba8e7d219287e767b659c189657c51a3f5433e1ba71f0d0613",
|
| 170 |
+
"pg17991.txt": "49cacb5455e93bebcd744a572447819a9a7f7161fb3516b5fdccbac7a4e2f571",
|
| 171 |
+
"pg17992.txt": "abaf80e7672d909e8adee9cf79ff18b6d79f785ac209cf44d7c06526ed627bd9",
|
| 172 |
+
"pg18143.txt": "dec49c928b81fddc0016e6be2def4045880142a272d299e5fe7acdcd11455fff",
|
| 173 |
+
"pg1952.txt": "20c7788eeae2fe3a0efd51a1f08b6723366a4be0778d65db219149a3522651d4",
|
| 174 |
+
"pg19657.txt": "91e3524d6e7816cda822f97fae3f49af204c7e32ad71ca23002de9e83e935f80",
|
| 175 |
+
"pg205.txt": "2d9a76a2e3e8195c69430516ebd33c4d0757a53ad432ff6186b7b794e6fe99f9",
|
| 176 |
+
"pg215.txt": "30f569c5671bba7bbc4a92ab6d8ee01a0d3be00f994819c77da62c28041eb8a2",
|
| 177 |
+
"pg219.txt": "1bcdf0a1ac611dd6b70a20d100a07085754da4e90a949d6031717b86cd6d020a",
|
| 178 |
+
"pg23.txt": "234c15348a66919bad1d534cdd48ee8ddf91f50115a706cbefaa8edd049672fd",
|
| 179 |
+
"pg2554.txt": "4a84a709a6e355f5fa555c7276c8a90125eb9d06f63bc8663410c3d1297eb688",
|
| 180 |
+
"pg2591.txt": "377ebb8ca06cd8bc0b1015ddb54766c44f50dece731c53bb030f1cc00f54092a",
|
| 181 |
+
"pg2600.txt": "2d5bb2ad5f422765e714617e21fa31bbaf8958aa79682c86fca6660fcc5d1b2b",
|
| 182 |
+
"pg2650.txt": "9ebdb837f733f41d6a822e010d1ccbf14b4ddafad0131482461edb72b55b0936",
|
| 183 |
+
"pg26685.txt": "b838eca71b85dd32b8ef0389d9b8a0a3fec41b85e060a1622f718da418760c46",
|
| 184 |
+
"pg2701.txt": "b5e88b71a1cda068e9ab1d94344c9d932338ba6da1a66def97a1aecd9cfcc87d",
|
| 185 |
+
"pg28054.txt": "2e75c187cb750df3f7c910fa90970b264b9a4fee212be2a73c6bb507d9ad22fe",
|
| 186 |
+
"pg2814.txt": "6be7d8400ac2549459307d1c1a20a25c86a736984269d3e9d989c817814be60c",
|
| 187 |
+
"pg2998.txt": "d23f9d231eed199a7ebab51af574a36dde00662b0310501b874b9f36db1bdd2f",
|
| 188 |
+
"pg30254.txt": "a0084df9930e6b7fb088a845a94fa8ae460aafa0f6a751a0f2d1b32ff3dccd2a",
|
| 189 |
+
"pg3207.txt": "3de1e492641d939567a8b0de827fb13e1ad992324f9ac8b204f60475009b7294",
|
| 190 |
+
"pg345.txt": "96cd16eacdbfebae8fdda5591f66e0cc8ee76be18e0cd1aca02bc00615782d28",
|
| 191 |
+
"pg35.txt": "2892e919000e17c83e1dac51b30f4675db50536b644d7579fe8a89bb399a9bdc",
|
| 192 |
+
"pg36.txt": "8417469e3ab664749f7b36ebb85799075818207b0380b54c27d07d592a44117d",
|
| 193 |
+
"pg3600.txt": "1b4c87312f0890e04cecee48e3a5fa263de65743230b819e16cb1bd72f7aee59",
|
| 194 |
+
"pg37106.txt": "494309c7c4b1a750f19bb69258760a602032ebb0a52955f5153818b3508d5fe0",
|
| 195 |
+
"pg408.txt": "87f8e832484792b761a510b953d67e3835fa46a9f4231a508bfeb914a98559ca",
|
| 196 |
+
"pg41211.txt": "c192ed74d4ffd4000497033b8a512e9f5ca805531281d5395259f954c12e30d2",
|
| 197 |
+
"pg42.txt": "6f5ebfb79a24bb24e88c8f7e5f35515321e545bf4f137fce6e406c9da0bb2b59",
|
| 198 |
+
"pg42131.txt": "bfb7cfb2ae429b4b408029accc0774ebae9875673f2b0130d51b14eb57008f3b",
|
| 199 |
+
"pg43.txt": "b43448a88391591f9cf82b25553df00faf47a2752a873185fcdd1156eebdb990",
|
| 200 |
+
"pg4300.txt": "e03094626f9528cf3fc287a49d6edbdbf47cd40483cb13ceb301e74e15ebbf9e",
|
| 201 |
+
"pg4363.txt": "6ee088f717d3760036086f108c03d57588036c9d344b7d71edbef4e631bd374a",
|
| 202 |
+
"pg46.txt": "8c36b93723d454982df045ec0efdf2664e5a48e6b5e2f670e883575f75ad0cc8",
|
| 203 |
+
"pg4650.txt": "1cc10ab3268a76aa01bd3f9722ae2ae0e5e36b869a18b0bb0832b75bc0327806",
|
| 204 |
+
"pg4791.txt": "cc2c572afc2b2965e252ed8626c4e023b54e272f3a83594f10bc8d7bc6a9f5d6",
|
| 205 |
+
"pg4968.txt": "22bbbb7f1a83c1fe457ef52ea1c00ff95461e15f62c0f5fd70130fbf82ae32e4",
|
| 206 |
+
"pg5097.txt": "53507b025cd3580f4fbfa546baabb36f25adfb48b581c5b5016d91ad69d36c68",
|
| 207 |
+
"pg514.txt": "36831477b339dbcd7e7206119d77e9e58b9727ef3c299d1a1dc4446e1db97850",
|
| 208 |
+
"pg51516.txt": "0c70c8e7e7ce5a9adda520495b074f486af3681c57cc21ba8556dc831aa0df04",
|
| 209 |
+
"pg5200.txt": "0933283a19a4daaef0d412e8f69c74fae126b0a946cb74b8f3c783a8e869830d",
|
| 210 |
+
"pg55.txt": "969bffab7740d4d8a0bac332d78bd0152ad4a40a2efc52f06c74a9bb6120be75",
|
| 211 |
+
"pg5711.txt": "7bbd0d116a119b0626b88d7f8e7e5196831e348395249df6d0ec61abc30e583b",
|
| 212 |
+
"pg6130.txt": "ca4f23115b384b0219427ef208e875f0351820b8f80f28259272997ea581c36b",
|
| 213 |
+
"pg64317.txt": "ce760ec377accd352b41bb8f64504a72d7aa18ab3afb42ded2b56cecacf29e35",
|
| 214 |
+
"pg6838.txt": "c5c049d44e8b44ca64c388396fb0ef270c14f7865554c6fb2cb2102c35f7e77b",
|
| 215 |
+
"pg7196.txt": "71e7f6fa3dd9c0f25008fe7cf176ce1b8d99126be50fdd61da6daf290540a8be",
|
| 216 |
+
"pg74.txt": "74d77384b123a6360db9ab58463cff8b38df8525fbdc6000c81ee388e1f3cf10",
|
| 217 |
+
"pg76.txt": "d617a37aa7ae1e1a93dcde2634db2bccb86824e31e29a55230bdbf77d6872d59",
|
| 218 |
+
"pg768.txt": "e533fe750589f0421d5d744576315f5c2b9b0d69e981179ea0551bbf134c5e02",
|
| 219 |
+
"pg799.txt": "b647a30294f786cc8b502748fd1d1dd17486fadc16ff4d91147b4a7b2402d270",
|
| 220 |
+
"pg844.txt": "1b8a58099bb1cdef6a845277a4bacf2f4a268702c165bde30124d4b5105d1851",
|
| 221 |
+
"pg8712.txt": "fbafaf23e91de6c540ac18e56613a0c8553fcff42842dad1c6ae02438f09d8a0",
|
| 222 |
+
"pg8800.txt": "3a7dd97b5fec82456c58237b33383a593480c2bfe088aeaaa4a519de2a10d39c",
|
| 223 |
+
"pg9645.txt": "1fd16ddf201decd5b376a27dc1576ff12fcfcf835ad344b26100d6d70071324c",
|
| 224 |
+
"pg98.txt": "d54c2b80d40a40b982cd88852c6180bb944d95acdb028af3d0e01a1750681784",
|
| 225 |
+
"pg996.txt": "2864143e9addf498c1c5fd585db4a1ff3e06651c344cddf0931342f1a0ce24d4"
|
| 226 |
+
}
|
| 227 |
+
}
|
v3_scale/corpus_v3/manifest.json
ADDED
|
@@ -0,0 +1,227 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"variant": "corpus_v3",
|
| 3 |
+
"builder": "checker/v3/build_v3.py",
|
| 4 |
+
"train_bytes": 93284487,
|
| 5 |
+
"test_bytes": 3486653,
|
| 6 |
+
"oneshot_bytes": 119763,
|
| 7 |
+
"composition_pre_filter": {
|
| 8 |
+
"broad_local_v1": 34443642,
|
| 9 |
+
"english": 41663184,
|
| 10 |
+
"french": 18630849
|
| 11 |
+
},
|
| 12 |
+
"dup_paragraph_bytes_dropped": 10514511,
|
| 13 |
+
"overlap_bytes_removed": 1463063,
|
| 14 |
+
"filter_passes": 1,
|
| 15 |
+
"heldout_books": [
|
| 16 |
+
"pg1184.txt",
|
| 17 |
+
"pg8712.txt"
|
| 18 |
+
],
|
| 19 |
+
"n_books": 97,
|
| 20 |
+
"books": {
|
| 21 |
+
"cg_alice.txt": "english",
|
| 22 |
+
"cg_candide_fr.txt": "french",
|
| 23 |
+
"cg_dorian_gray.txt": "english",
|
| 24 |
+
"cg_dracula.txt": "english",
|
| 25 |
+
"cg_frankenstein.txt": "english",
|
| 26 |
+
"cg_great_expectations.txt": "english",
|
| 27 |
+
"cg_les_mis_fr.txt": "french",
|
| 28 |
+
"cg_metamorphosis.txt": "english",
|
| 29 |
+
"cg_moby_dick.txt": "english",
|
| 30 |
+
"cg_pride_prejudice.txt": "english",
|
| 31 |
+
"cg_sherlock.txt": "english",
|
| 32 |
+
"cg_treasure_island.txt": "english",
|
| 33 |
+
"cg_two_cities.txt": "english",
|
| 34 |
+
"pg1080.txt": "english",
|
| 35 |
+
"pg1184.txt": "english",
|
| 36 |
+
"pg120.txt": "english",
|
| 37 |
+
"pg1232.txt": "english",
|
| 38 |
+
"pg1257.txt": "english",
|
| 39 |
+
"pg1260.txt": "english",
|
| 40 |
+
"pg1342.txt": "english",
|
| 41 |
+
"pg135.txt": "english",
|
| 42 |
+
"pg13735.txt": "french",
|
| 43 |
+
"pg13846.txt": "french",
|
| 44 |
+
"pg13951.txt": "french",
|
| 45 |
+
"pg1400.txt": "english",
|
| 46 |
+
"pg14155.txt": "french",
|
| 47 |
+
"pg145.txt": "english",
|
| 48 |
+
"pg1497.txt": "english",
|
| 49 |
+
"pg15200.txt": "english",
|
| 50 |
+
"pg158.txt": "english",
|
| 51 |
+
"pg161.txt": "english",
|
| 52 |
+
"pg16328.txt": "english",
|
| 53 |
+
"pg1661.txt": "english",
|
| 54 |
+
"pg16901.txt": "french",
|
| 55 |
+
"pg1727.txt": "english",
|
| 56 |
+
"pg17489.txt": "french",
|
| 57 |
+
"pg17493.txt": "french",
|
| 58 |
+
"pg17494.txt": "french",
|
| 59 |
+
"pg17808.txt": "french",
|
| 60 |
+
"pg17989.txt": "french",
|
| 61 |
+
"pg17990.txt": "french",
|
| 62 |
+
"pg17991.txt": "french",
|
| 63 |
+
"pg17992.txt": "french",
|
| 64 |
+
"pg18143.txt": "french",
|
| 65 |
+
"pg1952.txt": "english",
|
| 66 |
+
"pg19657.txt": "french",
|
| 67 |
+
"pg205.txt": "english",
|
| 68 |
+
"pg215.txt": "english",
|
| 69 |
+
"pg219.txt": "english",
|
| 70 |
+
"pg23.txt": "english",
|
| 71 |
+
"pg2554.txt": "english",
|
| 72 |
+
"pg2591.txt": "english",
|
| 73 |
+
"pg2600.txt": "english",
|
| 74 |
+
"pg2650.txt": "french",
|
| 75 |
+
"pg26685.txt": "french",
|
| 76 |
+
"pg2701.txt": "english",
|
| 77 |
+
"pg28054.txt": "english",
|
| 78 |
+
"pg2814.txt": "english",
|
| 79 |
+
"pg2998.txt": "french",
|
| 80 |
+
"pg30254.txt": "english",
|
| 81 |
+
"pg3207.txt": "english",
|
| 82 |
+
"pg345.txt": "english",
|
| 83 |
+
"pg35.txt": "english",
|
| 84 |
+
"pg36.txt": "english",
|
| 85 |
+
"pg3600.txt": "english",
|
| 86 |
+
"pg37106.txt": "english",
|
| 87 |
+
"pg408.txt": "english",
|
| 88 |
+
"pg41211.txt": "french",
|
| 89 |
+
"pg42.txt": "english",
|
| 90 |
+
"pg42131.txt": "french",
|
| 91 |
+
"pg43.txt": "english",
|
| 92 |
+
"pg4300.txt": "english",
|
| 93 |
+
"pg4363.txt": "english",
|
| 94 |
+
"pg46.txt": "english",
|
| 95 |
+
"pg4650.txt": "french",
|
| 96 |
+
"pg4791.txt": "french",
|
| 97 |
+
"pg4968.txt": "french",
|
| 98 |
+
"pg5097.txt": "french",
|
| 99 |
+
"pg514.txt": "english",
|
| 100 |
+
"pg51516.txt": "french",
|
| 101 |
+
"pg5200.txt": "english",
|
| 102 |
+
"pg55.txt": "english",
|
| 103 |
+
"pg5711.txt": "french",
|
| 104 |
+
"pg6130.txt": "english",
|
| 105 |
+
"pg64317.txt": "english",
|
| 106 |
+
"pg6838.txt": "french",
|
| 107 |
+
"pg7196.txt": "english",
|
| 108 |
+
"pg74.txt": "english",
|
| 109 |
+
"pg76.txt": "english",
|
| 110 |
+
"pg768.txt": "english",
|
| 111 |
+
"pg799.txt": "french",
|
| 112 |
+
"pg844.txt": "english",
|
| 113 |
+
"pg8712.txt": "french",
|
| 114 |
+
"pg8800.txt": "english",
|
| 115 |
+
"pg9645.txt": "french",
|
| 116 |
+
"pg98.txt": "english",
|
| 117 |
+
"pg996.txt": "english"
|
| 118 |
+
},
|
| 119 |
+
"overlap64_final": {
|
| 120 |
+
"fresh_fr": 0,
|
| 121 |
+
"freshcode": 0,
|
| 122 |
+
"neither_code": 0,
|
| 123 |
+
"test20k": 0,
|
| 124 |
+
"fresh_en": 0,
|
| 125 |
+
"queries": 0,
|
| 126 |
+
"freshcode_v2": 0,
|
| 127 |
+
"contes_perrault": 0,
|
| 128 |
+
"kahnn_code": 0,
|
| 129 |
+
"chat_seed": 0,
|
| 130 |
+
"test_old": 0,
|
| 131 |
+
"test_broad": 0,
|
| 132 |
+
"oneshot_old": 0,
|
| 133 |
+
"oneshot_broad": 0,
|
| 134 |
+
"test_v3": 0
|
| 135 |
+
},
|
| 136 |
+
"sha256": {
|
| 137 |
+
"train": "aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c",
|
| 138 |
+
"test": "a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84",
|
| 139 |
+
"oneshot": "67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a"
|
| 140 |
+
},
|
| 141 |
+
"raw_sha256": {
|
| 142 |
+
"pg1080.txt": "331340f2ada84a7f79d756a344f6ce7d78026cd1d2359d33f79ca4a13d9f1519",
|
| 143 |
+
"pg1184.txt": "64f8d5cfa51fcecb904abf7312d395d512a71817e7359b91288beb50517c3836",
|
| 144 |
+
"pg120.txt": "5bc08275eacea8640b1aa185c645283bb8c564d7b23df6b03437d229b4fc6ebd",
|
| 145 |
+
"pg1232.txt": "33496a305f7a4b933784fb5aec95f920fbc3d4687794817e4cc444aa6578d2e4",
|
| 146 |
+
"pg1257.txt": "1682d060845717283d22462d1d884f79c2741ecdba3cfe5ceed700602ccf0e75",
|
| 147 |
+
"pg1260.txt": "13414dee2951c3ee731d76d2ffd822016b2479c892162760c5d0eb2aa5fa7631",
|
| 148 |
+
"pg1342.txt": "3f6bb9d6f78e0293b56acd4714dd68cb7d6d1d293402031ce9d5a216bcaf9d75",
|
| 149 |
+
"pg135.txt": "6cf3b9d6fc5e6f733737425267d3af296d7f99f9b9fd1d255a3ab7689a6346ab",
|
| 150 |
+
"pg13735.txt": "782a78ce6e52f1cadb894bc871ffd1330861e1a2db4bcf3da0fb325d62c72469",
|
| 151 |
+
"pg13846.txt": "4e3c7f3329d591408dca1ebe0818cd900622dcf6d090e16240bf0bbfac07370c",
|
| 152 |
+
"pg13951.txt": "46c01913219e6ff165256b8cbcaec7034b004f785a147341911b3caee70ccfcc",
|
| 153 |
+
"pg1400.txt": "9a637118af8e953e9764ec603d9b0a032883384d465acac2e27966a80cf1c6f8",
|
| 154 |
+
"pg14155.txt": "93f0a2259dbaf5fde6e7b96adc22346d52f80eb1e9319336c8381a045efe40cd",
|
| 155 |
+
"pg145.txt": "dc2e0107e6ae07e8e33da934b2ec4e8e600826a2e0cd48df265119cd02e5fb50",
|
| 156 |
+
"pg1497.txt": "3d2e397122881d7737a48fb80e71cddf4c78cda54ceac7e0905fe945dfab48fd",
|
| 157 |
+
"pg15200.txt": "7145bcdb997fa09b56acc750031dd7c5fecf26f198475041dac9c179b1890f96",
|
| 158 |
+
"pg158.txt": "532b122b4e6a76cc556d6fdcd729b5892f5c4ce4a1b7060b9f832adfad8680dc",
|
| 159 |
+
"pg161.txt": "7bcd77c45cb32fb8d9f89402561270cc6a911c0a4e20667176d5be779fc3974a",
|
| 160 |
+
"pg16328.txt": "8909085dc48daf123d29c40ecbb3837b16ba7f96bf0cbb9fe72de133a2d35348",
|
| 161 |
+
"pg1661.txt": "922e2a12ccb43a4c9544c260b2166c6ad2097aeb5957faeee113f173bb857cd0",
|
| 162 |
+
"pg16901.txt": "7d264a3faf62bc4e0e37da29d1d2b07c022bb2e0565ee0e6e6b534ff30422290",
|
| 163 |
+
"pg1727.txt": "ffbdb29c3dda284b65c11243db1a98167826a70c81bca2ed4b6232f86c905fb9",
|
| 164 |
+
"pg17489.txt": "a5de514ba7b9f2e1790e7e259c4e8b7a35ae1d29e4bf9a5f8767039c58b80503",
|
| 165 |
+
"pg17493.txt": "ffddbcee83e3c537b5586bfbbaf97ea6fab2b04250b9efaa63c027ab20aaffd0",
|
| 166 |
+
"pg17494.txt": "79894e327a20044dad5b93dd7d1fae0a8c06b74b86cd9f19d37bb54ffbacd44f",
|
| 167 |
+
"pg17808.txt": "401ad78cdcb606765f48f7b478c681d73b49d0c3666a1f6044d1a38564a01c4c",
|
| 168 |
+
"pg17989.txt": "9ea7a8b127a9dfa36de0b0cf9c2e26293beea5c37a0fc988fb9308b63160f282",
|
| 169 |
+
"pg17990.txt": "d3df81d687dfd2ba8e7d219287e767b659c189657c51a3f5433e1ba71f0d0613",
|
| 170 |
+
"pg17991.txt": "49cacb5455e93bebcd744a572447819a9a7f7161fb3516b5fdccbac7a4e2f571",
|
| 171 |
+
"pg17992.txt": "abaf80e7672d909e8adee9cf79ff18b6d79f785ac209cf44d7c06526ed627bd9",
|
| 172 |
+
"pg18143.txt": "dec49c928b81fddc0016e6be2def4045880142a272d299e5fe7acdcd11455fff",
|
| 173 |
+
"pg1952.txt": "20c7788eeae2fe3a0efd51a1f08b6723366a4be0778d65db219149a3522651d4",
|
| 174 |
+
"pg19657.txt": "91e3524d6e7816cda822f97fae3f49af204c7e32ad71ca23002de9e83e935f80",
|
| 175 |
+
"pg205.txt": "2d9a76a2e3e8195c69430516ebd33c4d0757a53ad432ff6186b7b794e6fe99f9",
|
| 176 |
+
"pg215.txt": "30f569c5671bba7bbc4a92ab6d8ee01a0d3be00f994819c77da62c28041eb8a2",
|
| 177 |
+
"pg219.txt": "1bcdf0a1ac611dd6b70a20d100a07085754da4e90a949d6031717b86cd6d020a",
|
| 178 |
+
"pg23.txt": "234c15348a66919bad1d534cdd48ee8ddf91f50115a706cbefaa8edd049672fd",
|
| 179 |
+
"pg2554.txt": "4a84a709a6e355f5fa555c7276c8a90125eb9d06f63bc8663410c3d1297eb688",
|
| 180 |
+
"pg2591.txt": "377ebb8ca06cd8bc0b1015ddb54766c44f50dece731c53bb030f1cc00f54092a",
|
| 181 |
+
"pg2600.txt": "2d5bb2ad5f422765e714617e21fa31bbaf8958aa79682c86fca6660fcc5d1b2b",
|
| 182 |
+
"pg2650.txt": "9ebdb837f733f41d6a822e010d1ccbf14b4ddafad0131482461edb72b55b0936",
|
| 183 |
+
"pg26685.txt": "b838eca71b85dd32b8ef0389d9b8a0a3fec41b85e060a1622f718da418760c46",
|
| 184 |
+
"pg2701.txt": "b5e88b71a1cda068e9ab1d94344c9d932338ba6da1a66def97a1aecd9cfcc87d",
|
| 185 |
+
"pg28054.txt": "2e75c187cb750df3f7c910fa90970b264b9a4fee212be2a73c6bb507d9ad22fe",
|
| 186 |
+
"pg2814.txt": "6be7d8400ac2549459307d1c1a20a25c86a736984269d3e9d989c817814be60c",
|
| 187 |
+
"pg2998.txt": "d23f9d231eed199a7ebab51af574a36dde00662b0310501b874b9f36db1bdd2f",
|
| 188 |
+
"pg30254.txt": "a0084df9930e6b7fb088a845a94fa8ae460aafa0f6a751a0f2d1b32ff3dccd2a",
|
| 189 |
+
"pg3207.txt": "3de1e492641d939567a8b0de827fb13e1ad992324f9ac8b204f60475009b7294",
|
| 190 |
+
"pg345.txt": "96cd16eacdbfebae8fdda5591f66e0cc8ee76be18e0cd1aca02bc00615782d28",
|
| 191 |
+
"pg35.txt": "2892e919000e17c83e1dac51b30f4675db50536b644d7579fe8a89bb399a9bdc",
|
| 192 |
+
"pg36.txt": "8417469e3ab664749f7b36ebb85799075818207b0380b54c27d07d592a44117d",
|
| 193 |
+
"pg3600.txt": "1b4c87312f0890e04cecee48e3a5fa263de65743230b819e16cb1bd72f7aee59",
|
| 194 |
+
"pg37106.txt": "494309c7c4b1a750f19bb69258760a602032ebb0a52955f5153818b3508d5fe0",
|
| 195 |
+
"pg408.txt": "87f8e832484792b761a510b953d67e3835fa46a9f4231a508bfeb914a98559ca",
|
| 196 |
+
"pg41211.txt": "c192ed74d4ffd4000497033b8a512e9f5ca805531281d5395259f954c12e30d2",
|
| 197 |
+
"pg42.txt": "6f5ebfb79a24bb24e88c8f7e5f35515321e545bf4f137fce6e406c9da0bb2b59",
|
| 198 |
+
"pg42131.txt": "bfb7cfb2ae429b4b408029accc0774ebae9875673f2b0130d51b14eb57008f3b",
|
| 199 |
+
"pg43.txt": "b43448a88391591f9cf82b25553df00faf47a2752a873185fcdd1156eebdb990",
|
| 200 |
+
"pg4300.txt": "e03094626f9528cf3fc287a49d6edbdbf47cd40483cb13ceb301e74e15ebbf9e",
|
| 201 |
+
"pg4363.txt": "6ee088f717d3760036086f108c03d57588036c9d344b7d71edbef4e631bd374a",
|
| 202 |
+
"pg46.txt": "8c36b93723d454982df045ec0efdf2664e5a48e6b5e2f670e883575f75ad0cc8",
|
| 203 |
+
"pg4650.txt": "1cc10ab3268a76aa01bd3f9722ae2ae0e5e36b869a18b0bb0832b75bc0327806",
|
| 204 |
+
"pg4791.txt": "cc2c572afc2b2965e252ed8626c4e023b54e272f3a83594f10bc8d7bc6a9f5d6",
|
| 205 |
+
"pg4968.txt": "22bbbb7f1a83c1fe457ef52ea1c00ff95461e15f62c0f5fd70130fbf82ae32e4",
|
| 206 |
+
"pg5097.txt": "53507b025cd3580f4fbfa546baabb36f25adfb48b581c5b5016d91ad69d36c68",
|
| 207 |
+
"pg514.txt": "36831477b339dbcd7e7206119d77e9e58b9727ef3c299d1a1dc4446e1db97850",
|
| 208 |
+
"pg51516.txt": "0c70c8e7e7ce5a9adda520495b074f486af3681c57cc21ba8556dc831aa0df04",
|
| 209 |
+
"pg5200.txt": "0933283a19a4daaef0d412e8f69c74fae126b0a946cb74b8f3c783a8e869830d",
|
| 210 |
+
"pg55.txt": "969bffab7740d4d8a0bac332d78bd0152ad4a40a2efc52f06c74a9bb6120be75",
|
| 211 |
+
"pg5711.txt": "7bbd0d116a119b0626b88d7f8e7e5196831e348395249df6d0ec61abc30e583b",
|
| 212 |
+
"pg6130.txt": "ca4f23115b384b0219427ef208e875f0351820b8f80f28259272997ea581c36b",
|
| 213 |
+
"pg64317.txt": "ce760ec377accd352b41bb8f64504a72d7aa18ab3afb42ded2b56cecacf29e35",
|
| 214 |
+
"pg6838.txt": "c5c049d44e8b44ca64c388396fb0ef270c14f7865554c6fb2cb2102c35f7e77b",
|
| 215 |
+
"pg7196.txt": "71e7f6fa3dd9c0f25008fe7cf176ce1b8d99126be50fdd61da6daf290540a8be",
|
| 216 |
+
"pg74.txt": "74d77384b123a6360db9ab58463cff8b38df8525fbdc6000c81ee388e1f3cf10",
|
| 217 |
+
"pg76.txt": "d617a37aa7ae1e1a93dcde2634db2bccb86824e31e29a55230bdbf77d6872d59",
|
| 218 |
+
"pg768.txt": "e533fe750589f0421d5d744576315f5c2b9b0d69e981179ea0551bbf134c5e02",
|
| 219 |
+
"pg799.txt": "b647a30294f786cc8b502748fd1d1dd17486fadc16ff4d91147b4a7b2402d270",
|
| 220 |
+
"pg844.txt": "1b8a58099bb1cdef6a845277a4bacf2f4a268702c165bde30124d4b5105d1851",
|
| 221 |
+
"pg8712.txt": "fbafaf23e91de6c540ac18e56613a0c8553fcff42842dad1c6ae02438f09d8a0",
|
| 222 |
+
"pg8800.txt": "3a7dd97b5fec82456c58237b33383a593480c2bfe088aeaaa4a519de2a10d39c",
|
| 223 |
+
"pg9645.txt": "1fd16ddf201decd5b376a27dc1576ff12fcfcf835ad344b26100d6d70071324c",
|
| 224 |
+
"pg98.txt": "d54c2b80d40a40b982cd88852c6180bb944d95acdb028af3d0e01a1750681784",
|
| 225 |
+
"pg996.txt": "2864143e9addf498c1c5fd585db4a1ff3e06651c344cddf0931342f1a0ce24d4"
|
| 226 |
+
}
|
| 227 |
+
}
|
v3_scale/corpus_v3/oneshot.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a
|
| 3 |
+
size 119763
|
v3_scale/corpus_v3/test.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84
|
| 3 |
+
size 3486653
|
v3_scale/corpus_v3/train.bin
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c
|
| 3 |
+
size 93284487
|
v3_scale/reports/audit_overlap_broad_local_v1.json
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"fresh_en": {
|
| 3 |
+
"windows64": 119700,
|
| 4 |
+
"windows64_overlap": 0,
|
| 5 |
+
"bytes_covered": 0,
|
| 6 |
+
"positions": 8000,
|
| 7 |
+
"positions_contaminated": 0,
|
| 8 |
+
"ref_fused_invocab_all": 5.166846071044134,
|
| 9 |
+
"ref_fused_invocab_clean": 5.166846071044134,
|
| 10 |
+
"ref_fused_invocab_contaminated": null
|
| 11 |
+
},
|
| 12 |
+
"fresh_fr": {
|
| 13 |
+
"windows64": 1088625,
|
| 14 |
+
"windows64_overlap": 0,
|
| 15 |
+
"bytes_covered": 0,
|
| 16 |
+
"positions": 8000,
|
| 17 |
+
"positions_contaminated": 0,
|
| 18 |
+
"ref_fused_invocab_all": 2.716933445661331,
|
| 19 |
+
"ref_fused_invocab_clean": 2.716933445661331,
|
| 20 |
+
"ref_fused_invocab_contaminated": null
|
| 21 |
+
},
|
| 22 |
+
"neither_code": {
|
| 23 |
+
"windows64": 545973,
|
| 24 |
+
"windows64_overlap": 545558,
|
| 25 |
+
"bytes_covered": 546036,
|
| 26 |
+
"positions": 8000,
|
| 27 |
+
"positions_contaminated": 8000,
|
| 28 |
+
"ref_fused_invocab_all": 7.669746297111013,
|
| 29 |
+
"ref_fused_invocab_clean": null,
|
| 30 |
+
"ref_fused_invocab_contaminated": 7.669746297111013
|
| 31 |
+
},
|
| 32 |
+
"test20k": {
|
| 33 |
+
"windows64": 894655,
|
| 34 |
+
"windows64_overlap": 33766,
|
| 35 |
+
"bytes_covered": 45852,
|
| 36 |
+
"positions": 20000,
|
| 37 |
+
"positions_contaminated": 136,
|
| 38 |
+
"ref_fused_invocab_all": 3.2328136979299766,
|
| 39 |
+
"ref_fused_invocab_clean": 3.2575806347775704,
|
| 40 |
+
"ref_fused_invocab_contaminated": 0.0
|
| 41 |
+
},
|
| 42 |
+
"queries": {
|
| 43 |
+
"windows64": 27737,
|
| 44 |
+
"windows64_overlap": 0,
|
| 45 |
+
"bytes_covered": 0,
|
| 46 |
+
"positions": 7750,
|
| 47 |
+
"positions_contaminated": 0,
|
| 48 |
+
"ref_fused_invocab_all": 8.02061855670103,
|
| 49 |
+
"ref_fused_invocab_clean": 8.02061855670103,
|
| 50 |
+
"ref_fused_invocab_contaminated": null
|
| 51 |
+
},
|
| 52 |
+
"freshcode_v2": {
|
| 53 |
+
"windows64": 199729,
|
| 54 |
+
"windows64_overlap": 0,
|
| 55 |
+
"bytes_covered": 0,
|
| 56 |
+
"positions": 8000,
|
| 57 |
+
"positions_contaminated": 0,
|
| 58 |
+
"ref_fused_invocab_all": 8.456316410861865,
|
| 59 |
+
"ref_fused_invocab_clean": 8.456316410861865,
|
| 60 |
+
"ref_fused_invocab_contaminated": null
|
| 61 |
+
}
|
| 62 |
+
}
|
v3_scale/reports/audit_overlap_old_local_v1.json
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"fresh_en": {
|
| 3 |
+
"windows64": 119700,
|
| 4 |
+
"windows64_overlap": 0,
|
| 5 |
+
"bytes_covered": 0,
|
| 6 |
+
"positions": 8000,
|
| 7 |
+
"positions_contaminated": 0,
|
| 8 |
+
"ref_fused_invocab_all": 3.3951697584879246,
|
| 9 |
+
"ref_fused_invocab_clean": 3.3951697584879246,
|
| 10 |
+
"ref_fused_invocab_contaminated": null
|
| 11 |
+
},
|
| 12 |
+
"fresh_fr": {
|
| 13 |
+
"windows64": 1088625,
|
| 14 |
+
"windows64_overlap": 0,
|
| 15 |
+
"bytes_covered": 0,
|
| 16 |
+
"positions": 8000,
|
| 17 |
+
"positions_contaminated": 0,
|
| 18 |
+
"ref_fused_invocab_all": 0.28310789556464294,
|
| 19 |
+
"ref_fused_invocab_clean": 0.28310789556464294,
|
| 20 |
+
"ref_fused_invocab_contaminated": null
|
| 21 |
+
},
|
| 22 |
+
"neither_code": {
|
| 23 |
+
"windows64": 545973,
|
| 24 |
+
"windows64_overlap": 224303,
|
| 25 |
+
"bytes_covered": 241788,
|
| 26 |
+
"positions": 8000,
|
| 27 |
+
"positions_contaminated": 7162,
|
| 28 |
+
"ref_fused_invocab_all": 6.951274027715691,
|
| 29 |
+
"ref_fused_invocab_clean": 8.47107438016529,
|
| 30 |
+
"ref_fused_invocab_contaminated": 6.7669172932330826
|
| 31 |
+
},
|
| 32 |
+
"test20k": {
|
| 33 |
+
"windows64": 319877,
|
| 34 |
+
"windows64_overlap": 1396,
|
| 35 |
+
"bytes_covered": 4254,
|
| 36 |
+
"positions": 20000,
|
| 37 |
+
"positions_contaminated": 0,
|
| 38 |
+
"ref_fused_invocab_all": 3.238982443568613,
|
| 39 |
+
"ref_fused_invocab_clean": 3.238982443568613,
|
| 40 |
+
"ref_fused_invocab_contaminated": null
|
| 41 |
+
},
|
| 42 |
+
"queries": {
|
| 43 |
+
"windows64": 27737,
|
| 44 |
+
"windows64_overlap": 0,
|
| 45 |
+
"bytes_covered": 0,
|
| 46 |
+
"positions": 7099,
|
| 47 |
+
"positions_contaminated": 0,
|
| 48 |
+
"ref_fused_invocab_all": 0.0,
|
| 49 |
+
"ref_fused_invocab_clean": 0.0,
|
| 50 |
+
"ref_fused_invocab_contaminated": null
|
| 51 |
+
},
|
| 52 |
+
"freshcode_v2": {
|
| 53 |
+
"windows64": 199729,
|
| 54 |
+
"windows64_overlap": 0,
|
| 55 |
+
"bytes_covered": 0,
|
| 56 |
+
"positions": 8000,
|
| 57 |
+
"positions_contaminated": 0,
|
| 58 |
+
"ref_fused_invocab_all": 11.504241671839436,
|
| 59 |
+
"ref_fused_invocab_clean": 11.504241671839436,
|
| 60 |
+
"ref_fused_invocab_contaminated": null
|
| 61 |
+
}
|
| 62 |
+
}
|
v3_scale/reports/corpus_v3.sha256
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
aabd912d1bce05d84122039e4f7e059f7bbb6bd6b448367b91bbf1e9fa75249c corpus_v3/train.bin
|
| 2 |
+
a6730bbe877e7b28a5529793b27a0b59f16d04db05ea895f0ee6d4f91807ca84 corpus_v3/test.bin
|
| 3 |
+
67ff20b26ce5e3401cea57c817b1a8417e95f6cd131fd8fb631cd5cd8625331a corpus_v3/oneshot.bin
|
v3_scale/reports/recheck_bench_v4_oldref_1722.json
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 5868,
|
| 5 |
+
"fused": 5770,
|
| 6 |
+
"snn": 1937,
|
| 7 |
+
"fused_pct": 98.33,
|
| 8 |
+
"snn_pct": 33.01,
|
| 9 |
+
"cov": 0.7335,
|
| 10 |
+
"reported": [
|
| 11 |
+
98.33,
|
| 12 |
+
33.01,
|
| 13 |
+
0.7335
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 4833,
|
| 19 |
+
"fused": 556,
|
| 20 |
+
"snn": 403,
|
| 21 |
+
"fused_pct": 11.504,
|
| 22 |
+
"snn_pct": 8.339,
|
| 23 |
+
"cov": 0.604125,
|
| 24 |
+
"reported": [
|
| 25 |
+
11.504,
|
| 26 |
+
8.339,
|
| 27 |
+
0.604125
|
| 28 |
+
]
|
| 29 |
+
}
|
| 30 |
+
}
|
v3_scale/reports/recheck_corpus_v3_1700.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"sizes": {
|
| 3 |
+
"train.bin": 93284487,
|
| 4 |
+
"test.bin": 3486653,
|
| 5 |
+
"oneshot.bin": 119763
|
| 6 |
+
},
|
| 7 |
+
"sha_ok": {
|
| 8 |
+
"train": true,
|
| 9 |
+
"test": true,
|
| 10 |
+
"oneshot": true
|
| 11 |
+
},
|
| 12 |
+
"overlap64_final": {
|
| 13 |
+
"fresh_fr": 0,
|
| 14 |
+
"freshcode": 0,
|
| 15 |
+
"neither_code": 0,
|
| 16 |
+
"test20k": 0,
|
| 17 |
+
"fresh_en": 0,
|
| 18 |
+
"queries": 0,
|
| 19 |
+
"freshcode_v2": 0,
|
| 20 |
+
"contes_perrault": 0,
|
| 21 |
+
"kahnn_code": 0,
|
| 22 |
+
"chat_seed": 0,
|
| 23 |
+
"test_old": 0,
|
| 24 |
+
"test_broad": 0,
|
| 25 |
+
"oneshot_old": 0,
|
| 26 |
+
"oneshot_broad": 0,
|
| 27 |
+
"test_v3": 0
|
| 28 |
+
},
|
| 29 |
+
"heldout": [
|
| 30 |
+
"pg1184.txt",
|
| 31 |
+
"pg8712.txt"
|
| 32 |
+
],
|
| 33 |
+
"n_books": 97,
|
| 34 |
+
"comp": {
|
| 35 |
+
"broad_local_v1": 34443642,
|
| 36 |
+
"english": 41663184,
|
| 37 |
+
"french": 18630849
|
| 38 |
+
},
|
| 39 |
+
"manifests_identical": true
|
| 40 |
+
}
|
v3_scale/reports/recheck_scale_smoke_1706.json
ADDED
|
@@ -0,0 +1,198 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"acc": {
|
| 3 |
+
"old_ref": {
|
| 4 |
+
"win": {
|
| 5 |
+
"seen": {
|
| 6 |
+
"n": 300,
|
| 7 |
+
"invocab": 235,
|
| 8 |
+
"hits_in": 228,
|
| 9 |
+
"hits_all": 290,
|
| 10 |
+
"pct": 97.02127659574468
|
| 11 |
+
},
|
| 12 |
+
"fc2": {
|
| 13 |
+
"n": 300,
|
| 14 |
+
"invocab": 187,
|
| 15 |
+
"hits_in": 31,
|
| 16 |
+
"hits_all": 35,
|
| 17 |
+
"pct": 16.577540106951872
|
| 18 |
+
},
|
| 19 |
+
"fresh_en": {
|
| 20 |
+
"n": 300,
|
| 21 |
+
"invocab": 202,
|
| 22 |
+
"hits_in": 5,
|
| 23 |
+
"hits_all": 5,
|
| 24 |
+
"pct": 2.4752475247524752
|
| 25 |
+
},
|
| 26 |
+
"fresh_fr": {
|
| 27 |
+
"n": 300,
|
| 28 |
+
"invocab": 102,
|
| 29 |
+
"hits_in": 1,
|
| 30 |
+
"hits_all": 1,
|
| 31 |
+
"pct": 0.9803921568627451
|
| 32 |
+
},
|
| 33 |
+
"neither_code": {
|
| 34 |
+
"n": 300,
|
| 35 |
+
"invocab": 179,
|
| 36 |
+
"hits_in": 14,
|
| 37 |
+
"hits_all": 14,
|
| 38 |
+
"pct": 7.82122905027933
|
| 39 |
+
},
|
| 40 |
+
"test20k": {
|
| 41 |
+
"n": 300,
|
| 42 |
+
"invocab": 201,
|
| 43 |
+
"hits_in": 4,
|
| 44 |
+
"hits_all": 4,
|
| 45 |
+
"pct": 1.9900497512437811
|
| 46 |
+
},
|
| 47 |
+
"queries": {
|
| 48 |
+
"n": 300,
|
| 49 |
+
"invocab": 141,
|
| 50 |
+
"hits_in": 0,
|
| 51 |
+
"hits_all": 0,
|
| 52 |
+
"pct": 0.0
|
| 53 |
+
},
|
| 54 |
+
"test_v3": {
|
| 55 |
+
"n": 300,
|
| 56 |
+
"invocab": 120,
|
| 57 |
+
"hits_in": 17,
|
| 58 |
+
"hits_all": 17,
|
| 59 |
+
"pct": 14.166666666666666
|
| 60 |
+
}
|
| 61 |
+
},
|
| 62 |
+
"fresh_mean_excl_nc": 6.031649367746257,
|
| 63 |
+
"gap_excl_nc": 90.98962722799843,
|
| 64 |
+
"fresh_mean_incl_nc": 6.287303608108124,
|
| 65 |
+
"gap_incl_nc": 90.73397298763656
|
| 66 |
+
},
|
| 67 |
+
"broad14_ref": {
|
| 68 |
+
"win": {
|
| 69 |
+
"seen": {
|
| 70 |
+
"n": 300,
|
| 71 |
+
"invocab": 255,
|
| 72 |
+
"hits_in": 232,
|
| 73 |
+
"hits_all": 272,
|
| 74 |
+
"pct": 90.98039215686275
|
| 75 |
+
},
|
| 76 |
+
"fc2": {
|
| 77 |
+
"n": 300,
|
| 78 |
+
"invocab": 246,
|
| 79 |
+
"hits_in": 2,
|
| 80 |
+
"hits_all": 2,
|
| 81 |
+
"pct": 0.8130081300813008
|
| 82 |
+
},
|
| 83 |
+
"fresh_en": {
|
| 84 |
+
"n": 300,
|
| 85 |
+
"invocab": 234,
|
| 86 |
+
"hits_in": 9,
|
| 87 |
+
"hits_all": 9,
|
| 88 |
+
"pct": 3.8461538461538463
|
| 89 |
+
},
|
| 90 |
+
"fresh_fr": {
|
| 91 |
+
"n": 300,
|
| 92 |
+
"invocab": 247,
|
| 93 |
+
"hits_in": 79,
|
| 94 |
+
"hits_all": 79,
|
| 95 |
+
"pct": 31.983805668016196
|
| 96 |
+
},
|
| 97 |
+
"neither_code": {
|
| 98 |
+
"n": 300,
|
| 99 |
+
"invocab": 244,
|
| 100 |
+
"hits_in": 7,
|
| 101 |
+
"hits_all": 7,
|
| 102 |
+
"pct": 2.8688524590163933
|
| 103 |
+
},
|
| 104 |
+
"test20k": {
|
| 105 |
+
"n": 300,
|
| 106 |
+
"invocab": 233,
|
| 107 |
+
"hits_in": 5,
|
| 108 |
+
"hits_all": 5,
|
| 109 |
+
"pct": 2.1459227467811157
|
| 110 |
+
},
|
| 111 |
+
"queries": {
|
| 112 |
+
"n": 300,
|
| 113 |
+
"invocab": 186,
|
| 114 |
+
"hits_in": 15,
|
| 115 |
+
"hits_all": 15,
|
| 116 |
+
"pct": 8.064516129032258
|
| 117 |
+
},
|
| 118 |
+
"test_v3": {
|
| 119 |
+
"n": 300,
|
| 120 |
+
"invocab": 171,
|
| 121 |
+
"hits_in": 1,
|
| 122 |
+
"hits_all": 1,
|
| 123 |
+
"pct": 0.5847953216374269
|
| 124 |
+
}
|
| 125 |
+
},
|
| 126 |
+
"fresh_mean_excl_nc": 7.906366973617025,
|
| 127 |
+
"gap_excl_nc": 83.07402518324572,
|
| 128 |
+
"fresh_mean_incl_nc": 7.186722042959792,
|
| 129 |
+
"gap_incl_nc": 83.79367011390296
|
| 130 |
+
}
|
| 131 |
+
},
|
| 132 |
+
"pairs": {
|
| 133 |
+
"seen": {
|
| 134 |
+
"A_only": 50,
|
| 135 |
+
"B_only": 54,
|
| 136 |
+
"p": 0.7687918101386728,
|
| 137 |
+
"raw_A_only": 25,
|
| 138 |
+
"raw_B_only": 7,
|
| 139 |
+
"raw_p": 0.0021024015732109547
|
| 140 |
+
},
|
| 141 |
+
"fc2": {
|
| 142 |
+
"A_only": 31,
|
| 143 |
+
"B_only": 2,
|
| 144 |
+
"p": 1.3085082173347473e-07,
|
| 145 |
+
"raw_A_only": 35,
|
| 146 |
+
"raw_B_only": 2,
|
| 147 |
+
"raw_p": 1.0244548320770264e-08
|
| 148 |
+
},
|
| 149 |
+
"fresh_en": {
|
| 150 |
+
"A_only": 5,
|
| 151 |
+
"B_only": 9,
|
| 152 |
+
"p": 0.4239501953125,
|
| 153 |
+
"raw_A_only": 5,
|
| 154 |
+
"raw_B_only": 9,
|
| 155 |
+
"raw_p": 0.4239501953125
|
| 156 |
+
},
|
| 157 |
+
"fresh_fr": {
|
| 158 |
+
"A_only": 1,
|
| 159 |
+
"B_only": 79,
|
| 160 |
+
"p": 1.3400325923359048e-22,
|
| 161 |
+
"raw_A_only": 1,
|
| 162 |
+
"raw_B_only": 79,
|
| 163 |
+
"raw_p": 1.3400325923359048e-22
|
| 164 |
+
},
|
| 165 |
+
"neither_code": {
|
| 166 |
+
"A_only": 12,
|
| 167 |
+
"B_only": 5,
|
| 168 |
+
"p": 0.143463134765625,
|
| 169 |
+
"raw_A_only": 12,
|
| 170 |
+
"raw_B_only": 5,
|
| 171 |
+
"raw_p": 0.143463134765625
|
| 172 |
+
},
|
| 173 |
+
"test20k": {
|
| 174 |
+
"A_only": 4,
|
| 175 |
+
"B_only": 5,
|
| 176 |
+
"p": 1.0,
|
| 177 |
+
"raw_A_only": 4,
|
| 178 |
+
"raw_B_only": 5,
|
| 179 |
+
"raw_p": 1.0
|
| 180 |
+
},
|
| 181 |
+
"queries": {
|
| 182 |
+
"A_only": 0,
|
| 183 |
+
"B_only": 15,
|
| 184 |
+
"p": 6.103515625e-05,
|
| 185 |
+
"raw_A_only": 0,
|
| 186 |
+
"raw_B_only": 15,
|
| 187 |
+
"raw_p": 6.103515625e-05
|
| 188 |
+
},
|
| 189 |
+
"test_v3": {
|
| 190 |
+
"A_only": 17,
|
| 191 |
+
"B_only": 1,
|
| 192 |
+
"p": 0.00014495849609375,
|
| 193 |
+
"raw_A_only": 17,
|
| 194 |
+
"raw_B_only": 1,
|
| 195 |
+
"raw_p": 0.00014495849609375
|
| 196 |
+
}
|
| 197 |
+
}
|
| 198 |
+
}
|
v3_scale/reports/recheck_v4broad14_1748.json
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"broad14_ref/seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 6980,
|
| 5 |
+
"fused": 6417,
|
| 6 |
+
"snn": 3571,
|
| 7 |
+
"fused_pct": 91.934,
|
| 8 |
+
"snn_pct": 51.16,
|
| 9 |
+
"cov": 0.8725,
|
| 10 |
+
"reported": [
|
| 11 |
+
91.934,
|
| 12 |
+
51.16,
|
| 13 |
+
0.8725
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"broad14_ref/fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 6776,
|
| 19 |
+
"fused": 573,
|
| 20 |
+
"snn": 207,
|
| 21 |
+
"fused_pct": 8.456,
|
| 22 |
+
"snn_pct": 3.055,
|
| 23 |
+
"cov": 0.847,
|
| 24 |
+
"reported": [
|
| 25 |
+
8.456,
|
| 26 |
+
3.055,
|
| 27 |
+
0.847
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"broad14_ref/fresh_en": {
|
| 31 |
+
"n": 8000,
|
| 32 |
+
"invocab": 6503,
|
| 33 |
+
"fused": 336,
|
| 34 |
+
"snn": 325,
|
| 35 |
+
"fused_pct": 5.167,
|
| 36 |
+
"snn_pct": 4.998,
|
| 37 |
+
"cov": 0.812875,
|
| 38 |
+
"reported": [
|
| 39 |
+
5.167,
|
| 40 |
+
4.998,
|
| 41 |
+
0.812875
|
| 42 |
+
]
|
| 43 |
+
}
|
| 44 |
+
}
|
v3_scale/reports/recheck_v4broad14_1755.json
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"broad14_ref/seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 6980,
|
| 5 |
+
"fused": 6417,
|
| 6 |
+
"snn": 3571,
|
| 7 |
+
"fused_pct": 91.934,
|
| 8 |
+
"snn_pct": 51.16,
|
| 9 |
+
"cov": 0.8725,
|
| 10 |
+
"reported": [
|
| 11 |
+
91.934,
|
| 12 |
+
51.16,
|
| 13 |
+
0.8725
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"broad14_ref/fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 6776,
|
| 19 |
+
"fused": 573,
|
| 20 |
+
"snn": 207,
|
| 21 |
+
"fused_pct": 8.456,
|
| 22 |
+
"snn_pct": 3.055,
|
| 23 |
+
"cov": 0.847,
|
| 24 |
+
"reported": [
|
| 25 |
+
8.456,
|
| 26 |
+
3.055,
|
| 27 |
+
0.847
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"broad14_ref/fresh_en": {
|
| 31 |
+
"n": 8000,
|
| 32 |
+
"invocab": 6503,
|
| 33 |
+
"fused": 336,
|
| 34 |
+
"snn": 325,
|
| 35 |
+
"fused_pct": 5.167,
|
| 36 |
+
"snn_pct": 4.998,
|
| 37 |
+
"cov": 0.812875,
|
| 38 |
+
"reported": [
|
| 39 |
+
5.167,
|
| 40 |
+
4.998,
|
| 41 |
+
0.812875
|
| 42 |
+
]
|
| 43 |
+
},
|
| 44 |
+
"broad14_ref/fresh_fr": {
|
| 45 |
+
"n": 8000,
|
| 46 |
+
"invocab": 4748,
|
| 47 |
+
"fused": 129,
|
| 48 |
+
"snn": 20,
|
| 49 |
+
"fused_pct": 2.717,
|
| 50 |
+
"snn_pct": 0.421,
|
| 51 |
+
"cov": 0.5935,
|
| 52 |
+
"reported": [
|
| 53 |
+
2.717,
|
| 54 |
+
0.421,
|
| 55 |
+
0.5935
|
| 56 |
+
]
|
| 57 |
+
},
|
| 58 |
+
"broad14_ref/neither_code": {
|
| 59 |
+
"n": 8000,
|
| 60 |
+
"invocab": 6819,
|
| 61 |
+
"fused": 523,
|
| 62 |
+
"snn": 182,
|
| 63 |
+
"fused_pct": 7.67,
|
| 64 |
+
"snn_pct": 2.669,
|
| 65 |
+
"cov": 0.852375,
|
| 66 |
+
"reported": [
|
| 67 |
+
7.67,
|
| 68 |
+
2.669,
|
| 69 |
+
0.852375
|
| 70 |
+
]
|
| 71 |
+
}
|
| 72 |
+
}
|
v3_scale/reports/recheck_v4broad14_1758.json
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"broad14_ref/seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 6980,
|
| 5 |
+
"fused": 6417,
|
| 6 |
+
"snn": 3571,
|
| 7 |
+
"fused_pct": 91.934,
|
| 8 |
+
"snn_pct": 51.16,
|
| 9 |
+
"cov": 0.8725,
|
| 10 |
+
"reported": [
|
| 11 |
+
91.934,
|
| 12 |
+
51.16,
|
| 13 |
+
0.8725
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"broad14_ref/fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 6776,
|
| 19 |
+
"fused": 573,
|
| 20 |
+
"snn": 207,
|
| 21 |
+
"fused_pct": 8.456,
|
| 22 |
+
"snn_pct": 3.055,
|
| 23 |
+
"cov": 0.847,
|
| 24 |
+
"reported": [
|
| 25 |
+
8.456,
|
| 26 |
+
3.055,
|
| 27 |
+
0.847
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"broad14_ref/fresh_en": {
|
| 31 |
+
"n": 8000,
|
| 32 |
+
"invocab": 6503,
|
| 33 |
+
"fused": 336,
|
| 34 |
+
"snn": 325,
|
| 35 |
+
"fused_pct": 5.167,
|
| 36 |
+
"snn_pct": 4.998,
|
| 37 |
+
"cov": 0.812875,
|
| 38 |
+
"reported": [
|
| 39 |
+
5.167,
|
| 40 |
+
4.998,
|
| 41 |
+
0.812875
|
| 42 |
+
]
|
| 43 |
+
},
|
| 44 |
+
"broad14_ref/fresh_fr": {
|
| 45 |
+
"n": 8000,
|
| 46 |
+
"invocab": 4748,
|
| 47 |
+
"fused": 129,
|
| 48 |
+
"snn": 20,
|
| 49 |
+
"fused_pct": 2.717,
|
| 50 |
+
"snn_pct": 0.421,
|
| 51 |
+
"cov": 0.5935,
|
| 52 |
+
"reported": [
|
| 53 |
+
2.717,
|
| 54 |
+
0.421,
|
| 55 |
+
0.5935
|
| 56 |
+
]
|
| 57 |
+
},
|
| 58 |
+
"broad14_ref/neither_code": {
|
| 59 |
+
"n": 8000,
|
| 60 |
+
"invocab": 6819,
|
| 61 |
+
"fused": 523,
|
| 62 |
+
"snn": 182,
|
| 63 |
+
"fused_pct": 7.67,
|
| 64 |
+
"snn_pct": 2.669,
|
| 65 |
+
"cov": 0.852375,
|
| 66 |
+
"reported": [
|
| 67 |
+
7.67,
|
| 68 |
+
2.669,
|
| 69 |
+
0.852375
|
| 70 |
+
]
|
| 71 |
+
},
|
| 72 |
+
"broad14_ref/test20k": {
|
| 73 |
+
"n": 20000,
|
| 74 |
+
"invocab": 15652,
|
| 75 |
+
"fused": 506,
|
| 76 |
+
"snn": 501,
|
| 77 |
+
"fused_pct": 3.233,
|
| 78 |
+
"snn_pct": 3.201,
|
| 79 |
+
"cov": 0.7826,
|
| 80 |
+
"reported": [
|
| 81 |
+
3.233,
|
| 82 |
+
3.201,
|
| 83 |
+
0.7826
|
| 84 |
+
]
|
| 85 |
+
}
|
| 86 |
+
}
|
v3_scale/reports/recheck_v4code_1733.json
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"old_ref/seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 5868,
|
| 5 |
+
"fused": 5770,
|
| 6 |
+
"snn": 1937,
|
| 7 |
+
"fused_pct": 98.33,
|
| 8 |
+
"snn_pct": 33.01,
|
| 9 |
+
"cov": 0.7335,
|
| 10 |
+
"reported": [
|
| 11 |
+
98.33,
|
| 12 |
+
33.01,
|
| 13 |
+
0.7335
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"old_ref/fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 4833,
|
| 19 |
+
"fused": 556,
|
| 20 |
+
"snn": 403,
|
| 21 |
+
"fused_pct": 11.504,
|
| 22 |
+
"snn_pct": 8.339,
|
| 23 |
+
"cov": 0.604125,
|
| 24 |
+
"reported": [
|
| 25 |
+
11.504,
|
| 26 |
+
8.339,
|
| 27 |
+
0.604125
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"old_ref/fresh_en": {
|
| 31 |
+
"n": 8000,
|
| 32 |
+
"invocab": 5714,
|
| 33 |
+
"fused": 194,
|
| 34 |
+
"snn": 175,
|
| 35 |
+
"fused_pct": 3.395,
|
| 36 |
+
"snn_pct": 3.063,
|
| 37 |
+
"cov": 0.71425,
|
| 38 |
+
"reported": [
|
| 39 |
+
3.395,
|
| 40 |
+
3.063,
|
| 41 |
+
0.71425
|
| 42 |
+
]
|
| 43 |
+
},
|
| 44 |
+
"old_ref/fresh_fr": {
|
| 45 |
+
"n": 8000,
|
| 46 |
+
"invocab": 3179,
|
| 47 |
+
"fused": 9,
|
| 48 |
+
"snn": 9,
|
| 49 |
+
"fused_pct": 0.283,
|
| 50 |
+
"snn_pct": 0.283,
|
| 51 |
+
"cov": 0.397375,
|
| 52 |
+
"reported": [
|
| 53 |
+
0.283,
|
| 54 |
+
0.283,
|
| 55 |
+
0.397375
|
| 56 |
+
]
|
| 57 |
+
},
|
| 58 |
+
"old_ref/neither_code": {
|
| 59 |
+
"n": 8000,
|
| 60 |
+
"invocab": 4474,
|
| 61 |
+
"fused": 311,
|
| 62 |
+
"snn": 229,
|
| 63 |
+
"fused_pct": 6.951,
|
| 64 |
+
"snn_pct": 5.118,
|
| 65 |
+
"cov": 0.55925,
|
| 66 |
+
"reported": [
|
| 67 |
+
6.951,
|
| 68 |
+
5.118,
|
| 69 |
+
0.55925
|
| 70 |
+
]
|
| 71 |
+
}
|
| 72 |
+
}
|
v3_scale/reports/recheck_v4fresh_gen_1728.json
ADDED
|
@@ -0,0 +1,368 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bench": {
|
| 3 |
+
"old_ref/seen": {
|
| 4 |
+
"n": 8000,
|
| 5 |
+
"invocab": 5868,
|
| 6 |
+
"fused": 5770,
|
| 7 |
+
"snn": 1937,
|
| 8 |
+
"fused_pct": 98.33,
|
| 9 |
+
"snn_pct": 33.01,
|
| 10 |
+
"cov": 0.7335,
|
| 11 |
+
"reported": [
|
| 12 |
+
98.33,
|
| 13 |
+
33.01,
|
| 14 |
+
0.7335
|
| 15 |
+
]
|
| 16 |
+
},
|
| 17 |
+
"old_ref/fc2": {
|
| 18 |
+
"n": 8000,
|
| 19 |
+
"invocab": 4833,
|
| 20 |
+
"fused": 556,
|
| 21 |
+
"snn": 403,
|
| 22 |
+
"fused_pct": 11.504,
|
| 23 |
+
"snn_pct": 8.339,
|
| 24 |
+
"cov": 0.604125,
|
| 25 |
+
"reported": [
|
| 26 |
+
11.504,
|
| 27 |
+
8.339,
|
| 28 |
+
0.604125
|
| 29 |
+
]
|
| 30 |
+
},
|
| 31 |
+
"old_ref/fresh_en": {
|
| 32 |
+
"n": 8000,
|
| 33 |
+
"invocab": 5714,
|
| 34 |
+
"fused": 194,
|
| 35 |
+
"snn": 175,
|
| 36 |
+
"fused_pct": 3.395,
|
| 37 |
+
"snn_pct": 3.063,
|
| 38 |
+
"cov": 0.71425,
|
| 39 |
+
"reported": [
|
| 40 |
+
3.395,
|
| 41 |
+
3.063,
|
| 42 |
+
0.71425
|
| 43 |
+
]
|
| 44 |
+
},
|
| 45 |
+
"old_ref/fresh_fr": {
|
| 46 |
+
"n": 8000,
|
| 47 |
+
"invocab": 3179,
|
| 48 |
+
"fused": 9,
|
| 49 |
+
"snn": 9,
|
| 50 |
+
"fused_pct": 0.283,
|
| 51 |
+
"snn_pct": 0.283,
|
| 52 |
+
"cov": 0.397375,
|
| 53 |
+
"reported": [
|
| 54 |
+
0.283,
|
| 55 |
+
0.283,
|
| 56 |
+
0.397375
|
| 57 |
+
]
|
| 58 |
+
}
|
| 59 |
+
},
|
| 60 |
+
"gen": {
|
| 61 |
+
"old_ref/fused/en1": {
|
| 62 |
+
"byte_acc": 0.06,
|
| 63 |
+
"rep_byte_acc": 0.055,
|
| 64 |
+
"exact_bytes": false,
|
| 65 |
+
"vw": 0.4762,
|
| 66 |
+
"rep_vw": 0.4762,
|
| 67 |
+
"nw": 21,
|
| 68 |
+
"rep_nw": 21
|
| 69 |
+
},
|
| 70 |
+
"old_ref/fused/en2": {
|
| 71 |
+
"byte_acc": 0.085,
|
| 72 |
+
"rep_byte_acc": 0.085,
|
| 73 |
+
"exact_bytes": true,
|
| 74 |
+
"vw": 0.3158,
|
| 75 |
+
"rep_vw": 0.3158,
|
| 76 |
+
"nw": 19,
|
| 77 |
+
"rep_nw": 19
|
| 78 |
+
},
|
| 79 |
+
"old_ref/fused/fr1": {
|
| 80 |
+
"byte_acc": 0.025,
|
| 81 |
+
"rep_byte_acc": 0.02,
|
| 82 |
+
"exact_bytes": false,
|
| 83 |
+
"vw": 0.64,
|
| 84 |
+
"rep_vw": 0.64,
|
| 85 |
+
"nw": 25,
|
| 86 |
+
"rep_nw": 25
|
| 87 |
+
},
|
| 88 |
+
"old_ref/fused/fr2": {
|
| 89 |
+
"byte_acc": 0.09,
|
| 90 |
+
"rep_byte_acc": 0.09,
|
| 91 |
+
"exact_bytes": true,
|
| 92 |
+
"vw": 0.7333,
|
| 93 |
+
"rep_vw": 0.7333,
|
| 94 |
+
"nw": 15,
|
| 95 |
+
"rep_nw": 15
|
| 96 |
+
},
|
| 97 |
+
"old_ref/fused/chat": {
|
| 98 |
+
"byte_acc": 0.08,
|
| 99 |
+
"rep_byte_acc": 0.08,
|
| 100 |
+
"exact_bytes": true,
|
| 101 |
+
"vw": 0.5,
|
| 102 |
+
"rep_vw": 0.5,
|
| 103 |
+
"nw": 26,
|
| 104 |
+
"rep_nw": 26
|
| 105 |
+
},
|
| 106 |
+
"old_ref/fused/code": {
|
| 107 |
+
"byte_acc": 0.095,
|
| 108 |
+
"rep_byte_acc": 0.095,
|
| 109 |
+
"exact_bytes": true,
|
| 110 |
+
"vw": 0.4118,
|
| 111 |
+
"rep_vw": 0.4118,
|
| 112 |
+
"nw": 17,
|
| 113 |
+
"rep_nw": 17
|
| 114 |
+
},
|
| 115 |
+
"old_ref/fused/v3test": {
|
| 116 |
+
"byte_acc": 0.08,
|
| 117 |
+
"rep_byte_acc": 0.08,
|
| 118 |
+
"exact_bytes": true,
|
| 119 |
+
"vw": 0.52,
|
| 120 |
+
"rep_vw": 0.52,
|
| 121 |
+
"nw": 25,
|
| 122 |
+
"rep_nw": 25
|
| 123 |
+
},
|
| 124 |
+
"old_ref/v2b/en1": {
|
| 125 |
+
"byte_acc": 0.065,
|
| 126 |
+
"rep_byte_acc": 0.065,
|
| 127 |
+
"exact_bytes": true,
|
| 128 |
+
"vw": 0.9545,
|
| 129 |
+
"rep_vw": 0.9545,
|
| 130 |
+
"nw": 22,
|
| 131 |
+
"rep_nw": 22
|
| 132 |
+
},
|
| 133 |
+
"old_ref/v2b/en2": {
|
| 134 |
+
"byte_acc": 0.085,
|
| 135 |
+
"rep_byte_acc": 0.085,
|
| 136 |
+
"exact_bytes": true,
|
| 137 |
+
"vw": 0.9744,
|
| 138 |
+
"rep_vw": 0.9744,
|
| 139 |
+
"nw": 39,
|
| 140 |
+
"rep_nw": 39
|
| 141 |
+
},
|
| 142 |
+
"old_ref/v2b/fr1": {
|
| 143 |
+
"byte_acc": 0.06,
|
| 144 |
+
"rep_byte_acc": 0.06,
|
| 145 |
+
"exact_bytes": false,
|
| 146 |
+
"vw": 0.931,
|
| 147 |
+
"rep_vw": 0.931,
|
| 148 |
+
"nw": 29,
|
| 149 |
+
"rep_nw": 29
|
| 150 |
+
},
|
| 151 |
+
"old_ref/v2b/fr2": {
|
| 152 |
+
"byte_acc": 0.06,
|
| 153 |
+
"rep_byte_acc": 0.06,
|
| 154 |
+
"exact_bytes": true,
|
| 155 |
+
"vw": 1.0,
|
| 156 |
+
"rep_vw": 1.0,
|
| 157 |
+
"nw": 27,
|
| 158 |
+
"rep_nw": 27
|
| 159 |
+
},
|
| 160 |
+
"old_ref/v2b/chat": {
|
| 161 |
+
"byte_acc": 0.09,
|
| 162 |
+
"rep_byte_acc": 0.09,
|
| 163 |
+
"exact_bytes": true,
|
| 164 |
+
"vw": 0.9714,
|
| 165 |
+
"rep_vw": 0.9714,
|
| 166 |
+
"nw": 35,
|
| 167 |
+
"rep_nw": 35
|
| 168 |
+
},
|
| 169 |
+
"old_ref/v2b/code": {
|
| 170 |
+
"byte_acc": 0.11,
|
| 171 |
+
"rep_byte_acc": 0.11,
|
| 172 |
+
"exact_bytes": true,
|
| 173 |
+
"vw": 0.9412,
|
| 174 |
+
"rep_vw": 0.9412,
|
| 175 |
+
"nw": 17,
|
| 176 |
+
"rep_nw": 17
|
| 177 |
+
},
|
| 178 |
+
"old_ref/v2b/v3test": {
|
| 179 |
+
"byte_acc": 0.1,
|
| 180 |
+
"rep_byte_acc": 0.1,
|
| 181 |
+
"exact_bytes": true,
|
| 182 |
+
"vw": 0.913,
|
| 183 |
+
"rep_vw": 0.913,
|
| 184 |
+
"nw": 23,
|
| 185 |
+
"rep_nw": 23
|
| 186 |
+
},
|
| 187 |
+
"broad14_ref/fused/en1": {
|
| 188 |
+
"byte_acc": 0.065,
|
| 189 |
+
"rep_byte_acc": 0.065,
|
| 190 |
+
"exact_bytes": true,
|
| 191 |
+
"vw": 0.9211,
|
| 192 |
+
"rep_vw": 0.9211,
|
| 193 |
+
"nw": 38,
|
| 194 |
+
"rep_nw": 38
|
| 195 |
+
},
|
| 196 |
+
"broad14_ref/fused/en2": {
|
| 197 |
+
"byte_acc": 0.05,
|
| 198 |
+
"rep_byte_acc": 0.05,
|
| 199 |
+
"exact_bytes": true,
|
| 200 |
+
"vw": 0.5938,
|
| 201 |
+
"rep_vw": 0.5938,
|
| 202 |
+
"nw": 32,
|
| 203 |
+
"rep_nw": 32
|
| 204 |
+
},
|
| 205 |
+
"broad14_ref/fused/fr1": {
|
| 206 |
+
"byte_acc": 0.04,
|
| 207 |
+
"rep_byte_acc": 0.04,
|
| 208 |
+
"exact_bytes": true,
|
| 209 |
+
"vw": 0.6286,
|
| 210 |
+
"rep_vw": 0.6286,
|
| 211 |
+
"nw": 35,
|
| 212 |
+
"rep_nw": 35
|
| 213 |
+
},
|
| 214 |
+
"broad14_ref/fused/fr2": {
|
| 215 |
+
"byte_acc": 0.06,
|
| 216 |
+
"rep_byte_acc": 0.06,
|
| 217 |
+
"exact_bytes": true,
|
| 218 |
+
"vw": 0.8205,
|
| 219 |
+
"rep_vw": 0.8205,
|
| 220 |
+
"nw": 39,
|
| 221 |
+
"rep_nw": 39
|
| 222 |
+
},
|
| 223 |
+
"broad14_ref/fused/chat": {
|
| 224 |
+
"byte_acc": 0.065,
|
| 225 |
+
"rep_byte_acc": 0.065,
|
| 226 |
+
"exact_bytes": true,
|
| 227 |
+
"vw": 0.7895,
|
| 228 |
+
"rep_vw": 0.7895,
|
| 229 |
+
"nw": 38,
|
| 230 |
+
"rep_nw": 38
|
| 231 |
+
},
|
| 232 |
+
"broad14_ref/fused/code": {
|
| 233 |
+
"byte_acc": 0.065,
|
| 234 |
+
"rep_byte_acc": 0.065,
|
| 235 |
+
"exact_bytes": true,
|
| 236 |
+
"vw": 0.5429,
|
| 237 |
+
"rep_vw": 0.5429,
|
| 238 |
+
"nw": 35,
|
| 239 |
+
"rep_nw": 35
|
| 240 |
+
},
|
| 241 |
+
"broad14_ref/fused/v3test": {
|
| 242 |
+
"byte_acc": 0.075,
|
| 243 |
+
"rep_byte_acc": 0.075,
|
| 244 |
+
"exact_bytes": true,
|
| 245 |
+
"vw": 0.8974,
|
| 246 |
+
"rep_vw": 0.8974,
|
| 247 |
+
"nw": 39,
|
| 248 |
+
"rep_nw": 39
|
| 249 |
+
},
|
| 250 |
+
"broad14_ref/v2b/en1": {
|
| 251 |
+
"byte_acc": 0.055,
|
| 252 |
+
"rep_byte_acc": 0.055,
|
| 253 |
+
"exact_bytes": true,
|
| 254 |
+
"vw": 1.0,
|
| 255 |
+
"rep_vw": 1.0,
|
| 256 |
+
"nw": 34,
|
| 257 |
+
"rep_nw": 34
|
| 258 |
+
},
|
| 259 |
+
"broad14_ref/v2b/en2": {
|
| 260 |
+
"byte_acc": 0.08,
|
| 261 |
+
"rep_byte_acc": 0.08,
|
| 262 |
+
"exact_bytes": true,
|
| 263 |
+
"vw": 0.9737,
|
| 264 |
+
"rep_vw": 0.9737,
|
| 265 |
+
"nw": 38,
|
| 266 |
+
"rep_nw": 38
|
| 267 |
+
},
|
| 268 |
+
"broad14_ref/v2b/fr1": {
|
| 269 |
+
"byte_acc": 0.125,
|
| 270 |
+
"rep_byte_acc": 0.125,
|
| 271 |
+
"exact_bytes": true,
|
| 272 |
+
"vw": 0.5,
|
| 273 |
+
"rep_vw": 0.5,
|
| 274 |
+
"nw": 4,
|
| 275 |
+
"rep_nw": 4
|
| 276 |
+
},
|
| 277 |
+
"broad14_ref/v2b/fr2": {
|
| 278 |
+
"byte_acc": 0.05,
|
| 279 |
+
"rep_byte_acc": 0.05,
|
| 280 |
+
"exact_bytes": true,
|
| 281 |
+
"vw": 0.913,
|
| 282 |
+
"rep_vw": 0.913,
|
| 283 |
+
"nw": 23,
|
| 284 |
+
"rep_nw": 23
|
| 285 |
+
},
|
| 286 |
+
"broad14_ref/v2b/chat": {
|
| 287 |
+
"byte_acc": 0.12,
|
| 288 |
+
"rep_byte_acc": 0.12,
|
| 289 |
+
"exact_bytes": true,
|
| 290 |
+
"vw": 0.6667,
|
| 291 |
+
"rep_vw": 0.6667,
|
| 292 |
+
"nw": 3,
|
| 293 |
+
"rep_nw": 3
|
| 294 |
+
},
|
| 295 |
+
"broad14_ref/v2b/code": {
|
| 296 |
+
"byte_acc": 0.065,
|
| 297 |
+
"rep_byte_acc": 0.065,
|
| 298 |
+
"exact_bytes": true,
|
| 299 |
+
"vw": 0.8889,
|
| 300 |
+
"rep_vw": 0.8889,
|
| 301 |
+
"nw": 18,
|
| 302 |
+
"rep_nw": 18
|
| 303 |
+
},
|
| 304 |
+
"broad14_ref/v2b/v3test": {
|
| 305 |
+
"byte_acc": 0.11,
|
| 306 |
+
"rep_byte_acc": 0.11,
|
| 307 |
+
"exact_bytes": true,
|
| 308 |
+
"vw": 0.95,
|
| 309 |
+
"rep_vw": 0.95,
|
| 310 |
+
"nw": 40,
|
| 311 |
+
"rep_nw": 40
|
| 312 |
+
}
|
| 313 |
+
},
|
| 314 |
+
"gen_agg": {
|
| 315 |
+
"old_ref/fused": {
|
| 316 |
+
"n": 7,
|
| 317 |
+
"byte_acc": 0.0721,
|
| 318 |
+
"vw": 0.5139,
|
| 319 |
+
"rep8": 0.0555,
|
| 320 |
+
"lang_drift": 4,
|
| 321 |
+
"dom_drift": 1,
|
| 322 |
+
"copy_mean": 56.7
|
| 323 |
+
},
|
| 324 |
+
"old_ref/v2b": {
|
| 325 |
+
"n": 7,
|
| 326 |
+
"byte_acc": 0.0814,
|
| 327 |
+
"vw": 0.9551,
|
| 328 |
+
"rep8": 0.1813,
|
| 329 |
+
"lang_drift": 4,
|
| 330 |
+
"dom_drift": 1,
|
| 331 |
+
"copy_mean": 126.0
|
| 332 |
+
},
|
| 333 |
+
"broad14_ref/fused": {
|
| 334 |
+
"n": 7,
|
| 335 |
+
"byte_acc": 0.06,
|
| 336 |
+
"vw": 0.742,
|
| 337 |
+
"rep8": 0.0052,
|
| 338 |
+
"lang_drift": 3,
|
| 339 |
+
"dom_drift": 1,
|
| 340 |
+
"copy_mean": 70.0
|
| 341 |
+
},
|
| 342 |
+
"broad14_ref/v2b": {
|
| 343 |
+
"n": 7,
|
| 344 |
+
"byte_acc": 0.0864,
|
| 345 |
+
"vw": 0.8418,
|
| 346 |
+
"rep8": 0.2694,
|
| 347 |
+
"lang_drift": 2,
|
| 348 |
+
"dom_drift": 1,
|
| 349 |
+
"copy_mean": 101.4
|
| 350 |
+
}
|
| 351 |
+
},
|
| 352 |
+
"mcnemar_gen_bytes": {
|
| 353 |
+
"fused": {
|
| 354 |
+
"old_hits": 103,
|
| 355 |
+
"broad_hits": 84,
|
| 356 |
+
"old_only": 87,
|
| 357 |
+
"broad_only": 68,
|
| 358 |
+
"p": 0.14799763102341487
|
| 359 |
+
},
|
| 360 |
+
"v2b": {
|
| 361 |
+
"old_hits": 114,
|
| 362 |
+
"broad_hits": 121,
|
| 363 |
+
"old_only": 92,
|
| 364 |
+
"broad_only": 99,
|
| 365 |
+
"p": 0.6642959893958231
|
| 366 |
+
}
|
| 367 |
+
}
|
| 368 |
+
}
|
v3_scale/reports/recheck_v4t20q_1737.json
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"old_ref/seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 5868,
|
| 5 |
+
"fused": 5770,
|
| 6 |
+
"snn": 1937,
|
| 7 |
+
"fused_pct": 98.33,
|
| 8 |
+
"snn_pct": 33.01,
|
| 9 |
+
"cov": 0.7335,
|
| 10 |
+
"reported": [
|
| 11 |
+
98.33,
|
| 12 |
+
33.01,
|
| 13 |
+
0.7335
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"old_ref/fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 4833,
|
| 19 |
+
"fused": 556,
|
| 20 |
+
"snn": 403,
|
| 21 |
+
"fused_pct": 11.504,
|
| 22 |
+
"snn_pct": 8.339,
|
| 23 |
+
"cov": 0.604125,
|
| 24 |
+
"reported": [
|
| 25 |
+
11.504,
|
| 26 |
+
8.339,
|
| 27 |
+
0.604125
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"old_ref/fresh_en": {
|
| 31 |
+
"n": 8000,
|
| 32 |
+
"invocab": 5714,
|
| 33 |
+
"fused": 194,
|
| 34 |
+
"snn": 175,
|
| 35 |
+
"fused_pct": 3.395,
|
| 36 |
+
"snn_pct": 3.063,
|
| 37 |
+
"cov": 0.71425,
|
| 38 |
+
"reported": [
|
| 39 |
+
3.395,
|
| 40 |
+
3.063,
|
| 41 |
+
0.71425
|
| 42 |
+
]
|
| 43 |
+
},
|
| 44 |
+
"old_ref/fresh_fr": {
|
| 45 |
+
"n": 8000,
|
| 46 |
+
"invocab": 3179,
|
| 47 |
+
"fused": 9,
|
| 48 |
+
"snn": 9,
|
| 49 |
+
"fused_pct": 0.283,
|
| 50 |
+
"snn_pct": 0.283,
|
| 51 |
+
"cov": 0.397375,
|
| 52 |
+
"reported": [
|
| 53 |
+
0.283,
|
| 54 |
+
0.283,
|
| 55 |
+
0.397375
|
| 56 |
+
]
|
| 57 |
+
},
|
| 58 |
+
"old_ref/neither_code": {
|
| 59 |
+
"n": 8000,
|
| 60 |
+
"invocab": 4474,
|
| 61 |
+
"fused": 311,
|
| 62 |
+
"snn": 229,
|
| 63 |
+
"fused_pct": 6.951,
|
| 64 |
+
"snn_pct": 5.118,
|
| 65 |
+
"cov": 0.55925,
|
| 66 |
+
"reported": [
|
| 67 |
+
6.951,
|
| 68 |
+
5.118,
|
| 69 |
+
0.55925
|
| 70 |
+
]
|
| 71 |
+
},
|
| 72 |
+
"old_ref/test20k": {
|
| 73 |
+
"n": 20000,
|
| 74 |
+
"invocab": 13379,
|
| 75 |
+
"fused": 388,
|
| 76 |
+
"snn": 374,
|
| 77 |
+
"fused_pct": 2.9,
|
| 78 |
+
"snn_pct": 2.795,
|
| 79 |
+
"cov": 0.66895,
|
| 80 |
+
"reported": [
|
| 81 |
+
2.9,
|
| 82 |
+
2.795,
|
| 83 |
+
0.66895
|
| 84 |
+
]
|
| 85 |
+
},
|
| 86 |
+
"old_ref/queries": {
|
| 87 |
+
"n": 7099,
|
| 88 |
+
"invocab": 3349,
|
| 89 |
+
"fused": 0,
|
| 90 |
+
"snn": 0,
|
| 91 |
+
"fused_pct": 0.0,
|
| 92 |
+
"snn_pct": 0.0,
|
| 93 |
+
"cov": 0.4717565854345683,
|
| 94 |
+
"reported": [
|
| 95 |
+
0.0,
|
| 96 |
+
0.0,
|
| 97 |
+
0.4717565854345683
|
| 98 |
+
]
|
| 99 |
+
}
|
| 100 |
+
}
|
v3_scale/reports/recheck_v4testv3_1743.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"old_ref/seen": {
|
| 3 |
+
"n": 8000,
|
| 4 |
+
"invocab": 5868,
|
| 5 |
+
"fused": 5770,
|
| 6 |
+
"snn": 1937,
|
| 7 |
+
"fused_pct": 98.33,
|
| 8 |
+
"snn_pct": 33.01,
|
| 9 |
+
"cov": 0.7335,
|
| 10 |
+
"reported": [
|
| 11 |
+
98.33,
|
| 12 |
+
33.01,
|
| 13 |
+
0.7335
|
| 14 |
+
]
|
| 15 |
+
},
|
| 16 |
+
"old_ref/fc2": {
|
| 17 |
+
"n": 8000,
|
| 18 |
+
"invocab": 4833,
|
| 19 |
+
"fused": 556,
|
| 20 |
+
"snn": 403,
|
| 21 |
+
"fused_pct": 11.504,
|
| 22 |
+
"snn_pct": 8.339,
|
| 23 |
+
"cov": 0.604125,
|
| 24 |
+
"reported": [
|
| 25 |
+
11.504,
|
| 26 |
+
8.339,
|
| 27 |
+
0.604125
|
| 28 |
+
]
|
| 29 |
+
},
|
| 30 |
+
"old_ref/fresh_en": {
|
| 31 |
+
"n": 8000,
|
| 32 |
+
"invocab": 5714,
|
| 33 |
+
"fused": 194,
|
| 34 |
+
"snn": 175,
|
| 35 |
+
"fused_pct": 3.395,
|
| 36 |
+
"snn_pct": 3.063,
|
| 37 |
+
"cov": 0.71425,
|
| 38 |
+
"reported": [
|
| 39 |
+
3.395,
|
| 40 |
+
3.063,
|
| 41 |
+
0.71425
|
| 42 |
+
]
|
| 43 |
+
},
|
| 44 |
+
"old_ref/fresh_fr": {
|
| 45 |
+
"n": 8000,
|
| 46 |
+
"invocab": 3179,
|
| 47 |
+
"fused": 9,
|
| 48 |
+
"snn": 9,
|
| 49 |
+
"fused_pct": 0.283,
|
| 50 |
+
"snn_pct": 0.283,
|
| 51 |
+
"cov": 0.397375,
|
| 52 |
+
"reported": [
|
| 53 |
+
0.283,
|
| 54 |
+
0.283,
|
| 55 |
+
0.397375
|
| 56 |
+
]
|
| 57 |
+
},
|
| 58 |
+
"old_ref/neither_code": {
|
| 59 |
+
"n": 8000,
|
| 60 |
+
"invocab": 4474,
|
| 61 |
+
"fused": 311,
|
| 62 |
+
"snn": 229,
|
| 63 |
+
"fused_pct": 6.951,
|
| 64 |
+
"snn_pct": 5.118,
|
| 65 |
+
"cov": 0.55925,
|
| 66 |
+
"reported": [
|
| 67 |
+
6.951,
|
| 68 |
+
5.118,
|
| 69 |
+
0.55925
|
| 70 |
+
]
|
| 71 |
+
},
|
| 72 |
+
"old_ref/test20k": {
|
| 73 |
+
"n": 20000,
|
| 74 |
+
"invocab": 13379,
|
| 75 |
+
"fused": 388,
|
| 76 |
+
"snn": 374,
|
| 77 |
+
"fused_pct": 2.9,
|
| 78 |
+
"snn_pct": 2.795,
|
| 79 |
+
"cov": 0.66895,
|
| 80 |
+
"reported": [
|
| 81 |
+
2.9,
|
| 82 |
+
2.795,
|
| 83 |
+
0.66895
|
| 84 |
+
]
|
| 85 |
+
},
|
| 86 |
+
"old_ref/queries": {
|
| 87 |
+
"n": 7099,
|
| 88 |
+
"invocab": 3349,
|
| 89 |
+
"fused": 0,
|
| 90 |
+
"snn": 0,
|
| 91 |
+
"fused_pct": 0.0,
|
| 92 |
+
"snn_pct": 0.0,
|
| 93 |
+
"cov": 0.4717565854345683,
|
| 94 |
+
"reported": [
|
| 95 |
+
0.0,
|
| 96 |
+
0.0,
|
| 97 |
+
0.4717565854345683
|
| 98 |
+
]
|
| 99 |
+
},
|
| 100 |
+
"old_ref/test_v3": {
|
| 101 |
+
"n": 20000,
|
| 102 |
+
"invocab": 12922,
|
| 103 |
+
"fused": 532,
|
| 104 |
+
"snn": 510,
|
| 105 |
+
"fused_pct": 4.117,
|
| 106 |
+
"snn_pct": 3.947,
|
| 107 |
+
"cov": 0.6461,
|
| 108 |
+
"reported": [
|
| 109 |
+
4.117,
|
| 110 |
+
3.947,
|
| 111 |
+
0.6461
|
| 112 |
+
]
|
| 113 |
+
}
|
| 114 |
+
}
|
v3_scale/reports/results_v4.jsonl
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "seen", "n_positions": 8000, "fused_invocab_pct": 98.33, "snn_invocab_pct": 33.01, "vocab_coverage": 0.7335, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 128.8, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 2 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "fc2", "n_positions": 8000, "fused_invocab_pct": 11.504, "snn_invocab_pct": 8.339, "vocab_coverage": 0.604125, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 114.0, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 3 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "fresh_en", "n_positions": 8000, "fused_invocab_pct": 3.395, "snn_invocab_pct": 3.063, "vocab_coverage": 0.71425, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 239.1, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 4 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "fresh_fr", "n_positions": 8000, "fused_invocab_pct": 0.283, "snn_invocab_pct": 0.283, "vocab_coverage": 0.397375, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 187.0, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 5 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "neither_code", "n_positions": 8000, "fused_invocab_pct": 6.951, "snn_invocab_pct": 5.118, "vocab_coverage": 0.55925, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 155.4, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 6 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "test20k", "n_positions": 20000, "fused_invocab_pct": 2.9, "snn_invocab_pct": 2.795, "vocab_coverage": 0.66895, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 288.0, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 7 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "queries", "n_positions": 7099, "fused_invocab_pct": 0.0, "snn_invocab_pct": 0.0, "vocab_coverage": 0.4717565854345683, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 106.7, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 8 |
+
{"tag": "old_ref", "corpus": "old_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_old_local_v1.npz", "kind": "plain", "window": "test_v3", "n_positions": 20000, "fused_invocab_pct": 4.117, "snn_invocab_pct": 3.947, "vocab_coverage": 0.6461, "fusion_saved": {"scale": 0.034013241467013434, "sigma_fast": 0.03237474460777202, "sigma_slow": 0.9518276768525441}, "fusion_cal": {"scale": 0.03498957642842646, "sigma_fast": 0.03241877903014496, "sigma_slow": 0.9265267642339071}, "load_s": 69.1, "eval_s": 288.2, "peak_rss_mb": 710.1, "maxpos": 0, "script": "eval_bench_v4"}
|
| 9 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "seen", "n_positions": 8000, "fused_invocab_pct": 91.934, "snn_invocab_pct": 51.16, "vocab_coverage": 0.8725, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 113.5, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
| 10 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "fc2", "n_positions": 8000, "fused_invocab_pct": 8.456, "snn_invocab_pct": 3.055, "vocab_coverage": 0.847, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 114.7, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
| 11 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "fresh_en", "n_positions": 8000, "fused_invocab_pct": 5.167, "snn_invocab_pct": 4.998, "vocab_coverage": 0.812875, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 115.1, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
| 12 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "fresh_fr", "n_positions": 8000, "fused_invocab_pct": 2.717, "snn_invocab_pct": 0.421, "vocab_coverage": 0.5935, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 109.3, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
| 13 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "neither_code", "n_positions": 8000, "fused_invocab_pct": 7.67, "snn_invocab_pct": 2.669, "vocab_coverage": 0.852375, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 98.6, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
| 14 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "test20k", "n_positions": 20000, "fused_invocab_pct": 3.233, "snn_invocab_pct": 3.201, "vocab_coverage": 0.7826, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 241.0, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
| 15 |
+
{"tag": "broad14_ref", "corpus": "broad_local_v1", "ckpt": "/workspace/spira_shared/checker/ckpt/ref_fixed2cal_broad_local_v1.npz", "kind": "plain", "window": "queries", "n_positions": 7750, "fused_invocab_pct": 8.021, "snn_invocab_pct": 8.021, "vocab_coverage": 0.6258064516129033, "fusion_saved": {"scale": 0.034540013155740186, "sigma_fast": 0.03852611172600577, "sigma_slow": 1.115405241807302}, "fusion_cal": {"scale": 0.034140001563555386, "sigma_fast": 0.038580582326189834, "sigma_slow": 1.1300697293281554}, "load_s": 84.5, "eval_s": 84.6, "peak_rss_mb": 1280.0, "maxpos": 0, "script": "eval_bench_v4"}
|
v3_scale/reports/scale_bench_v4.json
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"models": {
|
| 3 |
+
"old (r\u00e9f)": {
|
| 4 |
+
"tag": "old_ref",
|
| 5 |
+
"acc_invocab": {
|
| 6 |
+
"seen": 98.32992501704159,
|
| 7 |
+
"fc2": 11.504241671839436,
|
| 8 |
+
"fresh_en": 3.3951697584879246,
|
| 9 |
+
"fresh_fr": 0.28310789556464294,
|
| 10 |
+
"neither_code": 6.951274027715691,
|
| 11 |
+
"test20k": 2.9000672696016148,
|
| 12 |
+
"queries": 0.0,
|
| 13 |
+
"test_v3": 4.117009750812568
|
| 14 |
+
},
|
| 15 |
+
"fresh_mean": 3.699932724384364,
|
| 16 |
+
"gap": 94.62999229265722,
|
| 17 |
+
"wall_s": 1576.3,
|
| 18 |
+
"peak_rss_mb": 710.1,
|
| 19 |
+
"scale": 0.03498957642842646,
|
| 20 |
+
"scale_ok": true
|
| 21 |
+
}
|
| 22 |
+
},
|
| 23 |
+
"pairs": []
|
| 24 |
+
}
|
v3_scale/reports/scale_bench_v4.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Banc scale v4 — fused_acc_invocab (%) — fenêtres fixes identiques (seen = corpus propre)
|
| 2 |
+
|
| 3 |
+
| modèle | corpus | seen | fc2 | fresh_en | fresh_fr | neither_code | test20k | queries | test_v3 | fresh moy. | écart seen−fresh | wall s | pic RAM Mo | scale cal |
|
| 4 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 5 |
+
| old (réf) | old_local_v1 | 98.33 | 11.50 | 3.40 | 0.28 | 6.95 | 2.90 | 0.00 | 4.12 | 3.70 | 94.63 | 1576 | 710 | 0.035 |
|
| 6 |
+
| broad 14 % (réf) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 7 |
+
| broad 45 % (cov b) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 8 |
+
| broad FULL (M8, AM 100 %) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 9 |
+
| broad_local_v1 cov a | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 10 |
+
| M8 broad CM 14 % | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 11 |
+
| corpus_v3 cov b | corpus_v3 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 12 |
+
|
| 13 |
+
## McNemar exact par paires (fused_hit, toutes positions)
|
| 14 |
+
| A | B | fenêtre | A seul | B seul | p | signe |
|
| 15 |
+
|---|---|---|---|---|---|---|
|
v3_scale/reports/scale_bench_v4_smoke.json
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"models": {
|
| 3 |
+
"old (r\u00e9f)": {
|
| 4 |
+
"tag": "old_ref",
|
| 5 |
+
"acc_invocab": {
|
| 6 |
+
"seen": 97.02127659574468,
|
| 7 |
+
"fc2": 16.577540106951872,
|
| 8 |
+
"fresh_en": 2.4752475247524752,
|
| 9 |
+
"fresh_fr": 0.9803921568627451,
|
| 10 |
+
"neither_code": 7.82122905027933,
|
| 11 |
+
"test20k": 1.9900497512437811,
|
| 12 |
+
"queries": 0.0,
|
| 13 |
+
"test_v3": 14.166666666666666
|
| 14 |
+
},
|
| 15 |
+
"fresh_mean": 6.031649367746257,
|
| 16 |
+
"gap": 90.98962722799843,
|
| 17 |
+
"wall_s": 93.3,
|
| 18 |
+
"peak_rss_mb": 710.7,
|
| 19 |
+
"scale": 0.03498957642842646,
|
| 20 |
+
"scale_ok": true
|
| 21 |
+
},
|
| 22 |
+
"broad 14 % (r\u00e9f)": {
|
| 23 |
+
"tag": "broad14_ref",
|
| 24 |
+
"acc_invocab": {
|
| 25 |
+
"seen": 90.98039215686275,
|
| 26 |
+
"fc2": 0.8130081300813009,
|
| 27 |
+
"fresh_en": 3.8461538461538463,
|
| 28 |
+
"fresh_fr": 31.983805668016196,
|
| 29 |
+
"neither_code": 2.8688524590163933,
|
| 30 |
+
"test20k": 2.1459227467811157,
|
| 31 |
+
"queries": 8.064516129032258,
|
| 32 |
+
"test_v3": 0.5847953216374269
|
| 33 |
+
},
|
| 34 |
+
"fresh_mean": 7.906366973617025,
|
| 35 |
+
"gap": 83.07402518324572,
|
| 36 |
+
"wall_s": 105.9,
|
| 37 |
+
"peak_rss_mb": 1280.0,
|
| 38 |
+
"scale": 0.034140001563555386,
|
| 39 |
+
"scale_ok": true
|
| 40 |
+
}
|
| 41 |
+
},
|
| 42 |
+
"pairs": [
|
| 43 |
+
{
|
| 44 |
+
"A": "old (r\u00e9f)",
|
| 45 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 46 |
+
"window": "seen",
|
| 47 |
+
"n": 300,
|
| 48 |
+
"A_only": 25,
|
| 49 |
+
"B_only": 7,
|
| 50 |
+
"p": 0.0021024015732109547
|
| 51 |
+
},
|
| 52 |
+
{
|
| 53 |
+
"A": "old (r\u00e9f)",
|
| 54 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 55 |
+
"window": "fc2",
|
| 56 |
+
"n": 300,
|
| 57 |
+
"A_only": 35,
|
| 58 |
+
"B_only": 2,
|
| 59 |
+
"p": 1.0244548320770264e-08
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"A": "old (r\u00e9f)",
|
| 63 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 64 |
+
"window": "fresh_en",
|
| 65 |
+
"n": 300,
|
| 66 |
+
"A_only": 5,
|
| 67 |
+
"B_only": 9,
|
| 68 |
+
"p": 0.4239501953125
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"A": "old (r\u00e9f)",
|
| 72 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 73 |
+
"window": "fresh_fr",
|
| 74 |
+
"n": 300,
|
| 75 |
+
"A_only": 1,
|
| 76 |
+
"B_only": 79,
|
| 77 |
+
"p": 1.3400325923359048e-22
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"A": "old (r\u00e9f)",
|
| 81 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 82 |
+
"window": "neither_code",
|
| 83 |
+
"n": 300,
|
| 84 |
+
"A_only": 12,
|
| 85 |
+
"B_only": 5,
|
| 86 |
+
"p": 0.143463134765625
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"A": "old (r\u00e9f)",
|
| 90 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 91 |
+
"window": "test20k",
|
| 92 |
+
"n": 300,
|
| 93 |
+
"A_only": 4,
|
| 94 |
+
"B_only": 5,
|
| 95 |
+
"p": 1.0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"A": "old (r\u00e9f)",
|
| 99 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 100 |
+
"window": "queries",
|
| 101 |
+
"n": 300,
|
| 102 |
+
"A_only": 0,
|
| 103 |
+
"B_only": 15,
|
| 104 |
+
"p": 6.103515625e-05
|
| 105 |
+
},
|
| 106 |
+
{
|
| 107 |
+
"A": "old (r\u00e9f)",
|
| 108 |
+
"B": "broad 14 % (r\u00e9f)",
|
| 109 |
+
"window": "test_v3",
|
| 110 |
+
"n": 300,
|
| 111 |
+
"A_only": 17,
|
| 112 |
+
"B_only": 1,
|
| 113 |
+
"p": 0.00014495849609375
|
| 114 |
+
}
|
| 115 |
+
]
|
| 116 |
+
}
|
v3_scale/reports/scale_bench_v4_smoke.md
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Banc scale v4 — fused_acc_invocab (%) — fenêtres fixes identiques (seen = corpus propre)
|
| 2 |
+
|
| 3 |
+
| modèle | corpus | seen | fc2 | fresh_en | fresh_fr | neither_code | test20k | queries | test_v3 | fresh moy. | écart seen−fresh | wall s | pic RAM Mo | scale cal |
|
| 4 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 5 |
+
| old (réf) | old_local_v1 | 97.02 | 16.58 | 2.48 | 0.98 | 7.82 | 1.99 | 0.00 | 14.17 | 6.03 | 90.99 | 93 | 711 | 0.035 |
|
| 6 |
+
| broad 14 % (réf) | broad_local_v1 | 90.98 | 0.81 | 3.85 | 31.98 | 2.87 | 2.15 | 8.06 | 0.58 | 7.91 | 83.07 | 106 | 1280 | 0.034 |
|
| 7 |
+
| broad 45 % (cov b) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 8 |
+
| broad FULL (M8, AM 100 %) | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 9 |
+
| broad_local_v1 cov a | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 10 |
+
| M8 broad CM 14 % | broad_local_v1 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 11 |
+
| corpus_v3 cov b | corpus_v3 | … | … | … | … | … | … | … | … | | en attente | | | |
|
| 12 |
+
|
| 13 |
+
## McNemar exact par paires (fused_hit, toutes positions)
|
| 14 |
+
| A | B | fenêtre | A seul | B seul | p | signe |
|
| 15 |
+
|---|---|---|---|---|---|---|
|
| 16 |
+
| old (réf) | broad 14 % (réf) | seen | 25 | 7 | 0.0021 | A>B * |
|
| 17 |
+
| old (réf) | broad 14 % (réf) | fc2 | 35 | 2 | 1e-08 | A>B * |
|
| 18 |
+
| old (réf) | broad 14 % (réf) | fresh_en | 5 | 9 | 0.42 | B>A |
|
| 19 |
+
| old (réf) | broad 14 % (réf) | fresh_fr | 1 | 79 | 1.3e-22 | B>A * |
|
| 20 |
+
| old (réf) | broad 14 % (réf) | neither_code | 12 | 5 | 0.14 | A>B |
|
| 21 |
+
| old (réf) | broad 14 % (réf) | test20k | 4 | 5 | 1 | B>A |
|
| 22 |
+
| old (réf) | broad 14 % (réf) | queries | 0 | 15 | 6.1e-05 | B>A * |
|
| 23 |
+
| old (réf) | broad 14 % (réf) | test_v3 | 17 | 1 | 0.00014 | A>B * |
|
v3_scale/reports/spira_relay.md
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|