--- license: apache-2.0 library_name: kip tags: - dataset-curation - deduplication - data-filtering - learned-hashing - simhash - minhash - quality-verification - distillation - teacher-student - linear-probe - data-pipeline - multilingual --- # KIP-Hash v2 — a frontier-distilled verify-and-dedup hash **One 128-bit signature, two jobs, no neural network at score time:** fuzzy dedup AND quality verification from the same bits, pure-stdlib inference, weights packed as int8 (5.9 MB). KIP-Hash v2 is the hashing layer of [KIP](https://huggingface.co/IO-AIR), an open $0-cost data pipeline — and the direct upgrade of [kip-hash-v1](https://huggingface.co/IO-AIR/kip-hash-v1). The v1 idea — teach a hash to verify rows — was validated with a GLM teacher on 136 rubric labels. v2 rebuilds the whole stack around a **frontier teacher** (the DeepSeek webapp's default chat model), a **pairwise-aware labeling protocol** the teacher's verdicts on *pairs*, not just rows), and a **learned projection** trained on teacher-adjudicated duplicate pairs and teacher-rejected hard negatives. To our knowledge this is still the only shipped hash whose single signature answers "is this a duplicate?" and "is this worth keeping?" at the same time — and v2 is the first version whose dedup bits are *trained on neural-verifier judgments of ambiguous pairs* rather than sampled by IDF heuristics. ## What changed from v1 → v2 | | v1 (GLM) | v2 (DeepSeek webapp) | |---|---|---| | teacher | glm-4-plus, one doc per call | frontier chat model, 6 rows/chat + 6 pairs/chat | | teacher labels | 136 rows, 0-5 scale | **844 rows, 0-100 scale**, + 58 adjudicated pairs | | languages | 15 | 15 (full spread kept) | | pair supervision | none | injected pairs (40, recall-tested) + organic (3) + ambiguous-band adjudication (58) | | dedup bits | IDF-weighted MinHash sampling (heuristic learning) | **learned projection W, contrastive on teacher-adjudicated pairs** | | verify from hash | 2-bit quality prefix | linear head on all **128 bits** (+ finer head on the projection) | | junk/boilerplate AUC (new teacher) | 0.689 | **0.755** | | Spearman vs teacher (cross-teacher for v1) | 0.298 | **0.424 hash-only / 0.543 projection** | | false merges in the ambiguous band (13 hard pairs) | 1 (weighted Jaccard) | **0** | | artifact | 18.4 MB | **5.9 MB** | ## How one hash does two jobs **Features.** Every row becomes a sparse L2-normalized log-tf vector over blake2b-hashed buckets (32768): word unigrams, word bigrams, char 4-grams (CJK text uses char unigram/bigram + 4-grams). The feature extractor is one pure-stdlib function, imported *by the training script*, so train/infer parity is structural, not by convention. **Dedup: learned locality bits.** The 128 bits are `sign(W·x)` where `W ∈ R^{32768×128}` is *trained* — not sampled: - positives = near-duplicates the teacher confirmed (paraphrase injections with known ground truth, organic corpus dups, plus the one true dup found in the ambiguous band); - hard negatives = pairs from the *same chat* that the teacher scored but did **not** call duplicates (same language, same domain, shared boilerplate/vocabulary — the exact pairs that fool plain MinHash), plus 44 ambiguous-band pairs adjudicated pair-by-pair (MinHash-128 similarity 0.07-0.69; the teacher says not-dup); - the objective is a cosine-margin contrastive loss in projection space (positives pulled ≥ 0.70, negatives pushed ≤ 0.30), anchored to the random-projection prior for generalization. Plain MinHash treats every shingle equally; the learned W weights *content* shingles over boilerplate, so two unrelated pages that share a navigation block stay far in Hamming space while a paraphrased duplicate stays close. Candidate generation is banding (8 bands × 16 bits), confirmation is full 128-bit Hamming distance (threshold 7 on pipeline-scale rows). **Verify: distilled verdicts from the same bits.** Ridge heads are fit on the frozen quantized model: - `quality_bits` — linear head over the ±1 bit vector → 0-100 verdict; this is the O(1) triage you can read from a *stored hash* without the text (v1 compressed this to 2 bits; v2 uses all 128); - `quality_proj` — head over the pre-sign projection → finer verdict; - `junk_bits` / `junk_proj` — junk/boilerplate flag with stored operating thresholds. The teacher's quality signal also shaped W itself during training: a straight-through quality head runs alongside the contrastive objective, so the bit planes carry quality structure, not just duplicate geometry. ## The teacher: DeepSeek webapp, batched, ephemeral Protocol (full verbatim prompts in `training/teacher_protocol.md`): 1. fresh chat → 6 corpus rows (same language, same domain; 20 batches carry seeded paraphrase injections) → structured JSON: `{"rows":[{i, quality 0-100, knowledge 0-100, flag}], "dup_pairs":[[i,j]]}` → **chat deleted** → jittered sleep → next chat. 2. 9 batches re-labeled in fresh chats for self-consistency: mean |Δquality| 7.0, median 4.0 (the noise ceiling the student could ever reach). 3. Ambiguous-band pairs (real shingle overlap — the false-merge zone) adjudicated pair-by-pair, 6 pairs per chat, chats deleted after. 4. Teacher dup-pair recall measured against known ground truth: **40/40 injected pairs found (100%)**; 3 organic corpus dups discovered on top. Every completion rides the webapp's own proof-of-work (DeepSeekHashV1 solved by DeepSeek's shipped `pow.wasm`), so the protocol is account-bound, rate-limited (jittered 1.5-4.5 s), and leaves no chat history behind. ## Evaluation Corpus: KIP shard_v25, 794 labeled rows, 15 languages; batch-level 80/20 split (no batch crosses), held-out synthetic paraphrases, teacher-adjudicated eval pairs never seen in training. ### Verify (157 held-out rows; junk n=12) | verifier | Spearman vs teacher | junk AUC | MAE (0-100) | |---|---|---|---| | v1 ensemble (GLM-distilled, cross-teacher) | 0.298 | 0.689 | — | | **v2 hash-only head (from 128 stored bits)** | 0.424 | **0.755** | 11.2 | | **v2 projection head (at signing time)** | **0.543** | 0.701 | 11.0 | v2 beats v1 on every metric *under the new frontier teacher* — the honest cross-teacher comparison, since v1 was distilled against GLM. ### Dedup — easy band (30 positives / 90 negatives) | engine | best F1 | P | R | |---|---|---|---| | **v2 learned bits** | 0.983 | 1.00 | 0.967 | | random-projection SimHash | 0.983 | 1.00 | 0.967 | | plain MinHash-128 | 1.000 | 1.00 | 1.00 | Everyone solves the easy band. The discrimination lives below. ### Dedup — the false-merge zone (14 teacher-adjudicated pairs, MinHash sim 0.07-0.69) | engine | not-dups wrongly merged (of 13) | true dups caught (of 1) | |---|---|---| | **v2 learned bits (threshold 7)** | **0** | **1** | | plain MinHash-128 (its own best threshold) | 2 | 1 | | IDF-weighted Jaccard (v1 verify stage) | 1 | 1 | This is the band where boilerplate and same-entity/template overlap manufactures false merges. The learned bits reject every teacher-rejected hard negative while still catching the true duplicate. ### Throughput (pure stdlib, one core) 45.0 docs/s signing (features + projection + all four heads + band keys). Slower than v1's 812 docs/s because v2 computes a real 128-dim learned projection per document instead of sampling 128 shingles — the price of trainable bits. A numpy-vectorized bulk path exists for large runs; the stdlib path needs nothing but Python. ## Usage Standalone (this repo, nothing to install): ```python from kip_hash import KipHashV2 vh = KipHashV2("kip-hash-v2.json") sig = vh.sign_text("...row text...") sig["quality"] # 0-100 distilled verdict, read from the 128 bits sig["quality_p"] # finer verdict from the projection sig["junk_prob"] # hash-only junk/boilerplate flag sig["hex"] # 128-bit signature, 32 hex chars vh.hamming(sig["bits"], sig2["bits"]) # <= 7 -> near-duplicate kept, dups, report = vh.dedup_stream(texts) # one-pass fuzzy dedup ``` Command line: `python3 kip_hash.py` runs a 5-text demo + selftest (near-dup pair vs different-topic vs same-topic-only, quality ranking, one-pass dedup report). Inside the KIP pipeline: the artifact is consumed through `kip.fullscale.learned_hash` (v2 hook) or directly via this module; `dedup_stream` reuses the survivor policy of the exact-dedup stage, and `quality`/`junk_prob` join the quality spine as features. ## Files | file | what | |---|---| | `kip-hash-v2.json` | the artifact: int8 W (32768×128), 4 heads, config, teacher/provenance/eval blocks (5.9 MB) | | `kip_hash.py` | pure-stdlib inference (features, projection, heads, hamming dedup, selftest) | | `eval/report.json` | every number in this card, machine-readable | | `training/teacher_protocol.md` | the verbatim DeepSeek webapp protocol + prompts | | `training/teacher_labels_deepseek.jsonl` | 844 teacher row verdicts | | `training/teacher_dups.jsonl` | per-chat dup verdicts + injection ground truth | | `training/teacher_pairs_adjudicated.jsonl` | 58 ambiguous-band pair verdicts | | `training/batches_manifest.jsonl` | batch metadata (no corpus text) | | `training/train_v2.py` | the deterministic training script | ## Limitations (honest) - **Evaluation scale.** 157 verify rows, 30/90 dedup pairs, 14 adjudicated hard pairs — directional results, re-measure at 100k+ scale before release-gating on them. - **Teacher noise ceiling.** The teacher itself reproduces within ±7 points (median 4); student metrics are bounded by that. - **OOD text.** Heads are calibrated on ~1600-char pipeline rows; short standalone texts saturate (same caveat as v1), and the corpus contained no `spam`-flagged rows, so the junk head knows boilerplate/incoherent/junk but not classic web spam. - **Thresholds are corpus-scale.** Hamming threshold 7 is selected on pipeline-scale rows; short texts need a looser threshold (~25-30). - **Corpus-shaped bits.** W is trained on a wiki-heavy multilingual sample; other domains should retrain with the shipped script and their own teacher labels (the protocol is fully documented). - **Lexical, not semantic.** Paraphrase-level recall is bounded by n-gram overlap; this is not an embedding space, and cross-language translation dups are intentionally NOT merged. - **Throughput.** 45 docs/s stdlib vs v1's 812 — trainable bits cost compute; use the numpy bulk path for >1M-row runs. - The DeepSeek teacher inherits that model's biases; prompts, labels, verdicts and adjudications all ship for auditability. ## Provenance - Built 2026-11-18/19 by the KIP pipeline ($0 compute; teacher channel = the user's own DeepSeek webapp session, ephemeral chats). - Corpus: KIP shard_v25 — wikipedia dumps, wikidata, wiktionary, eur-lex, openstax, RFCs (permissive sources only). - Artifact policy: weights + metadata only, **no corpus text**. - License: Apache-2.0. ## Citations - Penedo et al., 2024, [The FineWeb Datasets](https://arxiv.org/abs/2406.17557) — LLM-annotation → small-classifier distillation for data quality (FineWeb-Edu). - DeepSeek-AI, 2024/2025, DeepSeek V2/V3 technical reports — the frontier teacher family; webapp annotation protocol documented in `training/teacher_protocol.md`. - Manku, Jain & Das Sarma, 2007, [Detecting Near-Duplicates for Web Crawling](https://research.google.com/pubs/archive/33026.pdf) (WWW 2007) — simhash Hamming-space ancestry. - Wang et al., 2016, [A Survey on Learning to Hash](https://arxiv.org/abs/1606.00185) — supervised bits vs random projections. - Ioffe, 2010, [Improved Consistent Sampling, Weighted MinHash and L1 Sketching](https://dblp.org/rec/conf/icdm/Ioffe10.html) — v1's sampling rule (superseded here by the learned projection). - Orgad et al., 2025, [LLMs Know More Than They Show](https://arxiv.org/abs/2410.02707); Azaria & Mitchell, 2023, [The Internal State of an LLM Knows When It's Lying](https://arxiv.org/abs/2304.13734) — probing the teacher's verdicts. - DataComp-LM (arXiv:2406.11794); Nemotron-CC (arXiv:2412.02595) — model-based filtering context.