Commit History
-
Round 3b prep: control-validation driver (plain HF transformers, no AirLLM)
GENOMA LABS / research +1: Claude Opus 4.7 (1M context)scripts/run_control_niah.py loads any HF model via transformers, wraps model.generate() as the (prompt -> response) callable, runs the NIAH harness. Defaults to Qwen2.5-3B-Instruct (~6 GB). Smaller variants supported via --model. Sweeps a list of context lengths at one budget level (no eviction). Sanity check at end: if smallest-context accuracy < 50%, prints a clear warning that either the harness is broken or the model fundamentally can't NIAH. Otherwise PASS. This proves the harness end-to-end before we point it at the AirLLM Kimi path (where any harness bug would be expensive to debug due to layer-stream load times). When GS or NW gets GPU headroom, fire this for ~10 min total to land the validation cell. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>754890f -
Round 4 driver: AirLLM Kimi K2.6 + install_kv_eviction + NIAH sweep
GENOMA LABS / research +1: Claude Opus 4.7 (1M context)scripts/run_kimi_ruler_sweep.py wires up the full pipeline: - load_kimi_via_airllm: construct AirLLMKimi from snapshot + saved shards - install_eviction / remove_eviction: budget=0 -> baseline; otherwise apply H2O patch via the ported install_kv_eviction - make_generate_fn: wrap AirLLM model.generate as the (prompt -> response) callable the harness expects - run_sweep: iterate context_lengths x budgets, write per-cell CSV plus aggregate summary, pretty-print result matrix Defaults: 4 context lengths, baseline + 2 budgets, 25 trials per cell. Dry-run mode (--dry-run) validates wiring without loading the model. Pluggability of the NIAH harness validated end-to-end at the code level: the same generate_fn callable interface used in the self-test (oracle mocks) plugs straight into AirLLM. Ready to fire as soon as Round 2 split lands at /home/kali/airllm_kimi_shards. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>dea03e1 -
Round 3a: RULER NIAH single-needle harness (logic validated)
GENOMA LABS / research +1: Claude Opus 4.7 (1M context)scripts/niah_harness.py implements the canonical RULER NIAH-1 task as a pluggable harness for the upcoming H2O eviction quality sweep: - NIAHGenerator: builds (haystack, needle, question) at a target context length in chars, with position-fraction control for depth-coverage sweeps. Uses 20 deterministic filler sentences (PG-19-style English structure) and 12-char alphanumeric magic-string needles. - NIAHScorer: case-insensitive exact-match scoring on the magic string. - run_niah_cell: drives N trials at one context length, calls a pluggable generate_fn(prompt) -> response callable. Same harness will drive HF transformers + ollama + AirLLM Kimi backends without modification. - write_cell_csv: per-trial output for analysis. Self-test (no model required, runs in <1s): - oracle_correct mock: 100% accuracy (25/25) - oracle_wrong mock: 0% accuracy (0/25) - oracle_partial 70% mock: 72.5% accuracy (145/200, within expected band) - Per-trial CSV: 200 rows written cleanly Ready for …5ea05a3
-
Day 1 of 7-day Kimi K2.6 + RULER 128K sprint: transformers 5.x port + real-weights install validation
GENOMA LABS / research +1: Claude Opus 4.7 (1M context)Patch ported to dual-API (transformers 4.x + 5.x): - _maybe_evict reads / writes both DynamicCache.key_cache[i] (4.x) and DynamicCache.layers[i].keys / .values (5.x), detected by hasattr. - _make_evicting_forward accepts the union of 4.x and 5.x DeepseekV3Attention signatures via *args / **kwargs spreading. Calls original_forward positionally to avoid hidden_states= keyword collision under positional spread. - Defensive 2-tuple / 3-tuple unpacking of forward return. Validation: - B1 mock-cache multi-step: still PASSes on the ported patch. 500 steps x 4 layers, 1824 eviction events, cache stabilizes at exactly n_sink + budget + n_recent, 1029 steps/sec on CPU. (results/validate_eviction_random_init.csv) - New scripts/validate_install_on_real_kimi_layer.py exercises install_kv_eviction against a real DeepseekV3Attention layer instantiated with actual Kimi K2.6 layer-0 weights from the published checkpoint. Patched forward executes cleanly on TITAN RTX in 0.09s for seq_…95298f5 -
B3a-pivot: real Kimi K2.6 weights demo - eviction policy validated on actual MLA attention distribution
GENOMA LABS / researchLoaded all 7 attention weights (q_a_proj, q_b_proj, q_a_layernorm, kv_a_proj_with_mqa, kv_b_proj, kv_a_layernorm, o_proj) of Kimi K2.6 layer 0 from the published checkpoint, instantiated a transformers DeepseekV3Attention module with the canonical Kimi K2.6 config (61L 64H qk_dim=192 v_dim=128 kv_lora_rank=512), ran one full-prefix forward over 256 synthetic tokens on TITAN RTX in 0.07s, captured the real attention distribution, and applied the H2O heavy-hitter eviction policy. Result: H2O policy keeps heavy-hitters scoring 3.66x higher on average than the tokens it evicts (kept-mean 141.08 vs evicted-mean 38.58). Score range on real Kimi attention: 0.336 to 350.870 (1000x spread, std ~= mean = 64). This is concrete validation that the eviction policy makes sensible decisions on real frontier-MoE MLA attention distributions, not just on synthetic / random data as in the multi-step validation notebook. Adds: - scripts/kimi_layer_eviction_demo.py single-forward Kimi demo - noteboo…aac6c2e -
B1 validation: multi-step eviction test + transformers compatibility note
GENOMA LABS / researchAdds: - scripts/validate_eviction_random_init.py: 1000-step mock-cache validation exercising the H2O eviction logic across 4 layers with canonical DeepseekV3 cache dimensions (qk_dim=192, v_dim=128, FP16). - notebooks/02_validation_results.md: documented results, interpretation, reproduction instructions. - results/validate_eviction_random_init.csv: per-step cache size + eviction events, 1000 rows. Validation result: PASS. Cache stabilizes at exactly n_sink+budget+n_recent, 3664 eviction events triggered correctly across 916 post-cap steps, 913 steps/sec on CPU. Patch fixes (uncovered by attempting end-to-end with transformers 5.7.0): - Bound forward method called without re-passing self. - Defensive unpacking of original_forward return value (handles 2-tuple modern transformers and 3-tuple older versions). - Return arity matches what original_forward returned, so calling decoder layer unpacks correctly. Roadmap updates: - Multi-step validation marked complete. - API por…1ba26d6