|
|
| --- |
| pipeline_tag: text-generation |
| license: apache-2.0 |
| base_model: |
| - tsinghua-sigs-robot-lab/VeriLoop-E2 |
| language: |
| - en |
| - zh |
| tags: |
| - gguf |
| - llama.cpp |
| - mtp |
| - speculative-decoding |
| - dflash2 |
| - veriloop |
| - apache-2.0 |
| --- |
| |
| This is an experimental research artifact, it does actually run but I can't verify acceptance rates because my 12gb of vram apparently doesnt cut it because kv caches and all that |
|
|
| But yeah it does work- this was trained on a laptop gpu which is kinda crazy ngl |
|
|
| if you want to try it- use [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) |
|
|
| heres the self eval: |
|
|
| ``` bash |
| python -m toffee xspark eval --head xspark_out/dflash2/dflash2_head.pt \ |
| --target-model ./VeriLoop-E2 --data-dir xspark_data/harvest --anchors 512 |
| {"name": "NVIDIA GeForce RTX 4080 Laptop GPU", "vram_gb": 11.67608642578125, "event": "gpu_detected", "level": "info", "timestamp": "2026-11-06T14:22:03.355667Z"} |
| { |
| "variant": "dflash2", |
| "n_blocks": 512, |
| "block_size": 8, |
| "accept_len": 1.171875, |
| "oracle_len": 2.068359375, |
| "accept@0": 0.15625, |
| "recall@0": 0.49609375, |
| "accept@1": 0.015625, |
| "recall@1": 0.26171875, |
| "accept@2": 0.0, |
| "recall@2": 0.13671875, |
| "accept@3": 0.0, |
| "recall@3": 0.078125, |
| "accept@4": 0.0, |
| "recall@4": 0.048828125, |
| "accept@5": 0.0, |
| "recall@5": 0.029296875, |
| "accept@6": 0.0, |
| "recall@6": 0.017578125 |
| } |
| ``` |
|
|
| What the numbas say: |
|
|
| | metric | value | plain english | |
| |---|---|---| |
| | `recall@0` | 254/512 = **49.6%** | the true next token is *in your top-16* half the time — the head **knows** it | |
| | `accept@0` | 80/512 = **15.6%** | the walk *picks* it 16% of the time | |
| | gap | **174 anchors** | truth was sitting in the candidate list and the selector walked past it | |
| | `oracle_len` | **2.068** | accept length if the walk always picked the truth when available — the ceiling **of this head as-is** | |
| | `accept_len` | **1.172** | you're capturing (1.17−1)/(2.07−1) ≈ **16% of the available headroom** | |
|
|
|
|
| if you want to recreate this abomination use my [llm toolkit](https://github.com/damien1141/toffee) |
|
|
