README.md

---
license: other
language:
- en
tags:
- lucebox
- pflash
- prefill-compression
- gguf
- qwen3.5
base_model: Qwen/Qwen3.5-0.8B
---

# pflash-segment-probe

The segment probe of PFlash: it cuts a long prompt into whole structural units before the relevance head ranks them.

PFlash is Lucebox's prefill compression for long prompts. A small drafter, Qwen3.5-0.8B, reads the whole prompt
once, and three small components on its activations decide what the large reader model prefills:
1. [`Lucebox/pflash-segment-probe`](https://huggingface.co/Lucebox/pflash-segment-probe) cuts the prompt into
   whole units (statements and definitions, sentences, records, chat turns, tool calls);
2. [`Lucebox/pflash-request-probe`](https://huggingface.co/Lucebox/pflash-request-probe) finds the request,
   wherever it sits in the prompt;
3. [`Lucebox/pflash-scorer-head`](https://huggingface.co/Lucebox/pflash-scorer-head) ranks the units against that
   request.

The highest-ranked units that fit the budget go to the reader in their original order, and the reader prefills
only those. All three read the same tap, the drafter's block-14 residual (`post_block14_residual_before_block15`),
from the same single pass.

## How it works

- **Architecture**: LayerNorm(1024) → Linear(1024, 256) → GELU → three linear rows, each with a 5-tap convolution
  over its logit sequence plus the residual logit; about 263K parameters, 16 tensors. GGUF architecture
  `segmentprobe`, schema `qwen3_5_0_8b_segment_probe_v2`.
- **Three outputs per token**:
  - **unit**: a structural unit starts here. Units are the finest whole pieces: every statement, declaration,
    definition, comment or decorator block at file or container scope in code; sentences in prose; records, chat
    turns, diff hunks, tool calls and observations. Nothing inside a function body is a unit cut, so a kept unit is
    never a truncated function.
  - **container**: the enclosing region (class, file section, paragraph, session, assistant turn). Exported, not
    used by the served configuration.
  - **sub-unit**: a nested definition inside a body, used only to split units that are too long.
- **Segmentation** (settings in the GGUF): a cut where the unit score passes the threshold 0.8; cuts closer than
  `min_segment` 32 tokens to the previous one merge forward, so small leaves such as doc comments group with what
  follows; a unit longer than `max_segment` 512 tokens is split at line boundaries (`oversize_split` =
  `line_aligned`), preferring sub-unit starts. No parser runs at inference.
- **Why the threshold is asymmetric**: a missed boundary merges two whole units, which the head can still rank; a
  cut inside a unit truncates it, which it cannot repair. 0.8 keeps harmful cuts (inside a function body or
  mid-line) at or below 1 % of cuts on code, tool output, dialogue, records and prose, with boundary precision of
  0.97 or more.

## Files

| file | sha256 | note |
|---|---|---|
| `segment-probe.gguf` | `470b7b0efe3736518dcb4c1ba8eb5b806107007e378a74668c17e26434602e68` | the served file: `max_segment` 512, line-aligned splitting |

## Using it

PFlash runs inside the Lucebox server. Download the three components and the drafter, then set:

```bash
export PFLASH_SCORING_HEAD_GGUF=/path/to/seg-scoring-head.gguf
export PFLASH_SEGMENT_PROBE_GGUF=/path/to/segment-probe.gguf
export PFLASH_REQUEST_PROBE_GGUF=/path/to/request-probe.gguf
export PFLASH_SELECT_SEGMENTS=probe PFLASH_SELECT_SCORE=density PFLASH_SELECT_MODE=budget_only
export PFLASH_SELECT_QUERY_PROBE=1 PFLASH_QUERY_CANDIDATES=1 PFLASH_QUERY_PIN=512
export PFLASH_SELECT_QUERY_TOKENS=128 PFLASH_SELECT_QUERY_HEAD=1
export PFLASH_SELECT_PARAGRAPH_JOIN=0 PFLASH_SELECT_CHUNK_SIZE=256
export PFLASH_ADMISSION=cache PFLASH_ARRIVE=1 PFLASH_CHAT_RECALL=1
export PFLASH_CHAT_REBUILD_RATIO=2 PFLASH_CHAT_REBUILD_MIN_TOKENS=65536
export PFLASH_FALLBACK_SPREAD=0.11 PFLASH_COLD_HISTORY=raw
# server: --prefill-compression auto --prefill-threshold 16384
#         --prefill-curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225
#         --prefill-skip-park --cache-type-k q4_0 --cache-type-v q4_0 --max-ctx 139264
```

- **Drafter** (required, checked by sha256 at load):
  [`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF) `Qwen3.5-0.8B-BF16.gguf`, sha256
  `cedf89af31c9041b601fa58303285bc46d99c51baee1b13f5e919626ca526ee5`. Each component refuses to load against any
  other drafter file.
- PFlash acts when at least 16,384 prompt tokens are new to the server and less than that is already cached. The
  reader then prefills 31.5 % of the prompt at 16K tokens, 27 % at 32K–64K and 22.5 % at 120K.
- When the relevance scores are too flat for the budget, the request is served with full prefill
  (`PFLASH_FALLBACK_SPREAD`). An agent's uncached earlier conversation is never compressed
  (`PFLASH_COLD_HISTORY=raw`).
- Runtime support: [Luce-Org/lucebox PR #746](https://github.com/Luce-Org/lucebox/pull/746).

## Results

Measured with all three components together, reader Qwen3.8-27B IQ4_XS, the configuration above, NVIDIA L40S,
greedy, each benchmark's official metric:

| set | n | full prefill | PFlash |
|---|---:|---:|---:|
| Document QA | 30 | 27 | 27 |
| LongBench v2 | 12 | 6 | 7 |
| NoLiMa | 24 | 14 | 14 |
| InfiniteBench CodeDebug | 30 | 24 | 24 |
| BRIGHT | 28 | 23 | 23 |
| needle in a haystack, 16K–120K | 100 | 100 | 100 |
| **public sets, unique questions** (RULER QA's 15 are inside Document QA) | 224 | 194 | **195** |
| LongCodeQA, questions that need the code | 114 | 57 | 62 |

None of these differences is outside its 95 % interval. The reader prefills a median 29 % of the prompt, and the
first token arrives 2.7× sooner on average (mean 30.5 s → 11.2 s over the public requests; 60.4 s → 14.8 s median
on 82K–98K-token CodeDebug prompts).

## Limits

- Not lossless: a request can still lose something the answer needed.
- Cold agent histories are served with full prefill, so a resumed agent session gets no speed-up.
- "List everything matching X" over many similar units (emails, wiki pages) is the hardest case; the flat-score
  fallback catches most of it.
- Measured serving one request at a time.