README.md

---
license: other
language:
- en
tags:
- lucebox
- pflash
- prefill-compression
- gguf
- qwen3.5
base_model: Qwen/Qwen3.5-0.8B
---

# pflash-request-probe

The request probe of PFlash: it finds which tokens of a long prompt carry the user's request, so the relevance head
ranks the prompt against the right question.

PFlash is Lucebox's prefill compression for long prompts. A small drafter, Qwen3.5-0.8B, reads the whole prompt
once, and three small components on its activations decide what the large reader model prefills:
1. [`Lucebox/pflash-segment-probe`](https://huggingface.co/Lucebox/pflash-segment-probe) cuts the prompt into
   whole units (statements and definitions, sentences, records, chat turns, tool calls);
2. [`Lucebox/pflash-request-probe`](https://huggingface.co/Lucebox/pflash-request-probe) finds the request,
   wherever it sits in the prompt;
3. [`Lucebox/pflash-scorer-head`](https://huggingface.co/Lucebox/pflash-scorer-head) ranks the units against that
   request.

The highest-ranked units that fit the budget go to the reader in their original order, and the reader prefills
only those. All three read the same tap, the drafter's block-14 residual (`post_block14_residual_before_block15`),
from the same single pass.

## How it works

The request is not always at the end of the prompt. In a RAG prompt it can sit above the retrieved passages; in a
chat it can be a short follow-up after a long paste, or a question inside the pasted message itself.

- **Architecture**: LayerNorm(1024) → Linear(1024, 256) → GELU (erf) → Linear(256, 1), one logit per token for
  "this token is part of the current request"; 264,705 parameters. GGUF architecture `requestprobe`, schema
  `qwen3_5_0_8b_request_probe_v1`.
- **Query**: the segment probe's units are intersected with each message of the conversation. A unit's score is
  the highest request logit inside it, and units are ranked against each other, not against an absolute threshold.
  The head's query is the latest user message's top-ranked whole units, so a question inside a long paste is found
  wherever it sits. The query is never empty, and there is no fixed window.
- **Pin**: the latest user message's top-ranked whole units, up to 512 tokens (`PFLASH_QUERY_PIN`), are always
  kept verbatim in the served prompt and charged to the budget.
- **Fail closed**: if the probe, the segment probe or the message mapping is missing, the server refuses the
  configuration instead of silently serving without it.

## Files

| file | sha256 | size |
|---|---|---|
| `request-probe.gguf` | `5915557d4b60f5a08c8ef18fdd7a30cba56c3a24fda1c2b988da4a41313ebf49` | 1,060,192 bytes |

## Using it

PFlash runs inside the Lucebox server. Download the three components and the drafter, then set:

```bash
export PFLASH_SCORING_HEAD_GGUF=/path/to/seg-scoring-head.gguf
export PFLASH_SEGMENT_PROBE_GGUF=/path/to/segment-probe.gguf
export PFLASH_REQUEST_PROBE_GGUF=/path/to/request-probe.gguf
export PFLASH_SELECT_SEGMENTS=probe PFLASH_SELECT_SCORE=density PFLASH_SELECT_MODE=budget_only
export PFLASH_SELECT_QUERY_PROBE=1 PFLASH_QUERY_CANDIDATES=1 PFLASH_QUERY_PIN=512
export PFLASH_SELECT_QUERY_TOKENS=128 PFLASH_SELECT_QUERY_HEAD=1
export PFLASH_SELECT_PARAGRAPH_JOIN=0 PFLASH_SELECT_CHUNK_SIZE=256
export PFLASH_ADMISSION=cache PFLASH_ARRIVE=1 PFLASH_CHAT_RECALL=1
export PFLASH_CHAT_REBUILD_RATIO=2 PFLASH_CHAT_REBUILD_MIN_TOKENS=65536
export PFLASH_FALLBACK_SPREAD=0.11 PFLASH_COLD_HISTORY=raw
# server: --prefill-compression auto --prefill-threshold 16384
#         --prefill-curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225
#         --prefill-skip-park --cache-type-k q4_0 --cache-type-v q4_0 --max-ctx 139264
```

- **Drafter** (required, checked by sha256 at load):
  [`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF) `Qwen3.5-0.8B-BF16.gguf`, sha256
  `cedf89af31c9041b601fa58303285bc46d99c51baee1b13f5e919626ca526ee5`. Each component refuses to load against any
  other drafter file.
- PFlash acts when at least 16,384 prompt tokens are new to the server and less than that is already cached. The
  reader then prefills 31.5 % of the prompt at 16K tokens, 27 % at 32K–64K and 22.5 % at 120K.
- When the relevance scores are too flat for the budget, the request is served with full prefill
  (`PFLASH_FALLBACK_SPREAD`). An agent's uncached earlier conversation is never compressed
  (`PFLASH_COLD_HISTORY=raw`).
- Runtime support: [Luce-Org/lucebox PR #746](https://github.com/Luce-Org/lucebox/pull/746).

## Results

Measured with all three components together, reader Qwen3.8-27B IQ4_XS, the configuration above, NVIDIA L40S,
greedy, each benchmark's official metric:

| set | n | full prefill | PFlash |
|---|---:|---:|---:|
| Document QA | 30 | 27 | 27 |
| LongBench v2 | 12 | 6 | 7 |
| NoLiMa | 24 | 14 | 14 |
| InfiniteBench CodeDebug | 30 | 24 | 24 |
| BRIGHT | 28 | 23 | 23 |
| needle in a haystack, 16K–120K | 100 | 100 | 100 |
| **public sets, unique questions** (RULER QA's 15 are inside Document QA) | 224 | 194 | **195** |
| LongCodeQA, questions that need the code | 114 | 57 | 62 |

None of these differences is outside its 95 % interval. The reader prefills a median 29 % of the prompt, and the
first token arrives 2.7× sooner on average (mean 30.5 s → 11.2 s over the public requests; 60.4 s → 14.8 s median
on 82K–98K-token CodeDebug prompts).

## Limits

- Not lossless: a request can still lose something the answer needed.
- Cold agent histories are served with full prefill, so a resumed agent session gets no speed-up.
- "List everything matching X" over many similar units (emails, wiki pages) is the hardest case; the flat-score
  fallback catches most of it.
- Measured serving one request at a time.