Instructions to use belumind/sbd-sql-0.5b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use belumind/sbd-sql-0.5b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="belumind/sbd-sql-0.5b", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("belumind/sbd-sql-0.5b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("belumind/sbd-sql-0.5b", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use belumind/sbd-sql-0.5b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "belumind/sbd-sql-0.5b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "belumind/sbd-sql-0.5b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/belumind/sbd-sql-0.5b

SGLang

How to use belumind/sbd-sql-0.5b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "belumind/sbd-sql-0.5b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "belumind/sbd-sql-0.5b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "belumind/sbd-sql-0.5b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "belumind/sbd-sql-0.5b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use belumind/sbd-sql-0.5b with Docker Model Runner:
```
docker model run hf.co/belumind/sbd-sql-0.5b
```

sbd-sql-0.5b / README.md

johnathan2023

correct the stated cause: loss weighting explains ~14%, not most of it

c54c307 verified about 2 months ago

preview code

Raw History Blame Contribute Delete

12.6 kB

---
license: apache-2.0
language:
  - en
base_model: Qwen/Qwen2.5-0.5B
datasets:
  - gretelai/synthetic_text_to_sql
tags:
  - speculative-decoding
  - set-block-decoding
  - text-to-sql
  - lossless-acceleration
  - inference-acceleration
pipeline_tag: text-generation
library_name: transformers
---
# Belumind SBD-SQL 0.5B

A Set Block Decoding accelerator for SQL text. One 0.5B model both writes the next
token and predicts a block of tokens past the frontier; the decoder draws proposals
from three sources and verifies every one of them exactly. No draft model, no second
checkpoint, no extra forward pass spent on verification.

**1.87 tokens per forward pass on held-out SQL documents** — n=20, se 0.12, τ=0.60,
block 8 — **byte-identical to greedy decoding** at every τ tested, in both bf16 and
fp32. End to end that is **1.58×** the throughput of `model.generate(use_cache=True)`
on the same weights and the same GPU. That speed has a measured cost in
generation quality against the base model — see **What the fine-tune costs** below
before choosing these weights over your own.

---

## Quick start

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "belumind/sbd-sql-0.5b", trust_remote_code=True, dtype=torch.bfloat16
).cuda().eval()
tok = AutoTokenizer.from_pretrained("belumind/sbd-sql-0.5b")
prompt = """SELECT customer_id, SUM(total) AS revenue
FROM orders
WHERE status = 'shipped'
GROUP BY customer_id
ORDER BY revenue DESC;
SELECT customer_id, COUNT(*) AS n_orders
FROM orders
"""
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
out, stats = model.sbd_generate(
    input_ids=ids,
    max_new_tokens=96,
    block_size=8,
    threshold=0.6,                      # τ: a throughput knob, not a quality knob
    sources=("ntp", "matp", "ngram"),
    return_stats=True,
)
print(tok.decode(out[0, ids.shape[1]:]))
print(stats["tokens_per_forward"], stats["accepted"])
```

`sources=("ntp",)` turns speculation off. The decoder then behaves like plain greedy
decoding — 0.99 tok/fwd with ~0 rollbacks — so the machinery itself costs nothing.

This model continues SQL text. It is not an instruction-tuned text-to-SQL assistant:
give it SQL and it keeps writing SQL.

**[Step through a real run →](https://huggingface.co/spaces/belumind/sbd-sql-decoder)**

---

## Results

Prompt = the first 128 tokens of a document, 96 new tokens, batch 1, bf16, block 8,
A100-SXM4-80GB. Held-out documents from `gretelai/synthetic_text_to_sql` that were not
used in training.

| Proposal sources | τ | tokens/forward | rollbacks/seq | from MATP | from n-gram |
|---|---|---|---|---|---|
| NTP only (speculation off) | — | 0.99 | 0.1 | 0% | 0% |
| n-gram only | — | 1.50 ± 0.17 | 13.1 | 0% | 26% |
| MATP only | 0.90 | 1.31 ± 0.02 | 0.1 | 24% | 0% |
| MATP only | 0.50 | 1.71 ± 0.04 | 7.8 | 42% | 0% |
| **MATP + n-gram** | **0.60** | **1.87 ± 0.12** | 10.8 | 30% | 15% |

**The sources are complementary and neither dominates.** MATP alone reaches 1.71,
n-gram alone 1.50, together 1.87. Dropping either costs more than it saves.

**τ buys throughput, not quality.** Output agreement with the reference path is 100.0%
at every τ tested. Loosening τ changes how often the decoder rolls back, never whether
the result is right.

### Corpus sensitivity

The same decoder, same weights, same τ, measured on a second set of SQL emitted by a
template generator:

| Proposal sources | τ | tokens/forward | rollbacks/seq | from MATP | from n-gram |
|---|---|---|---|---|---|
| MATP only | 0.50 | 1.64 ± 0.05 | 8.6 | 38% | 0% |
| **MATP + n-gram** | **0.60** | **2.28 ± 0.12** | 10.3 | 22% | 33% |

Template-generated SQL repeats itself more, the n-gram source finds more to copy, and
throughput rises from 1.87 to 2.28 — a 22% swing from the corpus alone, larger than
anything τ does. Quote a tokens-per-forward number with the corpus attached; the
headline above is the real-document one.

### What the fine-tune adds

The base `Qwen/Qwen2.5-0.5B`, run through the identical decoder, contributes **nothing
at any of the 7 positions past the frontier**: MATP-only scores 0.99 tok/fwd, exactly
the no-speculation baseline, with 0% of accepted tokens coming from the head. The
masked-block capability is created by fine-tuning, not improved by it.

The base model does reach 3.86 tok/fwd on the same documents once n-gram proposals are
enabled — because it falls into repetition, and a copy-based proposer predicts a loop
perfectly. Tokens-per-forward rewards repetition, so it is read next to the generated
text, never alone.

### What the fine-tune costs

Speed is not free here. Measured against the base model on the same held-out
documents, paired at the token level with standard errors clustered by document
(40 documents, 20,440 tokens):

| | base | this checkpoint | Δ |
|---|---|---|---|
| NLL, nats/token | 0.815 | 1.220 | **+0.405** (95% CI +0.397, +0.413) |
| next-token accuracy | 79.3% | 71.0% | −8.4 points |

The distribution matters more than the mean. The median token barely moves
(+0.032); the damage is in the tail — p90 +1.61, p95 +2.40. This checkpoint
matches the base model on easy tokens and is substantially worse on hard ones.
It is also worst at the start of a document (+0.55 over the first tenth, +0.21
over the last), recovering as context accumulates.

The loss weighting is only a small part of it. Retraining from the same base with
the next-token term at weight 1.0 instead of 0.5, changing nothing else, moves the
gap from +0.405 to +0.347 - it recovers about 14%. The rest is the narrowness of
the fine-tune itself: 1500 steps over a small two-domain corpus is enough to shift
the model off the base prior, and no reweighting of that objective buys it back.
Closing the remaining gap would need general data mixed into training, which is a
different exercise from a five-minute fine-tune.

**What follows from this.** If you want the acceleration *and* your own model's
output quality, run `modeling_sbd.py` on your own checkpoint — the decoder does
not depend on these weights, and the NTP and n-gram sources need no fine-tuning
at all. Use these weights only when you want the MATP source specifically and
have measured that the quality trade is acceptable for your task.

---

## How it works

Three proposal sources feed one verifier.

1. **NTP at the frontier** — always valid, always available. Guarantees at least one
   token per forward, so the decoder can never fall below AR in tokens per forward.
2. **MATP over a trailing masked block** — a block of 8 mask tokens is appended after
   the frontier every step and **slides with it**, never aligned to a fixed grid. That
   keeps the last committed position purely causal, so a valid NTP prediction always
   exists. Grid-aligned blocks force an unbacked guess whenever the frontier lands
   inside a partially filled block, worth roughly 1.3 tok/fwd on the set where the two
   were compared.
3. **N-gram lookup over the prompt** — free, deterministic, and strongest exactly where
   MATP is weakest: identifiers echoed from earlier in the same document.

**Verification is folded into the next proposal forward**, which already computes true
conditionals for every committed position. It costs no extra forward pass — only
accepted length.

**Training.** Prefix-causal attention with exactly one bidirectional block at the tail.
This is both the definition of SBD and what keeps prefix KV caching valid; block-causal
attention across the whole sequence instead corrupts frontier NTP (81–88% correct) and
collapses decoding to 0.49 tok/fwd. 1500 steps at batch 12 x 512 tokens, sampling each batch 50/50 from SQL and JSON
documents (`gretelai/synthetic_text_to_sql` and a JSON-schema corpus), warm-started from
the base model, roughly 5 minutes on one A100-SXM4-80GB. The SQL half is what this
checkpoint is published for; the JSON half is training history, not a supported domain.

---

## Lossless output

The verifier is exact by construction: a proposed token is committed only if it equals
the argmax of the true conditional, recomputed by the next forward pass. Measured
against the NTP-only path of the same function, on held-out documents (n=8, 96 tokens
each):

| arithmetic | τ | agreement |
|---|---|---|
| bf16 | 0.60 | 100.0% |
| bf16 | 0.90 | 100.0% |
| fp32 | 0.60 | 100.0% |

Two things to know if you are validating a speculative decoder of your own.

**Compare against the same function.** Comparing a speculative decoder to `generate()`
measures the gap between two implementations, not the correctness of the verifier.
Every number above uses `sbd_generate` under `sources=("ntp",)` as the reference.

**An agreement rate below 100% is usually arithmetic, not speculation.** The same
weights along two different forward-shape schedules can produce different greedy
sequences in bf16, and a speculative run and its reference have different schedules by
construction. The tell: a match rate that does not move when you tighten τ is a
numerical-path problem. Real speculation error moves with τ.

---

## Scope

- **SQL text continuation.** JSON and Python were evaluated and are not supported by
  this checkpoint.
- **Batch size 1.** Speculative gains shrink as batching moves decoding from
  memory-bound toward compute-bound.
- **The wall-clock baseline is `transformers`.** The reference decoder keeps no KV cache
  — every step recomputes the whole sequence — and still finishes 96 tokens in
  **1.65 s against 2.60 s** for `model.generate(use_cache=True)`, same weights, same
  A100, batch 1: **1.58×**. It has not been benchmarked against vLLM, TensorRT-LLM or
  another optimised server, so that is the comparison the number refers to.
- **Prefix caching is available and unimplemented.** The sliding-block design keeps the
  prefix purely causal, so a cached implementation is valid; this reference one does not
  do it yet.
- **Headroom in the head.** MATP reaches 1.73 against a prefix-only NTP confidence
  ceiling of 2.66. Whether the rest is a capacity limit or a training-steps limit is
  untested at 0.5B.
- **Not yet combined with grammar constraints.** SQL grammars are weaker than JSON
  grammars, so the overlap should be smaller than for JSON. If you already run
  constrained decoding, treat this as an addition to it, not a replacement.

---

## Interactive demo

**[belumind/sbd-sql-decoder](https://huggingface.co/spaces/belumind/sbd-sql-decoder)**
steps through recorded runs of this exact decoder: every token coloured by the source
whose proposal was accepted, every rollback shown as the verifier refusing a guess. The
traces are real `sbd_generate` runs logged forward pass by forward pass, from a tracer
checked token-for-token against the shipped decoder.

`demo/demo_colab.ipynb` runs the Gradio app live on a Colab GPU, and the same folder
deploys unchanged to a GPU Space.

## Reproducing

`modeling_sbd.py` is the entire decoder — mask construction, n-gram proposer, verifier,
accounting — and every number above is `sbd_generate(..., return_stats=True)` averaged
over the stated n, with the reference produced by the same function under
`sources=("ntp",)`.

## Credit

Set Block Decoding is not our method. It was introduced in *Set Block Decoding is a
Language Model Inference Accelerator* by Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel
Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer and Yaron
Lipman, 2025 — [doi.org/10.48550/arXiv.2509.04185](https://doi.org/10.48550/arXiv.2509.04185).
This repository is an independent implementation of it at 0.5B on SQL text, with a
third proposal source added, and is not affiliated with or endorsed by the authors.

```bibtex
@article{gat2025setblockdecoding,
  title  = {Set Block Decoding is a Language Model Inference Accelerator},
  author = {Gat, Itai and Ben-Hamu, Heli and Havasi, Marton and Haziza, Daniel
            and Reizenstein, Jeremy and Synnaeve, Gabriel and Lopez-Paz, David
            and Karrer, Brian and Lipman, Yaron},
  year   = {2025},
  doi    = {10.48550/arXiv.2509.04185}
}
```

There is no paper for this checkpoint. To refer to it, cite the repository:

```bibtex
@misc{belumind2026sbdsql,
  title        = {Belumind SBD-SQL 0.5B},
  author       = {{Belumind}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/belumind/sbd-sql-0.5b}
}
```

---

Built by [Belumind](https://huggingface.co/belumind).

	---
	license: apache-2.0
	language:
	- en
	base_model: Qwen/Qwen2.5-0.5B
	datasets:
	- gretelai/synthetic_text_to_sql
	tags:
	- speculative-decoding
	- set-block-decoding
	- text-to-sql
	- lossless-acceleration
	- inference-acceleration
	pipeline_tag: text-generation
	library_name: transformers
	---

	# Belumind SBD-SQL 0.5B

	A Set Block Decoding accelerator for SQL text. One 0.5B model both writes the next
	token and predicts a block of tokens past the frontier; the decoder draws proposals
	from three sources and verifies every one of them exactly. No draft model, no second
	checkpoint, no extra forward pass spent on verification.

	1.87 tokens per forward pass on held-out SQL documents — n=20, se 0.12, τ=0.60,
	block 8 — byte-identical to greedy decoding at every τ tested, in both bf16 and
	fp32. End to end that is 1.58× the throughput of `model.generate(use_cache=True)`
	on the same weights and the same GPU. That speed has a measured cost in
	generation quality against the base model — see What the fine-tune costs below
	before choosing these weights over your own.

	---

	## Quick start

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model = AutoModelForCausalLM.from_pretrained(
	"belumind/sbd-sql-0.5b", trust_remote_code=True, dtype=torch.bfloat16
	).cuda().eval()
	tok = AutoTokenizer.from_pretrained("belumind/sbd-sql-0.5b")

	prompt = """SELECT customer_id, SUM(total) AS revenue
	FROM orders
	WHERE status = 'shipped'
	GROUP BY customer_id
	ORDER BY revenue DESC;

	SELECT customer_id, COUNT(*) AS n_orders
	FROM orders
	"""

	ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
	out, stats = model.sbd_generate(
	input_ids=ids,
	max_new_tokens=96,
	block_size=8,
	threshold=0.6, # τ: a throughput knob, not a quality knob
	sources=("ntp", "matp", "ngram"),
	return_stats=True,
	)
	print(tok.decode(out[0, ids.shape[1]:]))
	print(stats["tokens_per_forward"], stats["accepted"])
	```

	`sources=("ntp",)` turns speculation off. The decoder then behaves like plain greedy
	decoding — 0.99 tok/fwd with ~0 rollbacks — so the machinery itself costs nothing.

	This model continues SQL text. It is not an instruction-tuned text-to-SQL assistant:
	give it SQL and it keeps writing SQL.

	[Step through a real run →](https://huggingface.co/spaces/belumind/sbd-sql-decoder)

	---

	## Results

	Prompt = the first 128 tokens of a document, 96 new tokens, batch 1, bf16, block 8,
	A100-SXM4-80GB. Held-out documents from `gretelai/synthetic_text_to_sql` that were not
	used in training.

	\| Proposal sources \| τ \| tokens/forward \| rollbacks/seq \| from MATP \| from n-gram \|
	\|---\|---\|---\|---\|---\|---\|
	\| NTP only (speculation off) \| — \| 0.99 \| 0.1 \| 0% \| 0% \|
	\| n-gram only \| — \| 1.50 ± 0.17 \| 13.1 \| 0% \| 26% \|
	\| MATP only \| 0.90 \| 1.31 ± 0.02 \| 0.1 \| 24% \| 0% \|
	\| MATP only \| 0.50 \| 1.71 ± 0.04 \| 7.8 \| 42% \| 0% \|
	\| MATP + n-gram \| 0.60 \| 1.87 ± 0.12 \| 10.8 \| 30% \| 15% \|

	The sources are complementary and neither dominates. MATP alone reaches 1.71,
	n-gram alone 1.50, together 1.87. Dropping either costs more than it saves.

	τ buys throughput, not quality. Output agreement with the reference path is 100.0%
	at every τ tested. Loosening τ changes how often the decoder rolls back, never whether
	the result is right.

	### Corpus sensitivity

	The same decoder, same weights, same τ, measured on a second set of SQL emitted by a
	template generator:

	\| Proposal sources \| τ \| tokens/forward \| rollbacks/seq \| from MATP \| from n-gram \|
	\|---\|---\|---\|---\|---\|---\|
	\| MATP only \| 0.50 \| 1.64 ± 0.05 \| 8.6 \| 38% \| 0% \|
	\| MATP + n-gram \| 0.60 \| 2.28 ± 0.12 \| 10.3 \| 22% \| 33% \|

	Template-generated SQL repeats itself more, the n-gram source finds more to copy, and
	throughput rises from 1.87 to 2.28 — a 22% swing from the corpus alone, larger than
	anything τ does. Quote a tokens-per-forward number with the corpus attached; the
	headline above is the real-document one.

	### What the fine-tune adds

	The base `Qwen/Qwen2.5-0.5B`, run through the identical decoder, contributes **nothing
	at any of the 7 positions past the frontier**: MATP-only scores 0.99 tok/fwd, exactly
	the no-speculation baseline, with 0% of accepted tokens coming from the head. The
	masked-block capability is created by fine-tuning, not improved by it.

	The base model does reach 3.86 tok/fwd on the same documents once n-gram proposals are
	enabled — because it falls into repetition, and a copy-based proposer predicts a loop
	perfectly. Tokens-per-forward rewards repetition, so it is read next to the generated
	text, never alone.

	### What the fine-tune costs

	Speed is not free here. Measured against the base model on the same held-out
	documents, paired at the token level with standard errors clustered by document
	(40 documents, 20,440 tokens):

	\| \| base \| this checkpoint \| Δ \|
	\|---\|---\|---\|---\|
	\| NLL, nats/token \| 0.815 \| 1.220 \| +0.405 (95% CI +0.397, +0.413) \|
	\| next-token accuracy \| 79.3% \| 71.0% \| −8.4 points \|

	The distribution matters more than the mean. The median token barely moves
	(+0.032); the damage is in the tail — p90 +1.61, p95 +2.40. This checkpoint
	matches the base model on easy tokens and is substantially worse on hard ones.
	It is also worst at the start of a document (+0.55 over the first tenth, +0.21
	over the last), recovering as context accumulates.

	The loss weighting is only a small part of it. Retraining from the same base with
	the next-token term at weight 1.0 instead of 0.5, changing nothing else, moves the
	gap from +0.405 to +0.347 - it recovers about 14%. The rest is the narrowness of
	the fine-tune itself: 1500 steps over a small two-domain corpus is enough to shift
	the model off the base prior, and no reweighting of that objective buys it back.
	Closing the remaining gap would need general data mixed into training, which is a
	different exercise from a five-minute fine-tune.

	What follows from this. If you want the acceleration and your own model's
	output quality, run `modeling_sbd.py` on your own checkpoint — the decoder does
	not depend on these weights, and the NTP and n-gram sources need no fine-tuning
	at all. Use these weights only when you want the MATP source specifically and
	have measured that the quality trade is acceptable for your task.

	---

	## How it works

	Three proposal sources feed one verifier.

	1. NTP at the frontier — always valid, always available. Guarantees at least one
	token per forward, so the decoder can never fall below AR in tokens per forward.
	2. MATP over a trailing masked block — a block of 8 mask tokens is appended after
	the frontier every step and slides with it, never aligned to a fixed grid. That
	keeps the last committed position purely causal, so a valid NTP prediction always
	exists. Grid-aligned blocks force an unbacked guess whenever the frontier lands
	inside a partially filled block, worth roughly 1.3 tok/fwd on the set where the two
	were compared.
	3. N-gram lookup over the prompt — free, deterministic, and strongest exactly where
	MATP is weakest: identifiers echoed from earlier in the same document.

	Verification is folded into the next proposal forward, which already computes true
	conditionals for every committed position. It costs no extra forward pass — only
	accepted length.

	Training. Prefix-causal attention with exactly one bidirectional block at the tail.
	This is both the definition of SBD and what keeps prefix KV caching valid; block-causal
	attention across the whole sequence instead corrupts frontier NTP (81–88% correct) and
	collapses decoding to 0.49 tok/fwd. 1500 steps at batch 12 x 512 tokens, sampling each batch 50/50 from SQL and JSON
	documents (`gretelai/synthetic_text_to_sql` and a JSON-schema corpus), warm-started from
	the base model, roughly 5 minutes on one A100-SXM4-80GB. The SQL half is what this
	checkpoint is published for; the JSON half is training history, not a supported domain.

	---

	## Lossless output

	The verifier is exact by construction: a proposed token is committed only if it equals
	the argmax of the true conditional, recomputed by the next forward pass. Measured
	against the NTP-only path of the same function, on held-out documents (n=8, 96 tokens
	each):

	\| arithmetic \| τ \| agreement \|
	\|---\|---\|---\|
	\| bf16 \| 0.60 \| 100.0% \|
	\| bf16 \| 0.90 \| 100.0% \|
	\| fp32 \| 0.60 \| 100.0% \|

	Two things to know if you are validating a speculative decoder of your own.

	Compare against the same function. Comparing a speculative decoder to `generate()`
	measures the gap between two implementations, not the correctness of the verifier.
	Every number above uses `sbd_generate` under `sources=("ntp",)` as the reference.

	An agreement rate below 100% is usually arithmetic, not speculation. The same
	weights along two different forward-shape schedules can produce different greedy
	sequences in bf16, and a speculative run and its reference have different schedules by
	construction. The tell: a match rate that does not move when you tighten τ is a
	numerical-path problem. Real speculation error moves with τ.

	---

	## Scope

	- SQL text continuation. JSON and Python were evaluated and are not supported by
	this checkpoint.
	- Batch size 1. Speculative gains shrink as batching moves decoding from
	memory-bound toward compute-bound.
	- The wall-clock baseline is `transformers`. The reference decoder keeps no KV cache
	— every step recomputes the whole sequence — and still finishes 96 tokens in
	1.65 s against 2.60 s for `model.generate(use_cache=True)`, same weights, same
	A100, batch 1: 1.58×. It has not been benchmarked against vLLM, TensorRT-LLM or
	another optimised server, so that is the comparison the number refers to.
	- Prefix caching is available and unimplemented. The sliding-block design keeps the
	prefix purely causal, so a cached implementation is valid; this reference one does not
	do it yet.
	- Headroom in the head. MATP reaches 1.73 against a prefix-only NTP confidence
	ceiling of 2.66. Whether the rest is a capacity limit or a training-steps limit is
	untested at 0.5B.
	- Not yet combined with grammar constraints. SQL grammars are weaker than JSON
	grammars, so the overlap should be smaller than for JSON. If you already run
	constrained decoding, treat this as an addition to it, not a replacement.

	---

	## Interactive demo

	[belumind/sbd-sql-decoder](https://huggingface.co/spaces/belumind/sbd-sql-decoder)
	steps through recorded runs of this exact decoder: every token coloured by the source
	whose proposal was accepted, every rollback shown as the verifier refusing a guess. The
	traces are real `sbd_generate` runs logged forward pass by forward pass, from a tracer
	checked token-for-token against the shipped decoder.

	`demo/demo_colab.ipynb` runs the Gradio app live on a Colab GPU, and the same folder
	deploys unchanged to a GPU Space.

	## Reproducing

	`modeling_sbd.py` is the entire decoder — mask construction, n-gram proposer, verifier,
	accounting — and every number above is `sbd_generate(..., return_stats=True)` averaged
	over the stated n, with the reference produced by the same function under
	`sources=("ntp",)`.

	## Credit

	Set Block Decoding is not our method. It was introduced in *Set Block Decoding is a
	Language Model Inference Accelerator* by Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel
	Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer and Yaron
	Lipman, 2025 — [doi.org/10.48550/arXiv.2509.04185](https://doi.org/10.48550/arXiv.2509.04185).
	This repository is an independent implementation of it at 0.5B on SQL text, with a
	third proposal source added, and is not affiliated with or endorsed by the authors.

	```bibtex
	@article{gat2025setblockdecoding,
	title = {Set Block Decoding is a Language Model Inference Accelerator},
	author = {Gat, Itai and Ben-Hamu, Heli and Havasi, Marton and Haziza, Daniel
	and Reizenstein, Jeremy and Synnaeve, Gabriel and Lopez-Paz, David
	and Karrer, Brian and Lipman, Yaron},
	year = {2025},
	doi = {10.48550/arXiv.2509.04185}
	}
	```

	There is no paper for this checkpoint. To refer to it, cite the repository:

	```bibtex
	@misc{belumind2026sbdsql,
	title = {Belumind SBD-SQL 0.5B},
	author = {{Belumind}},
	year = {2026},
	howpublished = {Hugging Face model repository},
	url = {https://huggingface.co/belumind/sbd-sql-0.5b}
	}
	```

	---

	Built by [Belumind](https://huggingface.co/belumind).