Spaces:

Ruurd
/

llada-8b-instruct

Running on Zero

App Files Files Community

llada-8b-instruct / README.md

Ruurd

Add optional expanded-corpus bigram and exact trigram guidance

b31a17a verified 11 days ago

preview code

Raw History Blame Contribute Delete

15.4 kB

---
title: LLaDA-8B-Instruct
emoji: 🌊
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
python_version: "3.12"
app_file: app.py
pinned: false
license: other
startup_duration_timeout: 1h
short_description: Native LLaDA with diffusion and PMI Viterbi decoding
models:
- GSAI-ML/LLaDA-8B-Instruct
---
# LLaDA-8B-Instruct

Loads the official [GSAI-ML/LLaDA-8B-Instruct](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct) directly in
BF16 on ZeroGPU. Model and tokenizer revision: `08b83a6feb34df1a6011b80c3c00c7563e963b07`.

The interface shares the BYOD Gemma demo's decoding controls: temperature,
top-k, block diffusion, confidence-guided or random retention, PMI Viterbi,
ending delay, early stopping, and the live token visualization. The model
uses its native bidirectional forward pass, chat template, and mask token
126336. This is a comparison using the shared app decoder, not a claim to
reproduce the official benchmark sampler.

PMI uses a separate scorer trained on all 980,000 responses in the existing
training split, retokenized with LLaDA's tokenizer. It covers all 126,464
model output IDs, including output padding beyond the tokenizer vocabulary.
Scoring is PMI, retention confidence is diffusion probability, minimum count
is 1, and clipping is ±2. Edit the constants near the top of `app.py` to
change these defaults. Temperature and top-k are shared controls; bigram
weight is fixed at 0.5. Ending-delay strength remains adjustable.

LLaDA's available strategies are confidence-guided, random, and PMI Viterbi.
The adapter-based autoregressive confidence strategy applies to BYOD models.
The first request may take longer while ZeroGPU allocates hardware.


## Optional repetition-aware PMI Viterbi

Select **PMI Viterbi + Repetition Penalty** to use the block-local transition
score `0.5 * temperature * PMI(a,b) - gamma * C(a,b)`. Gamma defaults to 1.0;
0 reproduces the original PMI decoder. C accumulates earlier expected bigram
occurrences from full-vocabulary diffusion probabilities, updated only after
scoring each position. EOS/EOT pairs are neither penalized nor counted. Retained
tokens remain fixed constraints. Counts reset each iteration and block, without
crossing prompt/block boundaries. There are no additional model forward passes.
The original **Bigram Viterbi** option and generation defaults remain available.


## Adaptive Parallel Decoding

The **Adaptive Parallel Decoding (APD)** checkbox defaults OFF. APD is a
commitment layer after the selected sampling or sequence decoder. Each block
uses a cosine confidence threshold from 0.90 to 0.50, with a 15% warmup. It
commits all qualifying masked positions using their selected token's original
full-vocabulary diffusion probability. Remaining positions stay masked; a
best-confidence fallback makes progress and the final step resolves all masks.
Completed blocks exit early, independently of the convergence-stop checkbox.
Existing EOS/EOT retention delay is respected, with endings retained after
ordinary predictions; whole-answer EOS/EOT stopping still follows Early stopping.
Committed values stay fixed. Optional AR retention scoring and retained-token
revision are bypassed under APD. No additional model forwards, token budgets,
position factors, or grinding-phase detection are used. OFF retains the original
decoding configuration. This implements the requested APD subset, inspired by
[CAI-dLLM](https://arxiv.org/abs/2608.22646), rather than its complete controller.


## Optional sequence sampling and candidate-occurrence counts

Both checkboxes default OFF, preserving existing decoding. **Sample bigram-guided
sequences** replaces the maximum-path decision with exact forward-filter/backward
sampling from exp(total path score) on the same candidate lattice. Diffusion,
PMI, boundaries and any repetition scores are unchanged; no score perturbations
or extra model forwards are used. Seed controls reproducibility. This applies
to either bigram decoding strategy, independently of APD.

**Count every candidate bigram occurrence** applies only with the repetition
penalty strategy. Each earlier adjacent pair of candidate sets containing (a,b)
adds 1 to C(a,b), irrespective of probability. Duplicate fixed-token columns
count once; EOS/EOT pairs are excluded. Counts update after scoring each edge
and reset each block/iteration. Gamma still subtracts gamma*C in log-score units;
this is not a percentage-parameter remapping. Gamma=0 restores unpenalized results
for the selected maximum-path or sampling mode. Ordinary decoding ignores these
checkboxes; the original temperature weighting is unchanged.
## Independent temperature and bigram weight
The LLaDA Space now uses S(y) = sum log p_t(y_t) + w sum PMI - gamma sum C.
**Bigram weight** ranges from 0 to 5, default 1. A weight of 1 gives equal
coefficients, not necessarily equal numerical influence. Weight 0 disables PMI
but does not disable repetition penalties. The previous fixed 0.5*temperature
PMI coefficient described above is superseded for this Space.

**Sample bigram-guided sequences** samples P(y) proportional to exp(S(y)/T).
Temperature controls the entropy of the whole sequence score, including repetition,
without changing the ratios of score coefficients. T=0 selects the maximum path;
positive T is floored to 1e-6 for numerical stability. Maximum-path Viterbi ignores
temperature. Raw full-vocabulary diffusion probabilities still determine expected
repetition counts and retention/APD confidence. Default T is now 1, w is 1,
matching the former T=2 PMI coefficient and sequence distribution. Ordinary
diffusion retains its usual temperature behavior. No extra model forwards.

Unseen pairs have a neutral PMI correction of zero, never a forbidden transition.
Conditional scoring instead uses smoothed finite probabilities. Expected repetition
counts remain the default; the optional candidate-set count mode is unchanged.


## Optional repetition-risk penalty

With **PMI Viterbi + Repetition Penalty**, enable **Repetition-risk penalty**
(default OFF) to estimate earlier repetition risk from joint diffusion+PMI
sequence probabilities. This overrides the candidate-count checkbox if both
are enabled; OFF preserves the existing expected/candidate-count choice.

A forward/backward sum-product pass on the unpenalized candidate lattice yields
q_s(a,b). Earlier risk is approximated as r_t=1-prod_(s<t)(1-q_s).
The raw log-score correction is -gamma*log((1-r+9r)/(1-r)). At T=1, gamma=1,
risk 0.1 gives multiplier 0.5; risk 0.5 gives multiplier 0.1. Sequence temperature
subsequently scales the entire corrected score as usual. This is an approximate
risk heuristic, not the exact conditional probability of repetition or a guarantee
on normalized output probabilities. Overlapping occurrences are dependent.
Risk uses the same temperature as sequence sampling; maximum-path decoding uses
a T=1 soft distribution, including when sampling temperature is zero. Fixed
positions and existing PMI boundary terms are included. Duplicate fixed columns
have zero marginal. Counts reset per iteration/block and do not cross prompt or
block boundaries. EOS/EOT pairs neither seed risk nor receive penalties. Gamma=0
matches the unpenalized selected decoding mode exactly. A 1e-7 survival floor
keeps near-certain risk finite. Marginals are computed once before risk adjustment;
no iterative fixed-point calculation is performed.
There are no extra model forwards, external language models, or training changes.
The extra decoder passes add overhead. Existing defaults and other Spaces remain
unchanged. Independent temperature/bigram weight controls are preserved.
## Early-iteration repetition-penalty fade
**Fade repetition penalty over early iterations** is ON by default in the UI
for LLaDA. It only affects the repetition strategy, including expected-count,
candidate-count and risk modes. Turn it OFF to retain constant strength.
API calls that omit decay_repetition_penalty retain constant strength for
compatibility; pass true explicitly to match the new UI.
Gamma_t = gamma * f(t), with zero-based iteration t within each block.
Let L=6, c=(L-1)/2 and g(t)=1/(1+exp(1.6*(t-c))). Then
f(t)=(g(t)-g(L-1))/(g(0)-g(L-1)) for 0<=t<L-1, and zero afterward.
The first iteration is exactly full strength. From iteration 6 the penalty is
exactly zero, not merely asymptotically small, so the original unpenalized
sequence decoder runs and risk/count computations are bypassed.
The active window is absolute and does not stretch with total NFE. A one-step
block receives full strength, while longer blocks spend their later iterations
without repetition penalties. The schedule resets at each new block. APD early
exit does not rescale it. Edit REPETITION_FADE_ITERATIONS in app.py to change the
window. The sigmoid slope is fixed at 1.6. Sampling, diffusion confidence, PMI,
temperature and ending retention keep their existing scoring behavior.
There are no extra model forwards. All existing penalty modes and the constant
option remain available; improvement across budgets still needs evaluation.
## Tunable APD aggressiveness
With **Adaptive Parallel Decoding (APD)** ON, **APD aggressiveness** ranges from
0 to 1 and defaults to 0. The original warmup/cosine threshold schedule is
multiplied by (1-a). Thus a=0 retains 0.90 to 0.50, a=0.5 gives 0.45 to 0.25,
a=0.9 gives 0.09 to 0.05, and a=1 gives zero threshold, committing all finite
selected-token predictions in one forward per block. Higher values can reduce
passes but may reduce generation quality by locking less-confident predictions.
The control is interactive only while APD is enabled. API parameter is
apd_aggressiveness, default 0. APD OFF ignores it. The zero default reproduces
existing APD paths, thresholds and status text. Selected-token confidence stays
full-vocabulary and untempered, including when Viterbi selects a non-argmax token.
Progress fallback, final cleanup, MASK validity fallback, fixed positions, ending
retention delay, and early block exit remain unchanged. No extra model forwards.
Works with all existing decoding strategies, sequence sampling, repetition-risk
penalties and their fade. It changes commitment only. Repetition fading and all
other settings remain unchanged. Only this LLaDA Space was updated.
## APD minimum commitment per iteration
APD now commits at least ceil(block_length / allocated_block_iterations) pending
positions on each pass, capped by the number still masked. For a 64-token block
with 16 iterations, the minimum is 4 tokens per iteration. Nondivisible lengths
round up: a 65-token block at 16 iterations has a minimum of 5 until fewer remain.
This avoids leaving a large unresolved tail for the final pass. This is a fixed
per-step floor, not a reduced cumulative target after an earlier large commit.
All threshold-qualified pending positions still commit, even above the quota.
If they are fewer than the quota, fill the deficit by selected-token diffusion
confidence, without changing the selected IDs or adding model calls. Frozen rows
are excluded. With EOS/EOT delay, fill ordinary rows first; if fewer ordinary rows
remain than needed, fill endings only after all ordinary rows have been selected.
Final cleanup and early block exit remain. Each block uses its own length and
allocated iteration budget; unequal final blocks are handled independently.
No new UI control is added. APD OFF is unchanged. At aggressiveness 0 the original
threshold curve remains, but the quota is now guaranteed; the previous claim of
identical full APD behavior at zero is superseded by this intended commitment
change. Aggressiveness still lowers thresholds and can commit more than the quota.
All decoding strategies, sequence sampling, risk/expected/candidate penalties,
repetition decay and ending safeguards remain available. Other Spaces were not
updated. No extra model forwards or training changes are introduced.
## Optional expanded-corpus bigram and trigram guidance
**N-gram guidance** defaults to **Original bigram**, preserving the existing
checkpoint and behavior. **Expanded bigram** uses the new corpus; **Expanded
bigram + trigram** adds a second-order correction to that same expanded bigram.
These choices apply to both bigram decoding strategies, including sequence
sampling, repetition penalties/risk/decay, APD and its minimum quota. Ordinary
independent diffusion ignores this setting. Existing weights and model settings
are unchanged; no extra diffusion-model forwards or language model are needed.
The new corpus adds 800M LLaDA-tokenized tokens: 720M FineWeb-Edu English,
40M educational Python and 40M FineMath-4+, plus all original training responses.
The added-token ratio is 90% / 5% / 5%, not the initially proposed 50/25/25 mix.
Exact per-source text duplicates are skipped. Document boundaries are preserved;
EOS is appended to complete raw documents and nontruncated original responses.
The final sampled raw document may be cut to its exact token budget. Both new
scorers use the same corpus and exact tokenizer, not separately sampled data.
Sources and attribution:
- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), sample-10BT.
- [Python-Edu](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), via the
  [text-resolved mirror](https://huggingface.co/datasets/Avelina/python-edu-cleaned).
  The upstream publishes blob metadata, not file text; the mirror resolves it.
- [FineMath-4+](https://huggingface.co/datasets/HuggingFaceTB/finemath).
Pinned revisions, exact file paths, token/document counts and hashes accompany
these checkpoints in the local experiment/training records.
Score: sum log p_t + w*sum(PMI bigram) + 0.5*w*sum(delta trigram) - repetition.
Delta = log P(c|a,b) - log P(c|b), with alpha=50 bigram-backoff smoothing,
minimum trigram count 2, and delta clipped to [-2,2]. Unseen/singleton triples give
zero correction (backoff), never forbidden transitions. All bigram counts are
retained (PMI minimum count 1). The deployed trigram checkpoint stores triples
observed at least twice and their required contexts, retaining full prefix totals
including singleton contributions. Scorers are downloaded at startup from a
pinned model-repository revision, avoiding the Space's 1 GB storage limit.
The conditional bigram is the same smoothed backoff model used by the bigram
checkpoint. Edit TRIGRAM_RELATIVE_WEIGHT/MIN_COUNT/CLIP in app.py to tune it.
Trigrams are evaluated only wholly inside the current block; existing bigram
boundary terms remain. Fixed positions are singleton constraints. Exact max-path,
forward/backward sampling, path margins and joint bigram marginals use the
second-order chain. In risk mode, earlier-repetition estimates include trigram
scores as well as diffusion and PMI. Sequence temperature scales the complete
score; bigram weight 0 disables both n-gram corrections. For blocks of 1–2 tokens
there is no trigram term. The default Original bigram path is unchanged.
GPU candidate dimensions are vectorized, but trigram cost grows as K^3 rather
than K^2. Start at Top-k 16. At K=16, B=64 the extra FP32 trigram-potential table
is about 1 MiB; checkpoints and temporary buffers add memory. Longer corpus
matching does not check factual relevance to the question. Quality benefits and
latency overhead must be evaluated rather than assumed.

	---
	title: LLaDA-8B-Instruct
	emoji: 🌊
	colorFrom: blue
	colorTo: purple
	sdk: gradio
	sdk_version: 5.49.1
	python_version: "3.12"
	app_file: app.py
	pinned: false
	license: other
	startup_duration_timeout: 1h
	short_description: Native LLaDA with diffusion and PMI Viterbi decoding
	models:
	- GSAI-ML/LLaDA-8B-Instruct
	---

	# LLaDA-8B-Instruct

	Loads the official [GSAI-ML/LLaDA-8B-Instruct](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct) directly in
	BF16 on ZeroGPU. Model and tokenizer revision: `08b83a6feb34df1a6011b80c3c00c7563e963b07`.

	The interface shares the BYOD Gemma demo's decoding controls: temperature,
	top-k, block diffusion, confidence-guided or random retention, PMI Viterbi,
	ending delay, early stopping, and the live token visualization. The model
	uses its native bidirectional forward pass, chat template, and mask token
	126336. This is a comparison using the shared app decoder, not a claim to
	reproduce the official benchmark sampler.

	PMI uses a separate scorer trained on all 980,000 responses in the existing
	training split, retokenized with LLaDA's tokenizer. It covers all 126,464
	model output IDs, including output padding beyond the tokenizer vocabulary.
	Scoring is PMI, retention confidence is diffusion probability, minimum count
	is 1, and clipping is ±2. Edit the constants near the top of `app.py` to
	change these defaults. Temperature and top-k are shared controls; bigram
	weight is fixed at 0.5. Ending-delay strength remains adjustable.

	LLaDA's available strategies are confidence-guided, random, and PMI Viterbi.
	The adapter-based autoregressive confidence strategy applies to BYOD models.
	The first request may take longer while ZeroGPU allocates hardware.


	## Optional repetition-aware PMI Viterbi

	Select PMI Viterbi + Repetition Penalty to use the block-local transition
	score `0.5 * temperature * PMI(a,b) - gamma * C(a,b)`. Gamma defaults to 1.0;
	0 reproduces the original PMI decoder. C accumulates earlier expected bigram
	occurrences from full-vocabulary diffusion probabilities, updated only after
	scoring each position. EOS/EOT pairs are neither penalized nor counted. Retained
	tokens remain fixed constraints. Counts reset each iteration and block, without
	crossing prompt/block boundaries. There are no additional model forward passes.
	The original Bigram Viterbi option and generation defaults remain available.


	## Adaptive Parallel Decoding

	The Adaptive Parallel Decoding (APD) checkbox defaults OFF. APD is a
	commitment layer after the selected sampling or sequence decoder. Each block
	uses a cosine confidence threshold from 0.90 to 0.50, with a 15% warmup. It
	commits all qualifying masked positions using their selected token's original
	full-vocabulary diffusion probability. Remaining positions stay masked; a
	best-confidence fallback makes progress and the final step resolves all masks.
	Completed blocks exit early, independently of the convergence-stop checkbox.
	Existing EOS/EOT retention delay is respected, with endings retained after
	ordinary predictions; whole-answer EOS/EOT stopping still follows Early stopping.
	Committed values stay fixed. Optional AR retention scoring and retained-token
	revision are bypassed under APD. No additional model forwards, token budgets,
	position factors, or grinding-phase detection are used. OFF retains the original
	decoding configuration. This implements the requested APD subset, inspired by
	[CAI-dLLM](https://arxiv.org/abs/2608.22646), rather than its complete controller.


	## Optional sequence sampling and candidate-occurrence counts

	Both checkboxes default OFF, preserving existing decoding. **Sample bigram-guided
	sequences** replaces the maximum-path decision with exact forward-filter/backward
	sampling from exp(total path score) on the same candidate lattice. Diffusion,
	PMI, boundaries and any repetition scores are unchanged; no score perturbations
	or extra model forwards are used. Seed controls reproducibility. This applies
	to either bigram decoding strategy, independently of APD.

	Count every candidate bigram occurrence applies only with the repetition
	penalty strategy. Each earlier adjacent pair of candidate sets containing (a,b)
	adds 1 to C(a,b), irrespective of probability. Duplicate fixed-token columns
	count once; EOS/EOT pairs are excluded. Counts update after scoring each edge
	and reset each block/iteration. Gamma still subtracts gamma*C in log-score units;
	this is not a percentage-parameter remapping. Gamma=0 restores unpenalized results
	for the selected maximum-path or sampling mode. Ordinary decoding ignores these
	checkboxes; the original temperature weighting is unchanged.


	## Independent temperature and bigram weight

	The LLaDA Space now uses S(y) = sum log p_t(y_t) + w sum PMI - gamma sum C.
	Bigram weight ranges from 0 to 5, default 1. A weight of 1 gives equal
	coefficients, not necessarily equal numerical influence. Weight 0 disables PMI
	but does not disable repetition penalties. The previous fixed 0.5*temperature
	PMI coefficient described above is superseded for this Space.

	Sample bigram-guided sequences samples P(y) proportional to exp(S(y)/T).
	Temperature controls the entropy of the whole sequence score, including repetition,
	without changing the ratios of score coefficients. T=0 selects the maximum path;
	positive T is floored to 1e-6 for numerical stability. Maximum-path Viterbi ignores
	temperature. Raw full-vocabulary diffusion probabilities still determine expected
	repetition counts and retention/APD confidence. Default T is now 1, w is 1,
	matching the former T=2 PMI coefficient and sequence distribution. Ordinary
	diffusion retains its usual temperature behavior. No extra model forwards.

	Unseen pairs have a neutral PMI correction of zero, never a forbidden transition.
	Conditional scoring instead uses smoothed finite probabilities. Expected repetition
	counts remain the default; the optional candidate-set count mode is unchanged.


	## Optional repetition-risk penalty

	With PMI Viterbi + Repetition Penalty, enable Repetition-risk penalty
	(default OFF) to estimate earlier repetition risk from joint diffusion+PMI
	sequence probabilities. This overrides the candidate-count checkbox if both
	are enabled; OFF preserves the existing expected/candidate-count choice.

	A forward/backward sum-product pass on the unpenalized candidate lattice yields
	q_s(a,b). Earlier risk is approximated as r_t=1-prod_(s<t)(1-q_s).
	The raw log-score correction is -gamma*log((1-r+9r)/(1-r)). At T=1, gamma=1,
	risk 0.1 gives multiplier 0.5; risk 0.5 gives multiplier 0.1. Sequence temperature
	subsequently scales the entire corrected score as usual. This is an approximate
	risk heuristic, not the exact conditional probability of repetition or a guarantee
	on normalized output probabilities. Overlapping occurrences are dependent.

	Risk uses the same temperature as sequence sampling; maximum-path decoding uses
	a T=1 soft distribution, including when sampling temperature is zero. Fixed
	positions and existing PMI boundary terms are included. Duplicate fixed columns
	have zero marginal. Counts reset per iteration/block and do not cross prompt or
	block boundaries. EOS/EOT pairs neither seed risk nor receive penalties. Gamma=0
	matches the unpenalized selected decoding mode exactly. A 1e-7 survival floor
	keeps near-certain risk finite. Marginals are computed once before risk adjustment;
	no iterative fixed-point calculation is performed.

	There are no extra model forwards, external language models, or training changes.
	The extra decoder passes add overhead. Existing defaults and other Spaces remain
	unchanged. Independent temperature/bigram weight controls are preserved.


	## Early-iteration repetition-penalty fade

	Fade repetition penalty over early iterations is ON by default in the UI
	for LLaDA. It only affects the repetition strategy, including expected-count,
	candidate-count and risk modes. Turn it OFF to retain constant strength.
	API calls that omit decay_repetition_penalty retain constant strength for
	compatibility; pass true explicitly to match the new UI.

	Gamma_t = gamma * f(t), with zero-based iteration t within each block.
	Let L=6, c=(L-1)/2 and g(t)=1/(1+exp(1.6*(t-c))). Then
	f(t)=(g(t)-g(L-1))/(g(0)-g(L-1)) for 0<=t<L-1, and zero afterward.
	The first iteration is exactly full strength. From iteration 6 the penalty is
	exactly zero, not merely asymptotically small, so the original unpenalized
	sequence decoder runs and risk/count computations are bypassed.

	The active window is absolute and does not stretch with total NFE. A one-step
	block receives full strength, while longer blocks spend their later iterations
	without repetition penalties. The schedule resets at each new block. APD early
	exit does not rescale it. Edit REPETITION_FADE_ITERATIONS in app.py to change the
	window. The sigmoid slope is fixed at 1.6. Sampling, diffusion confidence, PMI,
	temperature and ending retention keep their existing scoring behavior.

	There are no extra model forwards. All existing penalty modes and the constant
	option remain available; improvement across budgets still needs evaluation.


	## Tunable APD aggressiveness

	With Adaptive Parallel Decoding (APD) ON, APD aggressiveness ranges from
	0 to 1 and defaults to 0. The original warmup/cosine threshold schedule is
	multiplied by (1-a). Thus a=0 retains 0.90 to 0.50, a=0.5 gives 0.45 to 0.25,
	a=0.9 gives 0.09 to 0.05, and a=1 gives zero threshold, committing all finite
	selected-token predictions in one forward per block. Higher values can reduce
	passes but may reduce generation quality by locking less-confident predictions.

	The control is interactive only while APD is enabled. API parameter is
	apd_aggressiveness, default 0. APD OFF ignores it. The zero default reproduces
	existing APD paths, thresholds and status text. Selected-token confidence stays
	full-vocabulary and untempered, including when Viterbi selects a non-argmax token.
	Progress fallback, final cleanup, MASK validity fallback, fixed positions, ending
	retention delay, and early block exit remain unchanged. No extra model forwards.

	Works with all existing decoding strategies, sequence sampling, repetition-risk
	penalties and their fade. It changes commitment only. Repetition fading and all
	other settings remain unchanged. Only this LLaDA Space was updated.


	## APD minimum commitment per iteration

	APD now commits at least ceil(block_length / allocated_block_iterations) pending
	positions on each pass, capped by the number still masked. For a 64-token block
	with 16 iterations, the minimum is 4 tokens per iteration. Nondivisible lengths
	round up: a 65-token block at 16 iterations has a minimum of 5 until fewer remain.
	This avoids leaving a large unresolved tail for the final pass. This is a fixed
	per-step floor, not a reduced cumulative target after an earlier large commit.

	All threshold-qualified pending positions still commit, even above the quota.
	If they are fewer than the quota, fill the deficit by selected-token diffusion
	confidence, without changing the selected IDs or adding model calls. Frozen rows
	are excluded. With EOS/EOT delay, fill ordinary rows first; if fewer ordinary rows
	remain than needed, fill endings only after all ordinary rows have been selected.
	Final cleanup and early block exit remain. Each block uses its own length and
	allocated iteration budget; unequal final blocks are handled independently.

	No new UI control is added. APD OFF is unchanged. At aggressiveness 0 the original
	threshold curve remains, but the quota is now guaranteed; the previous claim of
	identical full APD behavior at zero is superseded by this intended commitment
	change. Aggressiveness still lowers thresholds and can commit more than the quota.
	All decoding strategies, sequence sampling, risk/expected/candidate penalties,
	repetition decay and ending safeguards remain available. Other Spaces were not
	updated. No extra model forwards or training changes are introduced.

	## Optional expanded-corpus bigram and trigram guidance

	N-gram guidance defaults to Original bigram, preserving the existing
	checkpoint and behavior. Expanded bigram uses the new corpus; **Expanded
	bigram + trigram** adds a second-order correction to that same expanded bigram.
	These choices apply to both bigram decoding strategies, including sequence
	sampling, repetition penalties/risk/decay, APD and its minimum quota. Ordinary
	independent diffusion ignores this setting. Existing weights and model settings
	are unchanged; no extra diffusion-model forwards or language model are needed.

	The new corpus adds 800M LLaDA-tokenized tokens: 720M FineWeb-Edu English,
	40M educational Python and 40M FineMath-4+, plus all original training responses.
	The added-token ratio is 90% / 5% / 5%, not the initially proposed 50/25/25 mix.
	Exact per-source text duplicates are skipped. Document boundaries are preserved;
	EOS is appended to complete raw documents and nontruncated original responses.
	The final sampled raw document may be cut to its exact token budget. Both new
	scorers use the same corpus and exact tokenizer, not separately sampled data.

	Sources and attribution:
	- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), sample-10BT.
	- [Python-Edu](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), via the
	[text-resolved mirror](https://huggingface.co/datasets/Avelina/python-edu-cleaned).
	The upstream publishes blob metadata, not file text; the mirror resolves it.
	- [FineMath-4+](https://huggingface.co/datasets/HuggingFaceTB/finemath).
	Pinned revisions, exact file paths, token/document counts and hashes accompany
	these checkpoints in the local experiment/training records.

	Score: sum log p_t + wsum(PMI bigram) + 0.5w*sum(delta trigram) - repetition.
	Delta = log P(c\|a,b) - log P(c\|b), with alpha=50 bigram-backoff smoothing,
	minimum trigram count 2, and delta clipped to [-2,2]. Unseen/singleton triples give
	zero correction (backoff), never forbidden transitions. All bigram counts are
	retained (PMI minimum count 1). The deployed trigram checkpoint stores triples
	observed at least twice and their required contexts, retaining full prefix totals
	including singleton contributions. Scorers are downloaded at startup from a
	pinned model-repository revision, avoiding the Space's 1 GB storage limit.
	The conditional bigram is the same smoothed backoff model used by the bigram
	checkpoint. Edit TRIGRAM_RELATIVE_WEIGHT/MIN_COUNT/CLIP in app.py to tune it.

	Trigrams are evaluated only wholly inside the current block; existing bigram
	boundary terms remain. Fixed positions are singleton constraints. Exact max-path,
	forward/backward sampling, path margins and joint bigram marginals use the
	second-order chain. In risk mode, earlier-repetition estimates include trigram
	scores as well as diffusion and PMI. Sequence temperature scales the complete
	score; bigram weight 0 disables both n-gram corrections. For blocks of 1–2 tokens
	there is no trigram term. The default Original bigram path is unchanged.

	GPU candidate dimensions are vectorized, but trigram cost grows as K^3 rather
	than K^2. Start at Top-k 16. At K=16, B=64 the extra FP32 trigram-potential table
	is about 1 MiB; checkpoints and temporary buffers add memory. Longer corpus
	matching does not check factual relevance to the question. Quality benefits and
	latency overhead must be evaluated rather than assumed.