---
title: LLaDA-8B-Instruct
emoji: 🌊
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
python_version: "3.12"
app_file: app.py
pinned: false
license: other
startup_duration_timeout: 1h
short_description: Native LLaDA with diffusion and PMI Viterbi decoding
models:
- GSAI-ML/LLaDA-8B-Instruct
---

# LLaDA-8B-Instruct

Loads the official [GSAI-ML/LLaDA-8B-Instruct](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct) directly in
BF16 on ZeroGPU. Model and tokenizer revision: `08b83a6feb34df1a6011b80c3c00c7563e963b07`.

The interface shares the BYOD Gemma demo's decoding controls: temperature,
top-k, block diffusion, confidence-guided or random retention, PMI Viterbi,
ending delay, early stopping, and the live token visualization. The model
uses its native bidirectional forward pass, chat template, and mask token
126336. This is a comparison using the shared app decoder, not a claim to
reproduce the official benchmark sampler.

PMI uses a separate scorer trained on all 980,000 responses in the existing
training split, retokenized with LLaDA's tokenizer. It covers all 126,464
model output IDs, including output padding beyond the tokenizer vocabulary.
Scoring is PMI, retention confidence is diffusion probability, minimum count
is 1, and clipping is ±2. Edit the constants near the top of `app.py` to
change these defaults. Temperature and top-k are shared controls; bigram
weight is fixed at 0.5. Ending-delay strength remains adjustable.

LLaDA's available strategies are confidence-guided, random, and PMI Viterbi.
The adapter-based autoregressive confidence strategy applies to BYOD models.
The first request may take longer while ZeroGPU allocates hardware.


## Optional repetition-aware PMI Viterbi

Select **PMI Viterbi + Repetition Penalty** to use the block-local transition
score `0.5 * temperature * PMI(a,b) - gamma * C(a,b)`. Gamma defaults to 1.0;
0 reproduces the original PMI decoder. C accumulates earlier expected bigram
occurrences from full-vocabulary diffusion probabilities, updated only after
scoring each position. EOS/EOT pairs are neither penalized nor counted. Retained
tokens remain fixed constraints. Counts reset each iteration and block, without
crossing prompt/block boundaries. There are no additional model forward passes.
The original **Bigram Viterbi** option and generation defaults remain available.


## Adaptive Parallel Decoding

The **Adaptive Parallel Decoding (APD)** checkbox defaults OFF. APD is a
commitment layer after the selected sampling or sequence decoder. Each block
uses a cosine confidence threshold from 0.90 to 0.50, with a 15% warmup. It
commits all qualifying masked positions using their selected token's original
full-vocabulary diffusion probability. Remaining positions stay masked; a
best-confidence fallback makes progress and the final step resolves all masks.
Completed blocks exit early, independently of the convergence-stop checkbox.
Existing EOS/EOT retention delay is respected, with endings retained after
ordinary predictions; whole-answer EOS/EOT stopping still follows Early stopping.
Committed values stay fixed. Optional AR retention scoring and retained-token
revision are bypassed under APD. No additional model forwards, token budgets,
position factors, or grinding-phase detection are used. OFF retains the original
decoding configuration. This implements the requested APD subset, inspired by
[CAI-dLLM](https://arxiv.org/abs/2608.22646), rather than its complete controller.


## Optional sequence sampling and candidate-occurrence counts

Both checkboxes default OFF, preserving existing decoding. **Sample bigram-guided
sequences** replaces the maximum-path decision with exact forward-filter/backward
sampling from exp(total path score) on the same candidate lattice. Diffusion,
PMI, boundaries and any repetition scores are unchanged; no score perturbations
or extra model forwards are used. Seed controls reproducibility. This applies
to either bigram decoding strategy, independently of APD.

**Count every candidate bigram occurrence** applies only with the repetition
penalty strategy. Each earlier adjacent pair of candidate sets containing (a,b)
adds 1 to C(a,b), irrespective of probability. Duplicate fixed-token columns
count once; EOS/EOT pairs are excluded. Counts update after scoring each edge
and reset each block/iteration. Gamma still subtracts gamma*C in log-score units;
this is not a percentage-parameter remapping. Gamma=0 restores unpenalized results
for the selected maximum-path or sampling mode. Ordinary decoding ignores these
checkboxes; the original temperature weighting is unchanged.


## Independent temperature and bigram weight

The LLaDA Space now uses S(y) = sum log p_t(y_t) + w sum PMI - gamma sum C.
**Bigram weight** ranges from 0 to 5, default 1. A weight of 1 gives equal
coefficients, not necessarily equal numerical influence. Weight 0 disables PMI
but does not disable repetition penalties. The previous fixed 0.5*temperature
PMI coefficient described above is superseded for this Space.

**Sample bigram-guided sequences** samples P(y) proportional to exp(S(y)/T).
Temperature controls the entropy of the whole sequence score, including repetition,
without changing the ratios of score coefficients. T=0 selects the maximum path;
positive T is floored to 1e-6 for numerical stability. Maximum-path Viterbi ignores
temperature. Raw full-vocabulary diffusion probabilities still determine expected
repetition counts and retention/APD confidence. Default T is now 1, w is 1,
matching the former T=2 PMI coefficient and sequence distribution. Ordinary
diffusion retains its usual temperature behavior. No extra model forwards.

Unseen pairs have a neutral PMI correction of zero, never a forbidden transition.
Conditional scoring instead uses smoothed finite probabilities. Expected repetition
counts remain the default; the optional candidate-set count mode is unchanged.


## Optional repetition-risk penalty

With **PMI Viterbi + Repetition Penalty**, enable **Repetition-risk penalty**
(default OFF) to estimate earlier repetition risk from joint diffusion+PMI
sequence probabilities. This overrides the candidate-count checkbox if both
are enabled; OFF preserves the existing expected/candidate-count choice.

A forward/backward sum-product pass on the unpenalized candidate lattice yields
q_s(a,b). Earlier risk is approximated as r_t=1-prod_(s<t)(1-q_s).
The raw log-score correction is -gamma*log((1-r+9r)/(1-r)). At T=1, gamma=1,
risk 0.1 gives multiplier 0.5; risk 0.5 gives multiplier 0.1. Sequence temperature
subsequently scales the entire corrected score as usual. This is an approximate
risk heuristic, not the exact conditional probability of repetition or a guarantee
on normalized output probabilities. Overlapping occurrences are dependent.

Risk uses the same temperature as sequence sampling; maximum-path decoding uses
a T=1 soft distribution, including when sampling temperature is zero. Fixed
positions and existing PMI boundary terms are included. Duplicate fixed columns
have zero marginal. Counts reset per iteration/block and do not cross prompt or
block boundaries. EOS/EOT pairs neither seed risk nor receive penalties. Gamma=0
matches the unpenalized selected decoding mode exactly. A 1e-7 survival floor
keeps near-certain risk finite. Marginals are computed once before risk adjustment;
no iterative fixed-point calculation is performed.

There are no extra model forwards, external language models, or training changes.
The extra decoder passes add overhead. Existing defaults and other Spaces remain
unchanged. Independent temperature/bigram weight controls are preserved.


## Early-iteration repetition-penalty fade

**Fade repetition penalty over early iterations** is ON by default in the UI
for LLaDA. It only affects the repetition strategy, including expected-count,
candidate-count and risk modes. Turn it OFF to retain constant strength.
API calls that omit decay_repetition_penalty retain constant strength for
compatibility; pass true explicitly to match the new UI.

Gamma_t = gamma * f(t), with zero-based iteration t within each block.
Let L=6, c=(L-1)/2 and g(t)=1/(1+exp(1.6*(t-c))). Then
f(t)=(g(t)-g(L-1))/(g(0)-g(L-1)) for 0<=t<L-1, and zero afterward.
The first iteration is exactly full strength. From iteration 6 the penalty is
exactly zero, not merely asymptotically small, so the original unpenalized
sequence decoder runs and risk/count computations are bypassed.

The active window is absolute and does not stretch with total NFE. A one-step
block receives full strength, while longer blocks spend their later iterations
without repetition penalties. The schedule resets at each new block. APD early
exit does not rescale it. Edit REPETITION_FADE_ITERATIONS in app.py to change the
window. The sigmoid slope is fixed at 1.6. Sampling, diffusion confidence, PMI,
temperature and ending retention keep their existing scoring behavior.

There are no extra model forwards. All existing penalty modes and the constant
option remain available; improvement across budgets still needs evaluation.


## Tunable APD aggressiveness

With **Adaptive Parallel Decoding (APD)** ON, **APD aggressiveness** ranges from
0 to 1 and defaults to 0. The original warmup/cosine threshold schedule is
multiplied by (1-a). Thus a=0 retains 0.90 to 0.50, a=0.5 gives 0.45 to 0.25,
a=0.9 gives 0.09 to 0.05, and a=1 gives zero threshold, committing all finite
selected-token predictions in one forward per block. Higher values can reduce
passes but may reduce generation quality by locking less-confident predictions.

The control is interactive only while APD is enabled. API parameter is
apd_aggressiveness, default 0. APD OFF ignores it. The zero default reproduces
existing APD paths, thresholds and status text. Selected-token confidence stays
full-vocabulary and untempered, including when Viterbi selects a non-argmax token.
Progress fallback, final cleanup, MASK validity fallback, fixed positions, ending
retention delay, and early block exit remain unchanged. No extra model forwards.

Works with all existing decoding strategies, sequence sampling, repetition-risk
penalties and their fade. It changes commitment only. Repetition fading and all
other settings remain unchanged. Only this LLaDA Space was updated.


## APD minimum commitment per iteration

APD now commits at least ceil(block_length / allocated_block_iterations) pending
positions on each pass, capped by the number still masked. For a 64-token block
with 16 iterations, the minimum is 4 tokens per iteration. Nondivisible lengths
round up: a 65-token block at 16 iterations has a minimum of 5 until fewer remain.
This avoids leaving a large unresolved tail for the final pass. This is a fixed
per-step floor, not a reduced cumulative target after an earlier large commit.

All threshold-qualified pending positions still commit, even above the quota.
If they are fewer than the quota, fill the deficit by selected-token diffusion
confidence, without changing the selected IDs or adding model calls. Frozen rows
are excluded. With EOS/EOT delay, fill ordinary rows first; if fewer ordinary rows
remain than needed, fill endings only after all ordinary rows have been selected.
Final cleanup and early block exit remain. Each block uses its own length and
allocated iteration budget; unequal final blocks are handled independently.

No new UI control is added. APD OFF is unchanged. At aggressiveness 0 the original
threshold curve remains, but the quota is now guaranteed; the previous claim of
identical full APD behavior at zero is superseded by this intended commitment
change. Aggressiveness still lowers thresholds and can commit more than the quota.
All decoding strategies, sequence sampling, risk/expected/candidate penalties,
repetition decay and ending safeguards remain available. Other Spaces were not
updated. No extra model forwards or training changes are introduced.

## Optional expanded-corpus bigram and trigram guidance

**N-gram guidance** defaults to **Original bigram**, preserving the existing
checkpoint and behavior. **Expanded bigram** uses the new corpus; **Expanded
bigram + trigram** adds a second-order correction to that same expanded bigram.
These choices apply to both bigram decoding strategies, including sequence
sampling, repetition penalties/risk/decay, APD and its minimum quota. Ordinary
independent diffusion ignores this setting. Existing weights and model settings
are unchanged; no extra diffusion-model forwards or language model are needed.

The new corpus adds 800M LLaDA-tokenized tokens: 720M FineWeb-Edu English,
40M educational Python and 40M FineMath-4+, plus all original training responses.
The added-token ratio is 90% / 5% / 5%, not the initially proposed 50/25/25 mix.
Exact per-source text duplicates are skipped. Document boundaries are preserved;
EOS is appended to complete raw documents and nontruncated original responses.
The final sampled raw document may be cut to its exact token budget. Both new
scorers use the same corpus and exact tokenizer, not separately sampled data.

Sources and attribution:
- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), sample-10BT.
- [Python-Edu](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), via the
  [text-resolved mirror](https://huggingface.co/datasets/Avelina/python-edu-cleaned).
  The upstream publishes blob metadata, not file text; the mirror resolves it.
- [FineMath-4+](https://huggingface.co/datasets/HuggingFaceTB/finemath).
Pinned revisions, exact file paths, token/document counts and hashes accompany
these checkpoints in the local experiment/training records.

Score: sum log p_t + w*sum(PMI bigram) + 0.5*w*sum(delta trigram) - repetition.
Delta = log P(c|a,b) - log P(c|b), with alpha=50 bigram-backoff smoothing,
minimum trigram count 2, and delta clipped to [-2,2]. Unseen/singleton triples give
zero correction (backoff), never forbidden transitions. All bigram counts are
retained (PMI minimum count 1). The deployed trigram checkpoint stores triples
observed at least twice and their required contexts, retaining full prefix totals
including singleton contributions. Scorers are downloaded at startup from a
pinned model-repository revision, avoiding the Space's 1 GB storage limit.
The conditional bigram is the same smoothed backoff model used by the bigram
checkpoint. Edit TRIGRAM_RELATIVE_WEIGHT/MIN_COUNT/CLIP in app.py to tune it.

Trigrams are evaluated only wholly inside the current block; existing bigram
boundary terms remain. Fixed positions are singleton constraints. Exact max-path,
forward/backward sampling, path margins and joint bigram marginals use the
second-order chain. In risk mode, earlier-repetition estimates include trigram
scores as well as diffusion and PMI. Sequence temperature scales the complete
score; bigram weight 0 disables both n-gram corrections. For blocks of 1–2 tokens
there is no trigram term. The default Original bigram path is unchanged.

GPU candidate dimensions are vectorized, but trigram cost grows as K^3 rather
than K^2. Start at Top-k 16. At K=16, B=64 the extra FP32 trigram-potential table
is about 1 MiB; checkpoints and temporary buffers add memory. Longer corpus
matching does not check factual relevance to the question. Quality benefits and
latency overhead must be evaluated rather than assumed.