llmon-span-tagger

Model Summary: llmon-span-tagger is a 149M-parameter token-classification model finetuned from ModernBERT-base that splits a natural-language prompt into typed spans (instruction, constraint and data) so the prompt can be rendered as LLMON (LLM Object Notation) markup. It tags every token with a BIO label and decodes spans from runs of tags. Its span boundaries therefore do not depend on punctuation, and it can cut inside a single sentence: "Write a poem about dogs | using less than 50 words" is one instruction and one constraint. It is the default backend of the lang.unstructured_text segmenter in the LLMON repository.

• Developers: IBM Research
• GitHub Repository: generative-computing/llmon (code in lang/unstructured_text)
• Paper: LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface
• Release Date: November 12, 2026
• License: Apache 2.0
• Supported Languages: English only.

Intended use: The model converts prompts written as plain text into LLMON for models trained to consume it, such as Granite-4.1-3B-LLMON. It is meant for prompts whose author you trust, and is useful for:

• Converting existing prompts, prompt libraries or instruction datasets into LLMON.
• Inspecting how a prompt decomposes into instructions, constraints and data, for example while learning LLMON or debugging a prompt.
• Research on prompt structure and instruction/constraint segmentation.

Out of scope: The model is not a security control. It must not be used to decide which parts of untrusted input are trustworthy; see Limitations. It is also not intended for non-English text, for general-purpose named-entity or document tagging, or for prompts longer than 2,048 tokens.

Label scheme

The head predicts 7 BIO tags: O, plus B-/I- for each of three classes. Each class renders to an LLMON tag:

Class	Meaning	LLMON tag	Collapsed from (annotation taxonomy)
instruction	The task the model is asked to perform	instr	Instruction
constraint	A rule, limit, format requirement, persona or example that shapes the output	constraint	Constraint, Output Format, Example, Role
data	Material the task operates on	data	Content, Structured Data, Quoted Content, Context

O marks tokens that belong to no span, such as inter-span whitespace and trailing punctuation, which annotators left unlabeled.

Annotations nest; the most common case is a Constraint inside an Instruction. A token inside several spans was trained on its innermost span, which is what teaches the model to split within a sentence. The model's output is a flat, non-overlapping list of spans. The authoritative class definitions are in docs/guidelines.html.

Usage

The checkpoint is a fine-tuned encoder plus a separate linear head. It is not a transformers AutoModelForTokenClassification checkpoint, and pipeline("token-classification") will not load it. Use the llmon package, which also applies the post-processing the reported scores were measured with. The standalone snippet further down loads the model without it.

With the llmon package (recommended)

pip install "llmon[unstructured-text] @ git+https://github.com/generative-computing/llmon"

from lang.unstructured_text import segment

result = segment(
    "You are a pirate. Write a poem about dogs using less than 50 words. "
    "Do not mention cats.",
    model="ibm-research/llmon-span-tagger",
)

for span in result.spans:
    print(f"{span.label:<12} {span.confidence:.2f}  {span.text}")

constraint   0.87  You are a pirate
instruction  0.97  Write a poem about dogs
constraint   0.91  using less than 50 words
constraint   0.99  Do not mention cats

The same spans can be rendered as LLMON (with the special tokens a model reads) or as human-readable Shorthand:

>>> print(result.to_short())
\constraint:a\ You are a pirate /constraint:a/
\instr:a\ Write a poem about dogs /instr:a/
\constraint:b\ using less than 50 words /constraint:b/
\constraint:c\ Do not mention cats /constraint:c/

>>> print(result.to_llmon())
<|open|>constraint<|:|>a<|close|> You are a pirate <|open_end|>constraint<|:|>a<|close|>
...

Every span satisfies prompt[span.start:span.end] == span.text. Each span also carries a confidence (the mean of the per-token maximum probabilities) and a requires_review flag, which is set when confidence is below 0.60. A command-line tool is included as well:

export LLMON_SEG_TAGGER_REPO=ibm-research/llmon-span-tagger
echo "Summarize this article in 3 bullets. Do not use commas." | llmon-segment --format spans
llmon-segment --repl        # interactive; the model stays loaded

Standalone, with transformers only

This snippet produces the raw per-token tags without the post-processing described below, so expect more fragmented spans than segment() returns.

import torch
from huggingface_hub import snapshot_download
from torch import nn
from transformers import AutoModel, AutoTokenizer

path = snapshot_download("ibm-research/llmon-span-tagger")
tokenizer = AutoTokenizer.from_pretrained(path)
encoder = AutoModel.from_pretrained(path, subfolder="encoder").eval()

state = torch.load(f"{path}/head.pt", map_location="cpu", weights_only=True)
tags = state["tags"]  # ['O', 'B-instruction', 'I-instruction', ...]
head = nn.Linear(encoder.config.hidden_size, len(tags))
head.load_state_dict(state["head"])

text = "Write a poem about dogs using less than 50 words."
enc = tokenizer(text, return_tensors="pt", return_offsets_mapping=True,
                truncation=True, max_length=2048)
offsets = enc.pop("offset_mapping")[0].tolist()
with torch.no_grad():
    pred = head(encoder(**enc).last_hidden_state)[0].argmax(-1).tolist()

for (start, end), p in zip(offsets, pred):
    if end > start:  # skip special tokens
        print(repr(text[start:end]), tags[p])

Repository files

File	Contents
encoder/model.safetensors, encoder/config.json	The fine-tuned ModernBERT-base encoder (22 layers, hidden size 768, fp32)
head.pt	The linear head (768 → 7) state dict and the ordered tag list. A PyTorch pickle holding only tensors and strings; it loads with torch.load(..., weights_only=True)
tokenizer.json, tokenizer_config.json	The unmodified ModernBERT-base tokenizer

Post-processing

segment() applies two measured defaults on top of the raw decode. Both were swept on the holdout across three training seeds:

Pass	Default	Holdout F1	Mid-sentence boundary recall	Decision
none (raw decode)	n/a	75.5	61.9%	baseline
Absorb 1–2 word spans with confidence < 0.60 into a more confident neighbour	on	78.8	55.8%	kept
Merge adjacent same-class spans	off	76.0	43.7%	rejected: +0.5 F1 is not worth −18 points of boundary recall

The requires_review gate (confidence < 0.60) was calibrated on the same holdout. It flags 17% of spans, and 97% of the flagged spans are wrong. All thresholds can be changed through LLMON_SEG_* settings; see the repository's .env.example.

Evaluation

All figures come from a single released checkpoint, evaluated on a 134-prompt holdout stratified across the four source benchmarks. Spans are micro-averaged over the three classes and scored against top-level (outermost) ground truth. The baseline is the repository's dependency-free, regex-based rules backend.

Span accuracy (three-class)

Backend	Recall	Precision	F1
rules	68.2	69.3	68.8
llmon-span-tagger	78.3	77.0	77.7

Per-class F1

Backend	instruction	constraint	data
rules	0.722	0.641	0.724
llmon-span-tagger	0.792	0.747	0.810

Boundary recall. A predicted cut within 3 characters of a ground-truth boundary counts. Mid-sentence boundaries are about 31% of all boundaries.

Backend	At a sentence break	Mid-sentence
rules	81.5% (224/275)	18.3% (23/126)
llmon-span-tagger	92.0% (253/275)	61.1% (77/126)

Read these figures with the following caveats:

• Seed variance. Training seed alone moved holdout scores by up to 8.8 points. Each figure above is one sample, not a mean.
• The holdout was reused. The same holdout was used to tune the post-processing thresholds and, following the project's documented procedure, to choose the released checkpoint among training seeds. The scores are therefore likely optimistic for unseen prompts.
• Small, non-public test set. 134 prompts (535 labeled spans), not distributed. Running the evaluation tooling on your own data measures your data; it does not reproduce these numbers.
• Over-splitting. The model predicts 665 spans against 535 ground-truth labels (+24%), which lowers precision. This is the main open accuracy problem.

Inference costs roughly 100 ms per prompt on CPU. Full tables and methodology are in docs/RESULTS.md.

Training Data

The model was trained on 1,353 prompts (~6,000 spans) taken from four public benchmarks and annotated with span labels for this project:

Source	Prompt license
IFEval (Zhou et al., arXiv:2311.07911)	Apache 2.0
MMLU (Hendrycks et al., arXiv:2009.03300)	MIT
InfoBench (Qin et al., arXiv:2401.03601)	see upstream
ComplexBench (Wen et al., arXiv:2407.03978)	see upstream

Prompts were annotated in a nine-class taxonomy following the project's annotation guidelines, then collapsed to three classes when loaded. The data was split into 1,219 training prompts and the 134-prompt holdout, stratified by source. The annotations are not distributed. The prompt text remains subject to each benchmark's original terms.

Training Procedure

The encoder and head were fine-tuned together, with full-parameter updates, using token-level cross-entropy over the BIO tags (special tokens excluded from the loss).

Setting	Value
Base model	answerdotai/ModernBERT-base
Optimizer	AdamW, learning rate 2e-5
Batch	up to 8 prompts, capped at a 4,096-token padded budget
Epochs	up to 4, early stopping (patience 2) on validation loss, reverting to the best epoch
Validation	20% of training prompts, split by prompt key; the holdout is not used for early stopping
Max sequence length	2,048 tokens
Precision	fp32

The training, evaluation and threshold-sweep code is open source. docs/TRAINING.md covers retraining on your own corpus, and docs/CORPUS.md gives the corpus schema and a browser-based annotation tool.

Infrastructure

We trained llmon-span-tagger on IBM's computing cluster, BlueVela, the same cluster used for the Granite-based LLMON models. Each training run used a single NVIDIA H100 80GB GPU.

Limitations

• Do not run it over untrusted content to decide what is trusted. In LLMON, an instr block is treated as authoritative by a downstream model. If this tagger labels attacker-controlled text as an instruction, LLMON rendering promotes that text to a command. Apply the tagger only to text whose author you trust. Wrap retrieved documents, user uploads and tool results as data in your application; do not ask the tagger to classify them.
• The injection check is a tripwire, not a defense. The model has no notion of prompt injection. The llmon package runs seven English keyword patterns alongside it and refuses to render when one matches. A paraphrase, another language or an encoding gets through. A negative result is not evidence of safety.
• Text past 2,048 tokens is silently dropped. Input is truncated to 2,048 tokens, and rendering emits only labeled spans. Anything past the limit is missing from the LLMON output, and no error is raised. Check the input length, or split long prompts, before rendering.
• Narrow training distribution. Training prompts come from four English instruction-following and QA benchmarks. System prompts, agent and tool-use prompts, multi-turn conversations, code-heavy prompts and conversational user writing are under-represented, so expect lower accuracy on them.
• Three classes, flat output. The model cannot represent the nine-class distinctions (for example Constraint vs. Output Format) or nested spans.
• Over-splitting and boundary errors. The model produces about 24% more spans than the ground truth, and constraint is its weakest class.
• Confidence is a heuristic. Span confidence is the mean of per-token maximum softmax probabilities. It ranks errors well on the holdout but is not a calibrated probability, and the 0.60 gate may not transfer to other data.

Ethical Considerations and Risks

The model inherits the considerations of its base model, ModernBERT-base, including biases present in its pre-training data. Its fine-tuning data comes from public benchmarks and was not collected from users. The main risk specific to this model is misuse as a security control (see Limitations): a mislabeled span changes what a downstream LLM treats as an instruction. We urge the community to use this model responsibly, to keep untrusted content out of its input, and to evaluate it on prompts representative of their own deployment before relying on it.

Citation

@misc{hind2026llmon,
  title         = {LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface},
  author        = {Hind, Michael and Shbita, Basel and Wu, Bo and Ahmed, Farhan and DeLuca, Chad and Fulton, Nathan and Cox, David and Gutfreund, Dan},
  year          = {2026},
  eprint        = {2603.22519},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2603.22519}
}