latent-entity

Span extraction model fine-tuned from google/gemma-3-270m (268M parameters). Given an article title and text, it generates the spans of the text that match , one per line, or [NONE] when the text has none. The repository also contains two DeBERTa-v3-large token classifiers trained on the same annotations. Demo: dejanseo/latent-entity.

Models

Path	Architecture	Base model	Output	Loss
root, generative/	Gemma3ForCausalLM, 268M	google/gemma-3-270m	generated span text	cross-entropy on completion tokens
cross-entropy/	DebertaV2ForTokenClassification	microsoft/deberta-v3-large	BIO tags (O, B-SPAN, I-SPAN)	cross-entropy
focal-loss/	DebertaV2ForTokenClassification	microsoft/deberta-v3-large	BIO tags (O, B-SPAN, I-SPAN)	focal loss, alpha 0.25 / 15.0 / 1.0 (O / B-SPAN / I-SPAN), gamma 1.0

Prompt format (generative model)

Title: {title}

Text: {text}

Hooks:

The model completes the text after Hooks: with one span per line. Inputs are limited to 1,024 tokens, with 256 reserved for the completion. Longer texts are split into overlapping chunks (stride 256 tokens) and the spans are deduplicated, as in app.py.

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dejanseo/latent-entity")
model = AutoModelForCausalLM.from_pretrained("dejanseo/latent-entity")

prompt = f"Title: {title}\n\nText: {text}\n\nHooks:\n"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
spans = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip().split("\n")

Training

Training data is a JSON file where each item is title[SEP]text, with target spans wrapped in [SPAN]...[/SPAN]. 95% of chunks were used for training and 5% for validation (seed 42).

	Generative	DeBERTa classifiers
Max sequence length	1,024	512
Chunk stride	256	128
Batch size	1 x 16 accumulation	4 x 4 accumulation
Learning rate	2e-5	2e-5
Precision	bf16	bf16
Best checkpoint chosen by	eval loss	entity F1 (50% overlap match)

Files

train.py, train-cross-entropy.py, train-focal-loss.py and pretokenize.py are the training scripts. app.py and inference.py are Streamlit apps for the generative and classifier models.

Background

A latent entity is "a critical piece of information a title deliberately withholds to force a click" (DEJAN concept entry). The title is the observed data, the withheld item is the unobserved variable, and the click is the cost of inference. Four types are defined (Latent Entities, DEJAN blog):

Type	Withheld item	Example title
Latent subject	What	"5 tools every developer needs in their workflow"
Latent reason	Why	"I finally understand why Linux users swear by simple tools"
Latent process	How	"How a power drill defeated the Xbox 360's console security"
Latent outcome	What happened	"I ran local LLMs on a dying GPU and the results surprised me"