ychuai/community-notes-topic-classifier

Multilingual Community Notes topic classifier

TwHIN-BERT fine-tuned to assign zero or more of ten topics to a post and its associated Community Note summaries. The unit of classification is the combined post record, not an individual note.

Labels

Politics and Elections; War and Geopolitics; Health and Medicine; Economy and Finance; Technology and AI; Crime and Public Safety; Sports and Games; Celebrity and Entertainment; Religion and Spirituality; Gender and Identity.

Inputs and inference

Input format:

Original post:
The original post text

Community Note 1:
First summary

Community Note 2:
Second summary

Use summaries alone when the original post is unavailable, or post text alone when summaries are unavailable. Duplicate nonempty summary texts are included once. Input order should match the prepared record.

Long inputs are split into overlapping windows of 512 tokens with a 64-token stride. For each topic, take the maximum logit over all windows, then apply sigmoid. Scores ≥0.5 are selected independently. All settings are in topic_config.json.

Quick start with Transformers

Install the dependencies in your terminal (or use %pip install in a notebook):

pip install torch "transformers>=4.41,<5" accelerate

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "ychuai/community-notes-topic-classifier"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(
    repo, device_map="auto"
)
model.eval()

text = (
    "Original post:\nA new vaccine study was published.\n\n"
    "Community Note 1:\nThe study was conducted in adults."
)

inputs = tokenizer(
    text, return_tensors="pt", truncation=True, max_length=512
).to(model.device)

with torch.inference_mode():
    scores = model(**inputs).logits.sigmoid()[0]

for index, score in enumerate(scores.tolist()):
    if score >= 0.5:
        print(f"{model.config.id2label[index]}: {score:.3f}")

Example output for this input (small numerical differences can occur across devices):

Health and Medicine: 0.977

Use sigmoid rather than softmax: each topic is scored independently, and multiple topics or none may be selected.

This quick-start example truncates text beyond 512 tokens. For longer posts or multiple summaries, use the included helper below to preserve the model's overlapping-window max-logit pooling.

Long inputs: use all windows

Download the inference files and run the included helper:

from pathlib import Path
import runpy
from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="ychuai/community-notes-topic-classifier",
    allow_patterns=[
        "config.json", "model.safetensors", "tokenizer.json",
        "tokenizer_config.json", "special_tokens_map.json",
        "topic_config.json", "predict_topics.py",
    ],
)
TopicPredictor = runpy.run_path(
    str(Path(model_dir) / "predict_topics.py")
)["TopicPredictor"]
classifier = TopicPredictor(model_dir, device="cpu")

results = classifier.predict(
    post_text="A new vaccine study was published.",
    summaries=["The study was conducted in adults."],
)
for result in results:
    if result["selected"]:
        print(f"{result['topic']}: {result['score']:.3f}")

The helper returns all ten scores and their selected flags. Omit post_text to classify summaries alone. It can also be run from a downloaded model folder:

python predict_topics.py --post "A new vaccine study was published." --note "The study was conducted in adults."

These examples download the model from Hugging Face and run inference on your computer or notebook. No hosted inference endpoint or OpenAI API key is needed. The first model download is approximately 1.13 GB; later runs reuse the cache.

Training and evaluation

Base model: Twitter/twhin-bert-base.

Training inputs combine original post text with all associated nonempty note summaries. The pipeline adopts gpt-5.4-mini to label 20,000 combined post records with multi-label topic annotations, deduplicates identical combined inputs, and uses an 85%/15% training/validation split. Window logits are max-pooled before a post-level binary cross-entropy loss. The saved evaluation was recorded at epoch 5.

Held-out metric	Value
Micro F1	0.831219
Macro F1	0.809377
Samples F1	0.761806
Evaluation loss	0.187633

Intended use and limitations

Research topic annotation of multilingual posts and summaries. Topic predictions do not determine whether a statement is true or a note is Helpful. Multiple categories or no category can be selected. Language-specific performance has not been established by the saved evaluation. Domain shifts, synthetic-label bias, classification errors, and window max-pooling can affect predictions.

Combined summaries can include information added after the original post. These predictions are snapshot labels, not historical annotations. The model package contains weights, configuration, tokenizer, inference code, and aggregate metrics; the source corpus and annotation records are not bundled.

License

This fine-tuned model is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Related papers

Please cite the following related papers when using this model in research:

@article{chuai2026community,
  title={Community corrections have divergent downstream effects across corrected accounts},
  author={Chuai, Yuwei and Renault, Thomas and Pr{\"o}llochs, Nicolas and Lenzini, Gabriele and Mosleh, Mohsen},
  journal={arXiv preprint arXiv:2608.27526},
  year={2026}
}

@article{renault2026grok,
  title={@ Grok is this true? LLM-powered fact-checking on social media},
  author={Renault, Thomas and Mosleh, Mohsen and Rand, David},
  year={2026},
  publisher={OSF}
}