Horizon-Labs/multilingual-web-topic-format-small

Multilingual Web Topic & Format Classifier (small, 141M)

Labels a web page with its topic and its format, in English and 65 other languages, using the 24 + 24 categories of WebOrganizer (Wettig et al., 2025):

• topic: Adult Content, Art & Design, Software Development, Crime & Law, Education & Jobs, Electronics & Hardware, Entertainment, Social Life, Fashion & Beauty, Finance & Business, Food & Dining, Games, Health, History & Geography, Home & Hobbies, Industrial, Literature, Politics, Religion, Science, Math & Technology, Software, Sports & Fitness, Transportation, Travel & Tourism;
• format: Academic Writing, Content Listing, Creative Writing, Customer Support Page, Discussion Forum / Comment Section, FAQs, Incomplete Content, Knowledge Article, Legal Notices, Listicle, News Article, Nonfiction Writing, Organizational About Page, Organizational Announcement, Personal About Page, Personal Blog, Product Page, Q&A Forum, Spam / Ads, Structured Data, Technical Writing, Transcript / Interview, Tutorial / How-To Guide, User Reviews.

Use it to analyse, filter or rebalance multilingual pretraining data (for example FineWeb-2) by domain, as WebOrganizer did for English. One model returns both: 48 sigmoid outputs, topic_* and format_*; take the highest of each group. Built on mmBERT-small, Apache-2.0, ONNX included. It reads the first 512 tokens; no URL needed. A larger, more accurate version is available as Horizon-Labs/multilingual-web-topic-format-base. onnx/model_quantized.onnx (int8 embeddings, 268 MB) gives the same 0.5 decision as fp32 on all 48 outputs for 99.5% of 400 test pages.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Horizon-Labs/multilingual-web-topic-format-small", top_k=None)
s = {d["label"]: d["score"] for d in clf("Rezept: Für den Teig 500 g Mehl, 2 Eier und eine Prise Salz verrühren ...")[0]}
topic = max((k for k in s if k.startswith("topic_")), key=s.get)
fmt = max((k for k in s if k.startswith("format_")), key=s.get)
print(topic, fmt)   # topic_food_dining format_tutorial_how_to_guide

Evaluation

• 65 languages: 9,750 held-out FineWeb-2 pages (150 per language, never used for training) labelled with WebOrganizer's prompts by an independent judge, google/gemma-4-31b-it (a different model family from our teacher).
• English: 3,000 pages of the test split of WebOrganizer's own annotations by Llama-3.1-405B (topics, formats); used only for evaluation.

model	65 languages: topic accuracy	65 languages: format accuracy	65 languages: topic macro-F1	English: topic accuracy	English: format accuracy	note
this model (141M)	0.780	0.695	0.754	0.802	0.668	66 languages, topic + format in one model
Horizon-Labs/multilingual-web-topic-format-base (308M)	0.812	0.719	0.791	0.817	0.688
WebOrganizer/TopicClassifier-NoURL + FormatClassifier-NoURL (2 x 140M)	0.395	0.252	0.388	0.862	0.789	English; trained on the Llama-3.1-405B labels of the English test (in-domain)
Qwen3.8-27B, prompted (our teacher, 27B)	0.885	0.773	0.876	0.864	0.743	for reference

• The table shows the released checkpoint. Means over 2 training seeds (small: 65-language topic .781 / .780, format .697 / .695; base: .810 / .812, .715 / .719).
• WebOrganizer's classifiers are English models trained on the same Llama-3.1-405B annotations as the English test, so they lead there; on other languages they fall to about 0.4 (topic) and 0.25 (format) agreement with the judge.
• Caveat: the multilingual reference is an LLM. On English, the judge agrees with Llama-3.1-405B on 84% of topics and 71% of formats, and our teacher on 86% / 74%, but the judge and teacher agree with each other more, which favours this model on that test. Formats are more ambiguous than topics for every annotator.

Training

• Pages: 151,965 web pages, about 2,000 per language from FineWeb-2 and 22,000 English pages from FineWeb (ODC-BY).
• Labels: Qwen3.8-27B (Apache-2.0), prompted with WebOrganizer's topic and format instructions (Apache-2.0; without URL or demonstrations), gave a probability for each category; the model learns these as soft labels. The Llama-3.1-405B annotations were not used for training.
• mmBERT-small, max length 512 tokens; checkpoint chosen on 7,579 held-out pages. Code: code/ in this repository.

Limitations

• Labels come from LLMs, not people, and many pages fit several categories (WebOrganizer itself reports format as the harder taxonomy).
• Only the text is used; WebOrganizer's URL-aware classifiers can use URL hints that this model does not see.
• Rare categories in some languages (e.g. Transcript / Interview, Legal Notices) have little training data.