--- license: apache-2.0 language: - multilingual - en - ru - zh - de - ja - es - fr - it - pt - pl - nl - id - tr - cs - ko - ar - hu - fa - ro - vi - uk - nb - th - el - sv - da - fi - bg - sk - hr - hi - lt - he - bn - sl - et - ca - lv - ms - sr - ta - ur - mr - te - sw - ka - hy - is - mk - kk - az - tl - gl - eu - am - ne - gu - kn - ml - pa - si - uz - af - km - my library_name: transformers pipeline_tag: text-classification base_model: jhu-clsp/mmBERT-base datasets: - HuggingFaceFW/fineweb-2 tags: - topic-classification - domain-classification - web-classification - weborganizer - data-curation - pretraining-data - multilingual - modernbert - mmbert - onnx - transformers.js --- # Multilingual Web Topic & Format Classifier (base, 308M) Labels a web page with its **topic** and its **format**, in English and 65 other languages, using the 24 + 24 categories of [WebOrganizer](https://github.com/CodeCreator/WebOrganizer) (Wettig et al., 2025): - **topic**: Adult Content, Art & Design, Software Development, Crime & Law, Education & Jobs, Electronics & Hardware, Entertainment, Social Life, Fashion & Beauty, Finance & Business, Food & Dining, Games, Health, History & Geography, Home & Hobbies, Industrial, Literature, Politics, Religion, Science, Math & Technology, Software, Sports & Fitness, Transportation, Travel & Tourism; - **format**: Academic Writing, Content Listing, Creative Writing, Customer Support Page, Discussion Forum / Comment Section, FAQs, Incomplete Content, Knowledge Article, Legal Notices, Listicle, News Article, Nonfiction Writing, Organizational About Page, Organizational Announcement, Personal About Page, Personal Blog, Product Page, Q&A Forum, Spam / Ads, Structured Data, Technical Writing, Transcript / Interview, Tutorial / How-To Guide, User Reviews. Use it to analyse, filter or rebalance multilingual pretraining data (for example FineWeb-2) by domain, as WebOrganizer did for English. One model returns both: 48 sigmoid outputs, `topic_*` and `format_*`; take the highest of each group. Built on [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base), Apache-2.0, ONNX included. It reads the first 512 tokens; no URL needed. A smaller, faster version is available as [Horizon-Labs/multilingual-web-topic-format-small](https://huggingface.co/Horizon-Labs/multilingual-web-topic-format-small). `onnx/model_quantized.onnx` (int8 embeddings, 641 MB) gives the same 0.5 decision as fp32 on all 48 outputs for 98.8% of 400 test pages. ## Usage ```python from transformers import pipeline clf = pipeline("text-classification", model="Horizon-Labs/multilingual-web-topic-format-base", top_k=None) s = {d["label"]: d["score"] for d in clf("Rezept: Für den Teig 500 g Mehl, 2 Eier und eine Prise Salz verrühren ...")[0]} topic = max((k for k in s if k.startswith("topic_")), key=s.get) fmt = max((k for k in s if k.startswith("format_")), key=s.get) print(topic, fmt) # topic_food_dining format_tutorial_how_to_guide ``` ## Evaluation - **65 languages**: 9,750 held-out FineWeb-2 pages (150 per language, never used for training) labelled with WebOrganizer's prompts by an independent judge, [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it) (a different model family from our teacher). - **English**: 3,000 pages of the test split of WebOrganizer's own annotations by Llama-3.1-405B ([topics](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8), [formats](https://huggingface.co/datasets/WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP8)); used only for evaluation. | model | 65 languages: topic accuracy | 65 languages: format accuracy | 65 languages: topic macro-F1 | English: topic accuracy | English: format accuracy | note | |---|---|---|---|---|---|---| | **this model** (308M) | 0.812 | 0.719 | 0.791 | 0.817 | 0.688 | 66 languages, topic + format in one model | | [Horizon-Labs/multilingual-web-topic-format-small](https://huggingface.co/Horizon-Labs/multilingual-web-topic-format-small) (141M) | 0.780 | 0.695 | 0.754 | 0.802 | 0.668 | | | [WebOrganizer/TopicClassifier-NoURL](https://huggingface.co/WebOrganizer/TopicClassifier-NoURL) + [FormatClassifier-NoURL](https://huggingface.co/WebOrganizer/FormatClassifier-NoURL) (2 x 140M) | 0.395 | 0.252 | 0.388 | 0.862 | 0.789 | English; trained on the Llama-3.1-405B labels of the English test (in-domain) | | Qwen3.8-27B, prompted (our teacher, 27B) | 0.885 | 0.773 | 0.876 | 0.864 | 0.743 | for reference | - The table shows the released checkpoint. Means over 2 training seeds (small: 65-language topic .781 / .780, format .697 / .695; base: .810 / .812, .715 / .719). - WebOrganizer's classifiers are English models trained on the same Llama-3.1-405B annotations as the English test, so they lead there; on other languages they fall to about 0.4 (topic) and 0.25 (format) agreement with the judge. - Caveat: the multilingual reference is an LLM. On English, the judge agrees with Llama-3.1-405B on 84% of topics and 71% of formats, and our teacher on 86% / 74%, but the judge and teacher agree with each other more, which favours this model on that test. Formats are more ambiguous than topics for every annotator. ## Training - **Pages**: 151,965 web pages, about 2,000 per language from [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) and 22,000 English pages from FineWeb (ODC-BY). - **Labels**: Qwen3.8-27B (Apache-2.0), prompted with WebOrganizer's topic and format instructions (Apache-2.0; without URL or demonstrations), gave a probability for each category; the model learns these as soft labels. The Llama-3.1-405B annotations were not used for training. - mmBERT-base, max length 512 tokens; checkpoint chosen on 7,579 held-out pages. Code: `code/` in this repository. ## Limitations - Labels come from LLMs, not people, and many pages fit several categories (WebOrganizer itself reports format as the harder taxonomy). - Only the text is used; WebOrganizer's URL-aware classifiers can use URL hints that this model does not see. - Rare categories in some languages (e.g. Transcript / Interview, Legal Notices) have little training data.