| --- |
| license: mit |
| base_model: FacebookAI/xlm-roberta-large |
| pipeline_tag: text-classification |
| language: |
| - multilingual |
| tags: |
| - news |
| - topic-classification |
| - multi-label |
| - xlm-roberta |
| library_name: transformers |
| --- |
| |
| # SNT News Classifier v0.6 |
|
|
| Multi-label news topic classifier: **12 top-level (L1) and 71 sub-level (L2) |
| categories**, built on `xlm-roberta-large` with two independent sigmoid heads. Both levels are |
| genuinely multi-label — an article about a trade deal can be `world` + `money_and_business` + |
| `politics` at the same time. Per-class decision thresholds (tuned on a held-out validation split) |
| ship inside `config.json`; `predict_labels()` applies them and falls back to argmax so no article |
| is ever left unlabeled. |
|
|
| Built by [Sweenk](https://sweenk.com) to categorize its news feed; released so others can use and |
| scrutinize it. |
|
|
| **Links:** [GitHub — model & training code](https://github.com/sweenk/snt-model) · |
| [GitHub — data pipeline](https://github.com/sweenk/snt-data) |
|
|
| ## At a glance |
|
|
| - **Multi-label at both levels** — a story can be `world` + `money_and_business` + `politics` at once, each above its own per-class tuned threshold. |
| - **0.847 L1 macro-F1** on 26,412 held-out articles; all 12 top-level categories clear a 0.65 per-class floor. |
| - **Multilingual encoder** — `xlm-roberta-large` (100 languages), fine-tuned on ~264K news articles. |
| - **Honest about its labels** — the gold is LLM-teacher-generated, not human-annotated, and the evaluation section says exactly what that means. |
|
|
| ## Quick start |
|
|
| ```python |
| from transformers import AutoModel, AutoTokenizer |
| |
| model = AutoModel.from_pretrained("sweenk/snt-classifier", trust_remote_code=True) |
| tok = AutoTokenizer.from_pretrained("sweenk/snt-classifier") |
| |
| enc = tok("OpenAI raises $6.6B. The startup announced its latest funding round...", |
| return_tensors="pt", truncation=True, max_length=512) |
| print(model.predict_labels(**enc)) |
| # [{'l1': [{'key': 'money_and_business', 'p': 0.99}, {'key': 'tech_and_ai', 'p': 0.98}], |
| # 'primary_l1': 'money_and_business', |
| # 'l2': [{'key': 'companies_and_industries', 'p': 0.92}, ...]}] |
| ``` |
|
|
| Input convention: `"{title}\n\n{body}"`, truncated at 512 tokens. The classifier was trained |
| on title+body; titles alone work but body text improves routing (the training prompt explicitly |
| prioritizes body over headline). |
|
|
| ## Taxonomy — 12 L1 / 71 L2 |
|
|
| | L1 category | # L2 | L2 sub-categories | |
| |---|---|---| |
| | `sports` | 12 | `american_football`, `baseball`, `basketball`, `college_sports`, `combat_sports`, `golf`, `hockey`, `motorsports`, `olympics`, `other_sports`, `soccer`, `tennis` | |
| | `politics` | 6 | `elections_and_campaigns`, `government_and_policy`, `immigration_and_borders`, `political_figures_and_scandals`, `social_issues_and_activism`, `state_and_local_politics` | |
| | `world` | 4 | `geopolitics_and_diplomacy`, `humanitarian_crises`, `terrorism_and_security`, `war_and_conflict` | |
| | `entertainment_and_pop_culture` | 7 | `books_and_arts`, `celebrities_and_gossip`, `gaming`, `internet_culture_and_creators`, `media_and_journalism`, `movies_and_tv`, `music` | |
| | `money_and_business` | 8 | `companies_and_industries`, `cost_of_living`, `crypto_and_fintech`, `housing_and_real_estate`, `macro_economy_and_rates`, `markets_and_investing`, `personal_finance`, `work_and_careers` | |
| | `crime_and_justice` | 4 | `courts_and_trials`, `crime_and_policing`, `scams_and_fraud`, `true_crime` | |
| | `tech_and_ai` | 5 | `artificial_intelligence`, `big_tech_and_startups`, `cybersecurity_and_privacy`, `gadgets_and_apps`, `screen_time_and_digital_life` | |
| | `science_and_space` | 4 | `archaeology_and_history`, `psychology_and_behavior`, `scientific_discoveries`, `space_and_astronomy` | |
| | `health_and_wellness` | 5 | `fitness_and_exercise`, `medical_and_public_health`, `mental_health`, `nutrition_and_diet`, `sleep_and_longevity` | |
| | `lifestyle` | 8 | `education_and_schools`, `faith_and_spirituality`, `fashion_and_beauty`, `food_and_drink`, `home_and_garden`, `parenting_and_family`, `relationships_and_dating`, `travel_and_places` | |
| | `weather_and_environment` | 5 | `climate_change`, `disasters_and_accidents`, `energy_and_climate_solutions`, `nature_and_wildlife`, `severe_weather` | |
| | `human_stories` | 3 | `animals_and_pets`, `good_news_and_kindness`, `offbeat_and_unusual` | |
|
|
| The full machine-readable taxonomy (`l1_keys`, `l2_keys`, `l2_parent`, per-class thresholds) is in |
| `config.json`. |
|
|
| ## Evaluation — our own numbers, stated plainly |
|
|
| Held-out test split: **26,412 articles** (a 10% slice of the labeled corpus, |
| stratified by primary L1). "Tuned" = per-class thresholds optimized on the *validation* split, |
| then applied unchanged to test. |
|
|
| | Metric | @0.5 threshold | tuned thresholds | |
| |---|---|---| |
| | L1 macro F1 | 0.817 | **0.847** | |
| | L1 micro F1 | 0.831 | **0.857** | |
| | L1 primary accuracy (argmax) | 0.833 | — | |
| | L2 macro F1 | 0.647 | 0.685 | |
| | L2 micro F1 | 0.753 | 0.754 | |
|
|
| ### Per-class L1 F1 (test) |
|
|
| | L1 | F1 @0.5 | F1 tuned | threshold | |
| |---|---|---|---| |
| | `sports` | 0.941 | 0.954 | 0.850 | |
| | `politics` | 0.845 | 0.859 | 0.750 | |
| | `world` | 0.783 | 0.827 | 0.850 | |
| | `entertainment_and_pop_culture` | 0.887 | 0.898 | 0.850 | |
| | `money_and_business` | 0.786 | 0.825 | 0.950 | |
| | `crime_and_justice` | 0.823 | 0.857 | 0.950 | |
| | `tech_and_ai` | 0.802 | 0.857 | 0.950 | |
| | `science_and_space` | 0.802 | 0.833 | 0.950 | |
| | `health_and_wellness` | 0.821 | 0.860 | 0.900 | |
| | `lifestyle` | 0.865 | 0.867 | 0.700 | |
| | `weather_and_environment` | 0.816 | 0.857 | 0.950 | |
| | `human_stories` | 0.632 | 0.671 | 0.900 | |
|
|
| ### What you should know before trusting these numbers |
|
|
| Read this section — it is the honest part. |
|
|
| - **The gold labels are model-assisted, not human-annotated.** The corpus (~278K articles from |
| HuffPost archives, CommonCrawl News, daily.dev, and Sweenk production) was labeled by a |
| mechanical migration from an earlier taxonomy plus multiple passes of a Claude Sonnet teacher |
| with a rule-based prompt, spot-audited by humans (QA gates at 70–87% agreement on sampled |
| batches). Test F1 therefore measures agreement with an LLM teacher, not with human ground truth. |
| - **Class imbalance is real (~21x).** `politics`/`lifestyle`/`entertainment` have ~44-47K training |
| rows; `science_and_space` ~2.2K and `tech_and_ai` ~3.7K. Training compensates with per-class |
| `pos_weight` (clamped at 10) and caps the majority classes at 25K primary-label train rows so |
| they don't swamp the rare ones. The weakest class here is `human_stories` (F1 0.671) — |
| the fuzziest category by construction. |
| - **Labels were corrected over time, so F1 is not comparable across releases.** v0.5.1 re-labeled |
| ~1,500 systematically mislabeled rows with route-by-cause rules (accidents by cause, terror → |
| crime, pharma earnings → money); on those rows agreement with the corrected gold went |
| 25.9% → 75.3%. Because the gold labels themselves changed between releases, aggregate F1 deltas |
| are not like-for-like — treat each release's numbers as self-referential. |
| - **Multilingual ability is inherited, not measured.** The encoder is XLM-R, but nearly all |
| training articles are English. Expect degraded (unquantified) quality on non-English news. |
| - **L3 (named topics / entities) is not part of this model** — Sweenk handles that downstream with |
| a separate extraction step. |
|
|
| ## Architecture |
|
|
| `xlm-roberta-large` encoder → CLS pooling → dropout(0.1) → two parallel linear heads |
| (L1: 12 logits, L2: 71 logits), both sigmoid. Trained 3 epochs, BCE loss with |
| per-class `pos_weight` (L1) and loss weights L1:1.0 / L2:2.0, bf16 autocast, gradient checkpointing. |
| Inference upcasts logits to fp32 before sigmoid (bf16 sigmoid saturates above logit ~6.2, which |
| collapses co-confident multi-label pairs). |
|
|
| ## Versions |
|
|
| | Version | What changed | |
| |---|---| |
| | v0.5 | First multi-label release (12 L1 / 71 L2, dual sigmoid heads) | |
| | v0.5.1 | Corrective retrain: route-accidents-by-cause, terror→crime, earnings→money, govt-personnel→politics, wildlife→weather, body-over-headline; ~1,500 corrected labels; re-tuned thresholds | |
| | v0.6 | Retrain on the corrected corpus + ~23K newly teacher-labeled articles (Sweenk production + daily.dev science/tech); majority-class capping (25K/class) on top of `pos_weight`; re-tuned thresholds | |
|
|
| ## License & attribution |
|
|
| Model weights: MIT. Base model: [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) (MIT). |
| The training corpus contains article text from public news sources and is **not** redistributed |
| with this model. |
|
|
