SNT classifier v0.5.1

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,141 @@
+---
+license: mit
+base_model: FacebookAI/xlm-roberta-large
+pipeline_tag: text-classification
+language:
+- multilingual
+tags:
+- news
+- topic-classification
+- multi-label
+- xlm-roberta
+library_name: transformers
+---
+
+# SNT News Classifier v0.5.1
+
+Multi-label news topic classifier: **12 top-level (L1) and 71 sub-level (L2)
+categories**, built on `xlm-roberta-large` with two independent sigmoid heads. Both levels are
+genuinely multi-label — an article about a trade deal can be `world` + `money_and_business` +
+`politics` at the same time. Per-class decision thresholds (tuned on a held-out validation split)
+ship inside `config.json`; `predict_labels()` applies them and falls back to argmax so no article
+is ever left unlabeled.
+
+Built by [Sweenk](https://sweenk.com) to categorize its news feed; released so others can use and
+scrutinize it.
+
+## Quick start
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+model = AutoModel.from_pretrained("sweenk/snt-classifier", trust_remote_code=True)
+tok = AutoTokenizer.from_pretrained("sweenk/snt-classifier")
+
+enc = tok("OpenAI raises $6.6B. The startup announced its latest funding round...",
+          return_tensors="pt", truncation=True, max_length=512)
+print(model.predict_labels(**enc))
+# [{'l1': [{'key': 'money_and_business', 'p': 0.99}, {'key': 'tech_and_ai', 'p': 0.98}],
+#    'primary_l1': 'money_and_business',
+#    'l2': [{'key': 'companies_and_industries', 'p': 0.92}, ...]}]
+```
+
+Input convention: `"{title}\n\n{body}"`, truncated at 512 tokens. The classifier was trained
+on title+body; titles alone work but body text improves routing (the training prompt explicitly
+prioritizes body over headline).
+
+## Taxonomy — 12 L1 / 71 L2
+
+| L1 category | # L2 | L2 sub-categories |
+|---|---|---|
+| `sports` | 12 | `american_football`, `baseball`, `basketball`, `college_sports`, `combat_sports`, `golf`, `hockey`, `motorsports`, `olympics`, `other_sports`, `soccer`, `tennis` |
+| `politics` | 6 | `elections_and_campaigns`, `government_and_policy`, `immigration_and_borders`, `political_figures_and_scandals`, `social_issues_and_activism`, `state_and_local_politics` |
+| `world` | 4 | `geopolitics_and_diplomacy`, `humanitarian_crises`, `terrorism_and_security`, `war_and_conflict` |
+| `entertainment_and_pop_culture` | 7 | `books_and_arts`, `celebrities_and_gossip`, `gaming`, `internet_culture_and_creators`, `media_and_journalism`, `movies_and_tv`, `music` |
+| `money_and_business` | 8 | `companies_and_industries`, `cost_of_living`, `crypto_and_fintech`, `housing_and_real_estate`, `macro_economy_and_rates`, `markets_and_investing`, `personal_finance`, `work_and_careers` |
+| `crime_and_justice` | 4 | `courts_and_trials`, `crime_and_policing`, `scams_and_fraud`, `true_crime` |
+| `tech_and_ai` | 5 | `artificial_intelligence`, `big_tech_and_startups`, `cybersecurity_and_privacy`, `gadgets_and_apps`, `screen_time_and_digital_life` |
+| `science_and_space` | 4 | `archaeology_and_history`, `psychology_and_behavior`, `scientific_discoveries`, `space_and_astronomy` |
+| `health_and_wellness` | 5 | `fitness_and_exercise`, `medical_and_public_health`, `mental_health`, `nutrition_and_diet`, `sleep_and_longevity` |
+| `lifestyle` | 8 | `education_and_schools`, `faith_and_spirituality`, `fashion_and_beauty`, `food_and_drink`, `home_and_garden`, `parenting_and_family`, `relationships_and_dating`, `travel_and_places` |
+| `weather_and_environment` | 5 | `climate_change`, `disasters_and_accidents`, `energy_and_climate_solutions`, `nature_and_wildlife`, `severe_weather` |
+| `human_stories` | 3 | `animals_and_pets`, `good_news_and_kindness`, `offbeat_and_unusual` |
+
+The full machine-readable taxonomy (`l1_keys`, `l2_keys`, `l2_parent`, per-class thresholds) is in
+`config.json`.
+
+## Evaluation — our own numbers, stated plainly
+
+Held-out test split: **24,176 articles** (10% of the 241,757 train-eligible corpus
+rows). "Tuned" = per-class thresholds optimized on the *validation* split, then applied unchanged
+to test.
+
+| Metric | @0.5 threshold | tuned thresholds |
+|---|---|---|
+| L1 macro F1 | 0.803 | **0.824** |
+| L1 micro F1 | 0.832 | **0.854** |
+| L1 primary accuracy (argmax) | 0.832 | — |
+| L2 macro F1 | 0.624 | 0.671 |
+| L2 micro F1 | 0.760 | 0.757 |
+
+### Per-class L1 F1 (test)
+
+| L1 | F1 @0.5 | F1 tuned | threshold |
+|---|---|---|---|
+| `sports` | 0.928 | 0.936 | 0.900 |
+| `politics` | 0.849 | 0.861 | 0.700 |
+| `world` | 0.781 | 0.827 | 0.900 |
+| `entertainment_and_pop_culture` | 0.895 | 0.904 | 0.800 |
+| `money_and_business` | 0.772 | 0.810 | 0.900 |
+| `crime_and_justice` | 0.821 | 0.846 | 0.900 |
+| `tech_and_ai` | 0.721 | 0.726 | 0.950 |
+| `science_and_space` | 0.731 | 0.754 | 0.800 |
+| `health_and_wellness` | 0.830 | 0.859 | 0.900 |
+| `lifestyle` | 0.872 | 0.878 | 0.700 |
+| `weather_and_environment` | 0.793 | 0.817 | 0.850 |
+| `human_stories` | 0.646 | 0.672 | 0.900 |
+
+### What you should know before trusting these numbers
+
+Read this section — it is the honest part.
+
+- **The gold labels are model-assisted, not human-annotated.** The corpus (255K articles from
+  HuffPost archives, CommonCrawl News, daily.dev, and Sweenk production) was labeled by a
+  mechanical migration from an earlier taxonomy plus multiple passes of a Claude Sonnet teacher
+  with a rule-based prompt, spot-audited by humans (QA gates at 70–87% agreement on sampled
+  batches). Test F1 therefore measures agreement with an LLM teacher, not with human ground truth.
+- **Class imbalance is real (~21x).** `politics`/`lifestyle`/`entertainment` have ~44-47K training
+  rows; `science_and_space` has ~2.2K and `tech_and_ai` ~3.7K. Training used per-class
+  `pos_weight` (clamped at 10) to compensate. The weakest class is `human_stories`
+  (F1 0.672) — it is the fuzziest category by construction.
+- **v0.5.1 is a corrective retrain.** A 986-article human validation of the previous release
+  found errors clustered in specific boundaries (accidents dumped into `weather_and_environment`,
+  terror attacks into `world`, pharma earnings into `health_and_wellness`). ~1,500 mislabeled rows
+  were re-labeled with corrected routing rules and the model retrained. On those corrected rows the
+  previous model matches the corrected label 25.9% of the time; this model matches 75.3%. Aggregate
+  F1 is *not* directly comparable across releases because the gold labels themselves were corrected.
+- **Multilingual ability is inherited, not measured.** The encoder is XLM-R, but nearly all
+  training articles are English. Expect degraded (unquantified) quality on non-English news.
+- **L3 (named topics / entities) is not part of this model** — Sweenk handles that downstream with
+  a separate extraction step.
+
+## Architecture
+
+`xlm-roberta-large` encoder → CLS pooling → dropout(0.1) → two parallel linear heads
+(L1: 12 logits, L2: 71 logits), both sigmoid. Trained 3 epochs, BCE loss with
+per-class `pos_weight` (L1) and loss weights L1:1.0 / L2:2.0, bf16 autocast, gradient checkpointing.
+Inference upcasts logits to fp32 before sigmoid (bf16 sigmoid saturates above logit ~6.2, which
+collapses co-confident multi-label pairs).
+
+## Versions
+
+| Version | What changed |
+|---|---|
+| v0.5 | First multi-label release (12 L1 / 71 L2, dual sigmoid heads) |
+| v0.5.1 | Corrective retrain: route-accidents-by-cause, terror→crime, earnings→money, govt-personnel→politics, wildlife→weather, body-over-headline; ~1,500 corrected labels; re-tuned thresholds |
+
+## License & attribution
+
+Model weights: MIT. Base model: [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) (MIT).
+The training corpus contains article text from public news sources and is **not** redistributed
+with this model.

config.json

@@ -0,0 +1,261 @@
+{
+  "architectures": [
+    "SNTForNewsClassification"
+  ],
+  "auto_map": {
+    "AutoModel": "modeling_snt.SNTForNewsClassification"
+  },
+  "dropout": 0.1,
+  "dtype": "float32",
+  "encoder_name": "xlm-roberta-large",
+  "l1_keys": [
+    "sports",
+    "politics",
+    "world",
+    "entertainment_and_pop_culture",
+    "money_and_business",
+    "crime_and_justice",
+    "tech_and_ai",
+    "science_and_space",
+    "health_and_wellness",
+    "lifestyle",
+    "weather_and_environment",
+    "human_stories"
+  ],
+  "l1_thresholds": {
+    "crime_and_justice": 0.9,
+    "entertainment_and_pop_culture": 0.8,
+    "health_and_wellness": 0.9,
+    "human_stories": 0.9,
+    "lifestyle": 0.7,
+    "money_and_business": 0.9,
+    "politics": 0.7,
+    "science_and_space": 0.8,
+    "sports": 0.9,
+    "tech_and_ai": 0.95,
+    "weather_and_environment": 0.85,
+    "world": 0.9
+  },
+  "l2_keys": [
+    "soccer",
+    "american_football",
+    "college_sports",
+    "basketball",
+    "baseball",
+    "hockey",
+    "combat_sports",
+    "motorsports",
+    "tennis",
+    "golf",
+    "olympics",
+    "other_sports",
+    "elections_and_campaigns",
+    "government_and_policy",
+    "political_figures_and_scandals",
+    "immigration_and_borders",
+    "social_issues_and_activism",
+    "state_and_local_politics",
+    "war_and_conflict",
+    "geopolitics_and_diplomacy",
+    "terrorism_and_security",
+    "humanitarian_crises",
+    "celebrities_and_gossip",
+    "movies_and_tv",
+    "music",
+    "gaming",
+    "internet_culture_and_creators",
+    "books_and_arts",
+    "media_and_journalism",
+    "cost_of_living",
+    "personal_finance",
+    "housing_and_real_estate",
+    "work_and_careers",
+    "markets_and_investing",
+    "crypto_and_fintech",
+    "companies_and_industries",
+    "macro_economy_and_rates",
+    "crime_and_policing",
+    "courts_and_trials",
+    "scams_and_fraud",
+    "true_crime",
+    "artificial_intelligence",
+    "big_tech_and_startups",
+    "gadgets_and_apps",
+    "cybersecurity_and_privacy",
+    "screen_time_and_digital_life",
+    "space_and_astronomy",
+    "scientific_discoveries",
+    "psychology_and_behavior",
+    "archaeology_and_history",
+    "mental_health",
+    "fitness_and_exercise",
+    "nutrition_and_diet",
+    "sleep_and_longevity",
+    "medical_and_public_health",
+    "food_and_drink",
+    "travel_and_places",
+    "fashion_and_beauty",
+    "home_and_garden",
+    "parenting_and_family",
+    "relationships_and_dating",
+    "education_and_schools",
+    "faith_and_spirituality",
+    "severe_weather",
+    "disasters_and_accidents",
+    "climate_change",
+    "nature_and_wildlife",
+    "energy_and_climate_solutions",
+    "good_news_and_kindness",
+    "offbeat_and_unusual",
+    "animals_and_pets"
+  ],
+  "l2_parent": {
+    "american_football": "sports",
+    "animals_and_pets": "human_stories",
+    "archaeology_and_history": "science_and_space",
+    "artificial_intelligence": "tech_and_ai",
+    "baseball": "sports",
+    "basketball": "sports",
+    "big_tech_and_startups": "tech_and_ai",
+    "books_and_arts": "entertainment_and_pop_culture",
+    "celebrities_and_gossip": "entertainment_and_pop_culture",
+    "climate_change": "weather_and_environment",
+    "college_sports": "sports",
+    "combat_sports": "sports",
+    "companies_and_industries": "money_and_business",
+    "cost_of_living": "money_and_business",
+    "courts_and_trials": "crime_and_justice",
+    "crime_and_policing": "crime_and_justice",
+    "crypto_and_fintech": "money_and_business",
+    "cybersecurity_and_privacy": "tech_and_ai",
+    "disasters_and_accidents": "weather_and_environment",
+    "education_and_schools": "lifestyle",
+    "elections_and_campaigns": "politics",
+    "energy_and_climate_solutions": "weather_and_environment",
+    "faith_and_spirituality": "lifestyle",
+    "fashion_and_beauty": "lifestyle",
+    "fitness_and_exercise": "health_and_wellness",
+    "food_and_drink": "lifestyle",
+    "gadgets_and_apps": "tech_and_ai",
+    "gaming": "entertainment_and_pop_culture",
+    "geopolitics_and_diplomacy": "world",
+    "golf": "sports",
+    "good_news_and_kindness": "human_stories",
+    "government_and_policy": "politics",
+    "hockey": "sports",
+    "home_and_garden": "lifestyle",
+    "housing_and_real_estate": "money_and_business",
+    "humanitarian_crises": "world",
+    "immigration_and_borders": "politics",
+    "internet_culture_and_creators": "entertainment_and_pop_culture",
+    "macro_economy_and_rates": "money_and_business",
+    "markets_and_investing": "money_and_business",
+    "media_and_journalism": "entertainment_and_pop_culture",
+    "medical_and_public_health": "health_and_wellness",
+    "mental_health": "health_and_wellness",
+    "motorsports": "sports",
+    "movies_and_tv": "entertainment_and_pop_culture",
+    "music": "entertainment_and_pop_culture",
+    "nature_and_wildlife": "weather_and_environment",
+    "nutrition_and_diet": "health_and_wellness",
+    "offbeat_and_unusual": "human_stories",
+    "olympics": "sports",
+    "other_sports": "sports",
+    "parenting_and_family": "lifestyle",
+    "personal_finance": "money_and_business",
+    "political_figures_and_scandals": "politics",
+    "psychology_and_behavior": "science_and_space",
+    "relationships_and_dating": "lifestyle",
+    "scams_and_fraud": "crime_and_justice",
+    "scientific_discoveries": "science_and_space",
+    "screen_time_and_digital_life": "tech_and_ai",
+    "severe_weather": "weather_and_environment",
+    "sleep_and_longevity": "health_and_wellness",
+    "soccer": "sports",
+    "social_issues_and_activism": "politics",
+    "space_and_astronomy": "science_and_space",
+    "state_and_local_politics": "politics",
+    "tennis": "sports",
+    "terrorism_and_security": "world",
+    "travel_and_places": "lifestyle",
+    "true_crime": "crime_and_justice",
+    "war_and_conflict": "world",
+    "work_and_careers": "money_and_business"
+  },
+  "l2_thresholds": {
+    "american_football": 0.55,
+    "animals_and_pets": 0.5,
+    "archaeology_and_history": 0.05,
+    "artificial_intelligence": 0.3,
+    "baseball": 0.3,
+    "basketball": 0.55,
+    "big_tech_and_startups": 0.45,
+    "books_and_arts": 0.55,
+    "celebrities_and_gossip": 0.4,
+    "climate_change": 0.5,
+    "college_sports": 0.5,
+    "combat_sports": 0.55,
+    "companies_and_industries": 0.45,
+    "cost_of_living": 0.05,
+    "courts_and_trials": 0.3,
+    "crime_and_policing": 0.45,
+    "crypto_and_fintech": 0.2,
+    "cybersecurity_and_privacy": 0.25,
+    "disasters_and_accidents": 0.4,
+    "education_and_schools": 0.45,
+    "elections_and_campaigns": 0.45,
+    "energy_and_climate_solutions": 0.45,
+    "faith_and_spirituality": 0.35,
+    "fashion_and_beauty": 0.45,
+    "fitness_and_exercise": 0.35,
+    "food_and_drink": 0.4,
+    "gadgets_and_apps": 0.55,
+    "gaming": 0.65,
+    "geopolitics_and_diplomacy": 0.45,
+    "golf": 0.35,
+    "good_news_and_kindness": 0.25,
+    "government_and_policy": 0.4,
+    "hockey": 0.15,
+    "home_and_garden": 0.5,
+    "housing_and_real_estate": 0.3,
+    "humanitarian_crises": 0.25,
+    "immigration_and_borders": 0.45,
+    "internet_culture_and_creators": 0.05,
+    "macro_economy_and_rates": 0.5,
+    "markets_and_investing": 0.45,
+    "media_and_journalism": 0.3,
+    "medical_and_public_health": 0.45,
+    "mental_health": 0.3,
+    "motorsports": 0.55,
+    "movies_and_tv": 0.55,
+    "music": 0.5,
+    "nature_and_wildlife": 0.5,
+    "nutrition_and_diet": 0.45,
+    "offbeat_and_unusual": 0.3,
+    "olympics": 0.6,
+    "other_sports": 0.35,
+    "parenting_and_family": 0.4,
+    "personal_finance": 0.1,
+    "political_figures_and_scandals": 0.25,
+    "psychology_and_behavior": 0.05,
+    "relationships_and_dating": 0.5,
+    "scams_and_fraud": 0.25,
+    "scientific_discoveries": 0.3,
+    "screen_time_and_digital_life": 0.25,
+    "severe_weather": 0.45,
+    "sleep_and_longevity": 0.05,
+    "soccer": 0.4,
+    "social_issues_and_activism": 0.25,
+    "space_and_astronomy": 0.4,
+    "state_and_local_politics": 0.1,
+    "tennis": 0.35,
+    "terrorism_and_security": 0.45,
+    "travel_and_places": 0.4,
+    "true_crime": 0.05,
+    "war_and_conflict": 0.55,
+    "work_and_careers": 0.55
+  },
+  "model_type": "snt_classifier",
+  "snt_version": "v0.5.1",
+  "transformers_version": "5.8.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:911b368abfd5170be7b8ab73f6bb1b4b9a323e9dba76bb3069a10d9f477a0cce
+size 2239950948

modeling_snt.py

@@ -0,0 +1,101 @@
+"""SNT news classifier — HF-native wrapper (uploaded to the HF repo as-is).
+
+Usage:
+    from transformers import AutoModel, AutoTokenizer
+    model = AutoModel.from_pretrained("sweenk/snt-classifier", trust_remote_code=True)
+    tok = AutoTokenizer.from_pretrained("sweenk/snt-classifier")
+    enc = tok("Title. Body...", return_tensors="pt", truncation=True, max_length=512)
+    labels = model.predict_labels(**enc)
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+from transformers import AutoModel, PretrainedConfig, PreTrainedModel
+
+
+class SNTConfig(PretrainedConfig):
+    model_type = "snt_classifier"
+
+    def __init__(
+        self,
+        encoder_name: str = "xlm-roberta-large",
+        l1_keys: list[str] | None = None,
+        l2_keys: list[str] | None = None,
+        l2_parent: dict[str, str] | None = None,
+        l1_thresholds: dict[str, float] | None = None,
+        l2_thresholds: dict[str, float] | None = None,
+        snt_version: str = "v0.5.1",
+        dropout: float = 0.1,
+        **kwargs,
+    ):
+        self.encoder_name = encoder_name
+        self.l1_keys = l1_keys or []
+        self.l2_keys = l2_keys or []
+        self.l2_parent = l2_parent or {}
+        self.l1_thresholds = l1_thresholds or {}
+        self.l2_thresholds = l2_thresholds or {}
+        self.snt_version = snt_version
+        self.dropout = dropout
+        super().__init__(**kwargs)
+
+    @property
+    def n_l1(self) -> int:
+        return len(self.l1_keys)
+
+    @property
+    def n_l2(self) -> int:
+        return len(self.l2_keys)
+
+
+class SNTForNewsClassification(PreTrainedModel):
+    config_class = SNTConfig
+
+    def __init__(self, config: SNTConfig):
+        super().__init__(config)
+        # Attribute names MUST match DualHeadModel so state_dicts load 1:1.
+        self.encoder = AutoModel.from_pretrained(config.encoder_name)
+        hidden = self.encoder.config.hidden_size
+        self.dropout = nn.Dropout(config.dropout)
+        self.head_top = nn.Linear(hidden, config.n_l1)
+        self.head_sub = nn.Linear(hidden, config.n_l2)
+
+    def forward(self, input_ids, attention_mask, **kwargs):
+        out = self.encoder(input_ids=input_ids, attention_mask=attention_mask)
+        pooled = self.dropout(out.last_hidden_state[:, 0, :])
+        return {"l1_logits": self.head_top(pooled), "l2_logits": self.head_sub(pooled)}
+
+    @torch.no_grad()
+    def predict_labels(self, input_ids, attention_mask, **kwargs) -> list[dict]:
+        """Thresholded multi-label prediction, one dict per batch row.
+
+        Logits are upcast to fp32 before sigmoid — bf16 sigmoid saturates to
+        exactly 1.0 above logit ~6.2, collapsing co-confident categories.
+        """
+        out = self.forward(input_ids, attention_mask)
+        l1_probs = torch.sigmoid(out["l1_logits"].float())
+        l2_probs = torch.sigmoid(out["l2_logits"].float())
+        results = []
+        for row in range(l1_probs.shape[0]):
+            l1 = sorted(
+                (
+                    {"key": k, "p": round(float(p), 4)}
+                    for k, p in zip(self.config.l1_keys, l1_probs[row].tolist())
+                    if p >= self.config.l1_thresholds.get(k, 0.5)
+                ),
+                key=lambda hit: -hit["p"],
+            )
+            if not l1:  # argmax fallback — never return unlabeled
+                idx = int(l1_probs[row].argmax())
+                l1 = [{"key": self.config.l1_keys[idx], "p": round(float(l1_probs[row][idx]), 4)}]
+            l2 = sorted(
+                (
+                    {"key": k, "p": round(float(p), 4)}
+                    for k, p in zip(self.config.l2_keys, l2_probs[row].tolist())
+                    if p >= self.config.l2_thresholds.get(k, 0.5)
+                ),
+                key=lambda hit: -hit["p"],
+            )
+            results.append({"l1": l1, "primary_l1": l1[0]["key"], "l2": l2})
+        return results

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acbd420e2269cdc1ef45332d3d5c418be4aef6b8cb5a0b7ccae0893485307153
+size 17098086

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "add_prefix_space": true,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}