SNT classifier v0.6

README.md

@@ -12,7 +12,7 @@ tags:
 library_name: transformers
 ---
 
-# SNT News Classifier v0.5.1
+# SNT News Classifier v0.6
 
 Multi-label news topic classifier: **12 top-level (L1) and 71 sub-level (L2)
 categories**, built on `xlm-roberta-large` with two independent sigmoid heads. Both levels are
@@ -24,6 +24,10 @@ is ever left unlabeled.
 Built by [Sweenk](https://sweenk.com) to categorize its news feed; released so others can use and
 scrutinize it.
 
+**Links:** [GitHub — model & training code](https://github.com/sweenk/snt-model) ·
+[GitHub — data pipeline](https://github.com/sweenk/snt-data) ·
+[Live API](https://taxonomy.lasharela.com)
+
 ## Quick start
 
 ```python
@@ -66,54 +70,54 @@ The full machine-readable taxonomy (`l1_keys`, `l2_keys`, `l2_parent`, per-class
 
 ## Evaluation — our own numbers, stated plainly
 
-Held-out test split: **24,176 articles** (10% of the 241,757 train-eligible corpus
-rows). "Tuned" = per-class thresholds optimized on the *validation* split, then applied unchanged
-to test.
+Held-out test split: **26,412 articles** (a 10% slice of the labeled corpus,
+stratified by primary L1). "Tuned" = per-class thresholds optimized on the *validation* split,
+then applied unchanged to test.
 
 | Metric | @0.5 threshold | tuned thresholds |
 |---|---|---|
-| L1 macro F1 | 0.803 | **0.824** |
-| L1 micro F1 | 0.832 | **0.854** |
-| L1 primary accuracy (argmax) | 0.832 | — |
-| L2 macro F1 | 0.624 | 0.671 |
-| L2 micro F1 | 0.760 | 0.757 |
+| L1 macro F1 | 0.817 | **0.847** |
+| L1 micro F1 | 0.831 | **0.857** |
+| L1 primary accuracy (argmax) | 0.833 | — |
+| L2 macro F1 | 0.647 | 0.685 |
+| L2 micro F1 | 0.753 | 0.754 |
 
 ### Per-class L1 F1 (test)
 
 | L1 | F1 @0.5 | F1 tuned | threshold |
 |---|---|---|---|
-| `sports` | 0.928 | 0.936 | 0.900 |
-| `politics` | 0.849 | 0.861 | 0.700 |
-| `world` | 0.781 | 0.827 | 0.900 |
-| `entertainment_and_pop_culture` | 0.895 | 0.904 | 0.800 |
-| `money_and_business` | 0.772 | 0.810 | 0.900 |
-| `crime_and_justice` | 0.821 | 0.846 | 0.900 |
-| `tech_and_ai` | 0.721 | 0.726 | 0.950 |
-| `science_and_space` | 0.731 | 0.754 | 0.800 |
-| `health_and_wellness` | 0.830 | 0.859 | 0.900 |
-| `lifestyle` | 0.872 | 0.878 | 0.700 |
-| `weather_and_environment` | 0.793 | 0.817 | 0.850 |
-| `human_stories` | 0.646 | 0.672 | 0.900 |
+| `sports` | 0.941 | 0.954 | 0.850 |
+| `politics` | 0.845 | 0.859 | 0.750 |
+| `world` | 0.783 | 0.827 | 0.850 |
+| `entertainment_and_pop_culture` | 0.887 | 0.898 | 0.850 |
+| `money_and_business` | 0.786 | 0.825 | 0.950 |
+| `crime_and_justice` | 0.823 | 0.857 | 0.950 |
+| `tech_and_ai` | 0.802 | 0.857 | 0.950 |
+| `science_and_space` | 0.802 | 0.833 | 0.950 |
+| `health_and_wellness` | 0.821 | 0.860 | 0.900 |
+| `lifestyle` | 0.865 | 0.867 | 0.700 |
+| `weather_and_environment` | 0.816 | 0.857 | 0.950 |
+| `human_stories` | 0.632 | 0.671 | 0.900 |
 
 ### What you should know before trusting these numbers
 
 Read this section — it is the honest part.
 
-- **The gold labels are model-assisted, not human-annotated.** The corpus (255K articles from
+- **The gold labels are model-assisted, not human-annotated.** The corpus (~278K articles from
   HuffPost archives, CommonCrawl News, daily.dev, and Sweenk production) was labeled by a
   mechanical migration from an earlier taxonomy plus multiple passes of a Claude Sonnet teacher
   with a rule-based prompt, spot-audited by humans (QA gates at 70–87% agreement on sampled
   batches). Test F1 therefore measures agreement with an LLM teacher, not with human ground truth.
 - **Class imbalance is real (~21x).** `politics`/`lifestyle`/`entertainment` have ~44-47K training
-  rows; `science_and_space` has ~2.2K and `tech_and_ai` ~3.7K. Training used per-class
-  `pos_weight` (clamped at 10) to compensate. The weakest class is `human_stories`
-  (F1 0.672) — it is the fuzziest category by construction.
-- **v0.5.1 is a corrective retrain.** A 986-article human validation of the previous release
-  found errors clustered in specific boundaries (accidents dumped into `weather_and_environment`,
-  terror attacks into `world`, pharma earnings into `health_and_wellness`). ~1,500 mislabeled rows
-  were re-labeled with corrected routing rules and the model retrained. On those corrected rows the
-  previous model matches the corrected label 25.9% of the time; this model matches 75.3%. Aggregate
-  F1 is *not* directly comparable across releases because the gold labels themselves were corrected.
+  rows; `science_and_space` ~2.2K and `tech_and_ai` ~3.7K. Training compensates with per-class
+  `pos_weight` (clamped at 10) and caps the majority classes at 25K primary-label train rows so
+  they don't swamp the rare ones. The weakest class here is `human_stories` (F1 0.671) —
+  the fuzziest category by construction.
+- **Labels were corrected over time, so F1 is not comparable across releases.** v0.5.1 re-labeled
+  ~1,500 systematically mislabeled rows with route-by-cause rules (accidents by cause, terror →
+  crime, pharma earnings → money); on those rows agreement with the corrected gold went
+  25.9% → 75.3%. Because the gold labels themselves changed between releases, aggregate F1 deltas
+  are not like-for-like — treat each release's numbers as self-referential.
 - **Multilingual ability is inherited, not measured.** The encoder is XLM-R, but nearly all
   training articles are English. Expect degraded (unquantified) quality on non-English news.
 - **L3 (named topics / entities) is not part of this model** — Sweenk handles that downstream with
@@ -133,6 +137,7 @@ collapses co-confident multi-label pairs).
 |---|---|
 | v0.5 | First multi-label release (12 L1 / 71 L2, dual sigmoid heads) |
 | v0.5.1 | Corrective retrain: route-accidents-by-cause, terror→crime, earnings→money, govt-personnel→politics, wildlife→weather, body-over-headline; ~1,500 corrected labels; re-tuned thresholds |
+| v0.6 | Retrain on the corrected corpus + ~23K newly teacher-labeled articles (Sweenk production + daily.dev science/tech); majority-class capping (25K/class) on top of `pos_weight`; re-tuned thresholds |
 
 ## License & attribution
 

config.json

@@ -24,18 +24,18 @@
     "human_stories"
   ],
   "l1_thresholds": {
-    "crime_and_justice": 0.9,
-    "entertainment_and_pop_culture": 0.8,
+    "crime_and_justice": 0.95,
+    "entertainment_and_pop_culture": 0.85,
     "health_and_wellness": 0.9,
     "human_stories": 0.9,
     "lifestyle": 0.7,
-    "money_and_business": 0.9,
-    "politics": 0.7,
-    "science_and_space": 0.8,
-    "sports": 0.9,
+    "money_and_business": 0.95,
+    "politics": 0.75,
+    "science_and_space": 0.95,
+    "sports": 0.85,
     "tech_and_ai": 0.95,
-    "weather_and_environment": 0.85,
-    "world": 0.9
+    "weather_and_environment": 0.95,
+    "world": 0.85
   },
   "l2_keys": [
     "soccer",
@@ -184,79 +184,79 @@
     "work_and_careers": "money_and_business"
   },
   "l2_thresholds": {
-    "american_football": 0.55,
-    "animals_and_pets": 0.5,
-    "archaeology_and_history": 0.05,
-    "artificial_intelligence": 0.3,
-    "baseball": 0.3,
-    "basketball": 0.55,
-    "big_tech_and_startups": 0.45,
-    "books_and_arts": 0.55,
-    "celebrities_and_gossip": 0.4,
-    "climate_change": 0.5,
-    "college_sports": 0.5,
-    "combat_sports": 0.55,
-    "companies_and_industries": 0.45,
+    "american_football": 0.5,
+    "animals_and_pets": 0.55,
+    "archaeology_and_history": 0.2,
+    "artificial_intelligence": 0.45,
+    "baseball": 0.35,
+    "basketball": 0.45,
+    "big_tech_and_startups": 0.3,
+    "books_and_arts": 0.3,
+    "celebrities_and_gossip": 0.45,
+    "climate_change": 0.35,
+    "college_sports": 0.35,
+    "combat_sports": 0.35,
+    "companies_and_industries": 0.4,
     "cost_of_living": 0.05,
-    "courts_and_trials": 0.3,
-    "crime_and_policing": 0.45,
-    "crypto_and_fintech": 0.2,
-    "cybersecurity_and_privacy": 0.25,
+    "courts_and_trials": 0.4,
+    "crime_and_policing": 0.35,
+    "crypto_and_fintech": 0.15,
+    "cybersecurity_and_privacy": 0.35,
     "disasters_and_accidents": 0.4,
-    "education_and_schools": 0.45,
-    "elections_and_campaigns": 0.45,
+    "education_and_schools": 0.3,
+    "elections_and_campaigns": 0.35,
     "energy_and_climate_solutions": 0.45,
-    "faith_and_spirituality": 0.35,
-    "fashion_and_beauty": 0.45,
-    "fitness_and_exercise": 0.35,
-    "food_and_drink": 0.4,
-    "gadgets_and_apps": 0.55,
-    "gaming": 0.65,
+    "faith_and_spirituality": 0.45,
+    "fashion_and_beauty": 0.3,
+    "fitness_and_exercise": 0.4,
+    "food_and_drink": 0.3,
+    "gadgets_and_apps": 0.3,
+    "gaming": 0.15,
     "geopolitics_and_diplomacy": 0.45,
-    "golf": 0.35,
-    "good_news_and_kindness": 0.25,
-    "government_and_policy": 0.4,
+    "golf": 0.45,
+    "good_news_and_kindness": 0.3,
+    "government_and_policy": 0.3,
     "hockey": 0.15,
-    "home_and_garden": 0.5,
-    "housing_and_real_estate": 0.3,
+    "home_and_garden": 0.35,
+    "housing_and_real_estate": 0.25,
     "humanitarian_crises": 0.25,
-    "immigration_and_borders": 0.45,
+    "immigration_and_borders": 0.4,
     "internet_culture_and_creators": 0.05,
-    "macro_economy_and_rates": 0.5,
-    "markets_and_investing": 0.45,
+    "macro_economy_and_rates": 0.4,
+    "markets_and_investing": 0.5,
     "media_and_journalism": 0.3,
     "medical_and_public_health": 0.45,
-    "mental_health": 0.3,
-    "motorsports": 0.55,
-    "movies_and_tv": 0.55,
-    "music": 0.5,
-    "nature_and_wildlife": 0.5,
-    "nutrition_and_diet": 0.45,
+    "mental_health": 0.5,
+    "motorsports": 0.25,
+    "movies_and_tv": 0.35,
+    "music": 0.45,
+    "nature_and_wildlife": 0.35,
+    "nutrition_and_diet": 0.55,
     "offbeat_and_unusual": 0.3,
-    "olympics": 0.6,
-    "other_sports": 0.35,
-    "parenting_and_family": 0.4,
+    "olympics": 0.4,
+    "other_sports": 0.25,
+    "parenting_and_family": 0.35,
     "personal_finance": 0.1,
-    "political_figures_and_scandals": 0.25,
-    "psychology_and_behavior": 0.05,
-    "relationships_and_dating": 0.5,
+    "political_figures_and_scandals": 0.3,
+    "psychology_and_behavior": 0.1,
+    "relationships_and_dating": 0.35,
     "scams_and_fraud": 0.25,
-    "scientific_discoveries": 0.3,
-    "screen_time_and_digital_life": 0.25,
-    "severe_weather": 0.45,
+    "scientific_discoveries": 0.35,
+    "screen_time_and_digital_life": 0.1,
+    "severe_weather": 0.5,
     "sleep_and_longevity": 0.05,
-    "soccer": 0.4,
-    "social_issues_and_activism": 0.25,
-    "space_and_astronomy": 0.4,
-    "state_and_local_politics": 0.1,
-    "tennis": 0.35,
-    "terrorism_and_security": 0.45,
-    "travel_and_places": 0.4,
+    "soccer": 0.3,
+    "social_issues_and_activism": 0.4,
+    "space_and_astronomy": 0.5,
+    "state_and_local_politics": 0.05,
+    "tennis": 0.3,
+    "terrorism_and_security": 0.35,
+    "travel_and_places": 0.35,
     "true_crime": 0.05,
-    "war_and_conflict": 0.55,
-    "work_and_careers": 0.55
+    "war_and_conflict": 0.35,
+    "work_and_careers": 0.4
   },
   "model_type": "snt_classifier",
-  "snt_version": "v0.5.1",
+  "snt_version": "v0.6",
   "transformers_version": "5.8.0"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:911b368abfd5170be7b8ab73f6bb1b4b9a323e9dba76bb3069a10d9f477a0cce
+oid sha256:6600e39a91f7e017d0eceb02baa03ca325e6574fae170b2bb68f58735b71c9b4
 size 2239950948