Douban Sentiment Classification at Scale — Deployed Weights

中文短评三分类(正面 / 中性 / 负面)情感分析的部署权重:三个 MacBERT-base 学生模型 + 三个基于冻结 MegatronBERT-1.3B 特征的 MLP 头,组成论文中的混合集成系统。

This repository hosts the deployed v3 ensemble weights from the project douban-sentiment. The system performs three-class sentiment classification (positive / neutral / negative) and binary classification on terse Chinese Douban movie reviews, and reaches 89.5% accuracy / 85.9 macro-F1 (3-class) and 95.9% / 95.6 (binary) on a 600-row human-verified gold set under stratified 5-fold out-of-fold evaluation.

Model description

The deployed system is a hybrid ensemble with three channels:

Component Path Role
3-class student macbert_3c/ chinese-macbert-base, distilled from a 7-channel teacher ensemble (hard-label CE + soft-target KL, α=0.5, τ=2, 8 epochs)
Binary student macbert_bin/ same recipe, binary task
Neutral specialist macbert_neu/ MacBERT binary detector trained on 115K corpus-scale pseudo-labels (zero gold labels) — separates "genuinely neutral" from polarised reviews
3-class judge head heads/mlp_3c_mb13b_m256.pt MLP on frozen Erlangshen-MegatronBERT-1.3B features
Binary judge head heads/mlp_bin_mb13b_m256.pt same backbone, binary head
Neutral judge head heads/mlp_neu_mb13b_m256_h256.pt same backbone, neutral-detection head (hidden=256)

The judge heads consume 4096-dimensional features: the concatenation of the CLS token and the attention-masked mean of last_hidden_state from IDEA-CCNL/Erlangshen-MegatronBert-1.3B-Sentiment (external dependency, not included in this repo — download it separately). Each .pt file stores state_dict, in_dim, n_classes, hidden, scaler_mean, scaler_scale; standardise features with the stored scaler before the MLP.

Ensemble recipe (as deployed)

dep_3c   = (p_student_3c + 2 · p_judge_3c) / 3          # student:judge = 1:2
dep_bin  = (p_student_bin + 2 · p_judge_bin) / 3

p_n      = 0.4 · P_neu(macbert_neu) + 0.6 · P_neu(mlp_neu head)
r_±      = d_± / (d_+ + d_-)  from dep_3c               # renormalised pos/neg
spec     = [(1−p_n)·r_+,  p_n,  (1−p_n)·r_-]
final_3c = (dep_3c + 0.5 · spec) / 1.5                  # then temperature T=0.37

Intended uses & limitations

Intended for: sentiment analysis of short Chinese movie-review text (Douban-style, typically < 200 characters), research and demonstration. The neutral class targets mixed-polarity reviews (e.g. “特效无敌,但是剧情简直像一坨”).

Limitations:

  • Trained and evaluated on Douban movie reviews; performance on other domains, platforms, or review lengths is unverified.
  • The neutral/mixed class remains the hardest: neutral recall is 70.7% on the gold set and degrades on very terse corpus-style comments (honest OOD study in the paper reports ~60% three-class accuracy on fresh wild reviews vs. LLM reference labels).
  • Not a substitute for human judgement on high-stakes decisions.

How to use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

REPO = "Shuakami/douban-sentiment-weights"

# --- MacBERT students (standard sequence-classification checkpoints) ---
stu3_tok = AutoTokenizer.from_pretrained(f"{REPO}/macbert_3c")
stu3     = AutoModelForSequenceClassification.from_pretrained(f"{REPO}/macbert_3c")

# --- MLP judge heads ---
# 1) load backbone (external):
#    bb_id = "IDEA-CCNL/Erlangshen-MegatronBert-1.3B-Sentiment"
# 2) features = cat([last_hidden_state[:,0], masked_mean(last_hidden_state)])  # 4096-d
# 3) blob = torch.load("heads/mlp_3c_mb13b_m256.pt")
#    x = (feats - blob["scaler_mean"]) / blob["scaler_scale"]
#    net = MLP(in_dim=4096, hidden=blob["hidden"], n_classes=blob["n_classes"])
#    net.load_state_dict(blob["state_dict"])

A runnable reference implementation (FastAPI demo backend) is in web/app.py of the project repository, and a live demo is hosted at https://huggingface.co/spaces/Shuakami/douban-sentiment.

Training data

  • Gold set: 600 Douban reviews with human-verified semantic labels (label_semantic), single-film-page sample; class distribution 316 positive / 92 neutral / 192 negative.
  • Corpus: Moviedata-10M (csuldw/AntSpider), 910,998 cleaned Douban comments with star ratings only. Pseudo-labels were produced by the deployed ensemble; the confident pool (max p ≥ 0.7) contains 271,905 rows used to train macbert_neu.
  • lab2000: a 2,000-row cross-movie benchmark with two-pass LLM blind annotation and 479 human-adjudicated rows, released in the project repository for OOD/active-learning evaluation.

Training procedure

  • Students: knowledge distillation from a 7-channel teacher ensemble (TF-IDF LR + frozen-backbone MLP heads), hard-label CE + soft-target KL (α=0.5, τ=2), 8 epochs, batch 32, lr 2e-5, bf16, max_len 256, stratified 5-fold OOF for evaluation; production weights trained on 100% of the gold set.
  • macbert_neu: binary MacBERT classifier on 115,000 pseudo-labelled corpus rows (per-class cap 60K/side), 2 epochs — no gold labels.
  • Judge heads: MLP (4096→512→256→C, GELU, dropout 0.2) on frozen MegatronBERT-1.3B features, 300 epochs, trained on gold-600.

Evaluation results

Gold-600, stratified 5-fold out-of-fold (every prediction on unseen rows):

System 3-class acc / macro-F1 Binary acc / macro-F1
TF-IDF + LR baseline 0.723 / 0.619 0.841 / 0.823
Qwen3-4B zero-shot 0.783 —
Corpus student (zero gold training) 0.875 / 0.846 0.949 / 0.946
Deployed v3 ensemble (this repo) 0.895 / 0.859 0.959 / 0.956

Out-of-distribution (lab2000, cross-movie wild reviews): deployed v3 reaches 65.0% three-class / 85.0% binary; adding 1,000 actively-selected boundary examples lifts the SFT student to 71.4% (+8.8pp, McNemar p = 4.0×10⁻⁸); scaling to ~17K frontier-LLM pseudo-labels reaches 77.1% three-class / 92.3% binary (details in the paper).

Limitations and bias

  • Neutral-class recall is the weakest point (70.7% gold OOF); models tend to over-predict polarised classes on ambiguous text.
  • Star ratings correlate imperfectly with semantic sentiment (measured corpus-wide agreement ≈48% three-class); do not use ratings as labels.
  • LLM-generated pseudo-labels carry teacher biases; the paper documents where they help (scale) and where they cannot (beyond-teacher knowledge).

Citation

@misc{shuakami2026douban,
  author       = {Shuakami},
  title        = {Weakly-Supervised Chinese Review Sentiment
                  Classification at Scale: Pseudo-Labeling,
                  Specialist Decomposition, and Honest
                  Out-of-Distribution Evaluation},
  year         = {2026},
  howpublished = {\url{https://github.com/shuakami/douban-sentiment}},
  note         = {Undergraduate thesis; bilingual PDF in papers/}
}

Please cite the project repository and the papers in papers/ if you use these weights.

License & data note

Code and weights are provided for research / coursework use only (no formal OSS license granted). The external Douban corpus (Moviedata-10M) is research-use only with attribution required and is not redistributed in this repository; the 600 gold labels are hand-annotated for this project. See the project repository for full data and provenance documentation.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Shuakami/douban-sentiment-weights

Finetuned
(69)
this model

Space using Shuakami/douban-sentiment-weights 1