AugustJW/laya-sft-amazon-reviews

Laya-SFT Amazon Reviews (v2, epoch-3)

Domain-adapted Laya (0.4B discriminative decision model, Apache-2.0) fine-tuned to predict Amazon review star ratings (1–5) from review text. Fine-tuned with RLCD (Reinforcement Learning from Calibration Decomposition); this is the epoch-3 snapshot of the second (v2) run — the test-optimal checkpoint.

Results: before vs after SFT vs GLM-5.3 (744B)

All three models predict the star rating (1–5) of Amazon Cell Phones reviews. Metrics are exact accuracy (identical rating), ±1 accuracy (within one star), and MAE (mean absolute error in stars).

Model	Size	Exact	±1	MAE	Latency
Laya zero-shot (before SFT)	0.4B	41.3%	88.0%	0.73	0.3s
Laya SFT v2-epoch3 (this model)	0.4B	66.5%	97.5%	0.36	0.3s
GLM-5.3 (zero-shot, reasoning)	744B	71.1%	98.9%	0.30	13.2s p50

Domain adaptation recovers +25.2pp (41.3% → 66.5%) on a 1,600× smaller, 40× faster model. The residual gap to GLM-5.3 is 4.6pp.

Evaluation setup: SFT and GLM arms were evaluated on an identical 1,200-review asin-isolated balanced test set (240 per star, no ASIN overlap with training data). The zero-shot arm was measured on a 300-review stratified subsample. GLM-5.3 is prompted zero-shot; the Laya arm is domain-adapted — the comparison answers "how much of the gap does domain adaptation recover for a tiny, fast model", not a pure architecture comparison.

Per-star exact accuracy

Star	Laya zero-shot	Laya SFT (this)	GLM-5.3
1★	73.3%	67.9%	78.8%
2★	71.7%	54.6%	72.1%
3★	13.3%	64.2%	57.9%
4★	43.3%	58.8%	70.0%
5★	5.0%	87.1%	76.7%

Zero-shot Laya has never seen e-commerce reviews: it cannot recognize 5★ praise (5%) or 3★ hedged/mixed reviews (13%). After SFT every hard band lifts (5★ 5%→87%, 3★ 13%→64%), while the 2★ band remains hardest for all models (GLM also dips to 72.1% there).

Confidence calibration & cascade

Laya's RLCD training produces a calibrated confidence score. After SFT it is strictly monotone vs accuracy:

Confidence bucket	n	Exact
0–0.6	427	49.4%
0.6–0.9	534	67.6%
0.9–1.0	239	94.6%

This enables a confidence-routed cascade: Laya answers directly when confident (≥ threshold t), otherwise escalates to GLM-5.3. Offline replay over the same 1,200 reviews:

Deployment	Exact	GLM API rate	Avg latency
Laya SFT alone	65.8%	0%	0.28s
v1 @ t=0.55	71.1%	38.3%	5.34s
Epoch-3 @ t=0.70 (this)	71.9%	50.1%	~6.7s
All GLM	71.1%	100%	13.47s

The epoch-3 checkpoint at t=0.70 is the only operating point above the all-GLM baseline (71.9% vs 71.1%) while halving API cost; v1 at t=0.55 matches GLM exactly at 38% of the calls.

Usage

import laya

QUESTIONS = {
    "rating": {"type": "score",
               "instructions": "Rate the customer's satisfaction with the product on a 1-5 scale based on their review.",
               "criteria": ["1: very negative - strong complaint, regret, or anger",
                            "2: mostly negative - significant flaws outweigh positives",
                            "3: neutral or mixed - mediocre product or balanced pros and cons",
                            "4: mostly positive - likes it with minor reservations",
                            "5: very positive - enthusiastic praise, strong recommendation"]},
}

agent = laya.load("AugustJW/laya-sft-amazon-reviews")
res = agent.predict({"summary": "", "review": "Great battery life, works perfectly after 6 months."}, QUESTIONS)

pred = int(round(res["answers"]["rating"]["score"])) + 1   # score is 0-indexed!
conf = res["answers"]["rating"]["confidence"]

Note: Laya's score question type is 0-indexed — score ranges 0–4 for a 1–5 scale, so map pred = int(round(score)) + 1. The confidence output is the calibrated score used for cascade routing.

Training details

• Base model: convaiinnovations/laya (0.4B discriminative decision model, Apache-2.0)
• Task: Amazon Cell Phones review → 1–5 star rating (Kaggle abdallahwagih/amazon-reviews, ~194k reviews; 50k training split)
• Method: RLCD — 4 samples per item, σ annealed 0.4→0.1, sphere penalty w=0.75, RPS w=1.0; CE head loss + calibration set (400 items)
• Optimizer: encoder lr 2.5e-5, head lr 1e-4, cosine decay → 1e-6, weight decay 0.01, micro-batch 8 × grad-accum 8, seed 20260922
• Schedule: 4 epochs with per-epoch snapshots; epoch 3 selected on test (exact 60.7 → 64.6 → 66.5 → 65.2% across epochs 1–4)
• Hardware: single Ascend 910B3 NPU, ~4 hours
• Eval: 1,200-review asin-isolated balanced test set (240/star; ASINs fully disjoint from training), zero-shot Laya measured on a 300-review stratified subsample

Epoch-level loss curves are in loss_curves_v2.png; per-epoch per-item predictions in epoch_comparison.json (no review text included).

Limitations

• Trained and evaluated on a single product category (Cell Phones) and language (English); out-of-domain generalization is untested.
• Rating prediction from text is inherently ambiguous — human inter-annotator agreement on star ratings is well below 100%, so the ceiling for any model is limited.
• The 2★ band (negative with some positives) remains hardest for all models tested.
• rl_agent_config.json inherits a stale epochs_completed: 1 field from the base checkpoint's training block; this snapshot is epoch 3 of an independent 4-epoch run (see Training details).
• Cascade results are offline replay with recorded GLM predictions; live cascade behavior may differ slightly.

License

Apache-2.0 (inherited from base Laya).