ARC-NLP at PAN 2023: Hierarchical Long Text Classification for Trigger Detection
Paper • 2307.14912 • Published
Request access to FanMeta
FanMeta was trained on fanfiction written by AO3 authors. To respect their work, the model files are shared only on request, for non-commercial research or educational use. Requests are reviewed manually.
Log in or Sign Up to review the conditions and access this model content.
FanMeta: fanfiction metadata recommendation
FanMeta reads a fanfiction story and recommends its AO3-style metadata:
| Task | Type | Labels |
|---|---|---|
| Rating | single-label (4) | General Audiences, Teen And Up Audiences, Mature, Explicit |
| Archive Warnings | multi-label (4) | No Archive Warnings Apply, Graphic Depictions Of Violence, Major Character Death, Rape/Non-Con |
| Additional Tags | multi-label (16, max 5 shown) | Alcohol, Humor, Angst, Blood/Injury, Character Death, Established Relationship, Family, Fluff, Friendship, Hurt/Comfort, Mental Health, Mythical Creatures, Pining, Smut, Suicide, Whump |
This model was built for an undergraduate thesis (skripsi) and powers the FanMeta web app: https://github.com/FathiyahAzzahra/FanMeta.
How it works
The architecture is adapted from ARC-NLP's hierarchical long-text classifier for the PAN 2023 Trigger Detection task (Sahin et al., 2023; see References): split a long story into segments, fine-tune a Transformer on the segments, then combine the segment embeddings with an LSTM. FanMeta's own changes are:
- Multi-task output: one model predicts Rating, Archive Warnings and Additional Tags at once, with a separate LSTM for each task (Task-Specific LSTM), instead of trigger warnings only.
- Data and labels: RoBERTa-base trained on AO3 stories for these three AO3 metadata fields, with the authors' Additional Tags mapped to 16 target tags (see Training data).
- Indonesian input through NLLB-200 translation, and per-label thresholds tuned on validation data.
Steps:
- The story is split into 200-word segments with a 50-word overlap.
- A fine-tuned RoBERTa encoder (
FacebookAI/roberta-base, max 256 tokens) reads every segment. - A Task-Specific LSTM (one LSTM per task, hidden size 100) combines all segment embeddings into one prediction per task.
- Indonesian stories are first translated to English with NLLB-200 (distilled 600M).
Files
| File | Size | Content |
|---|---|---|
02_roberta_stage1_C.pt |
499 MB | Fine-tuned RoBERTa encoder |
04_lstm_C_seed42.pt |
4 MB | Task-Specific LSTM heads (seed 42) |
The thresholds and settings (06_model_app.json) and the code to load the model
(src/ml/predictor.py, src/ml/fanmeta_utils.py) are in the GitHub repository. After your request is
approved, log in with huggingface-cli login; the app downloads both files automatically.
Results (seed 42, tuned thresholds, macro F1)
| Test set | Rating | Warnings | Tags |
|---|---|---|---|
| English (independent) | 0.620 | 0.579 | 0.443 |
| Indonesian (via NLLB translation) | 0.512 | 0.566 | 0.370 |
Training data and limitations
Trained on English AO3 fanfiction of 500–3,500 words from two sources:
- stories scraped from AO3 for this thesis, and
- extra stories for rare tags from the Hugging Face dataset
midwestern-simulation-active/ao3_random_subset.
The Indonesian test set combines scraped stories and Indonesian stories from the same Hugging Face dataset. The training and test data are not shared, because the stories belong to their authors.
Labels come from the tags the authors chose themselves, mapped to the 16 target tags, so they are noisy and inconsistent between authors.
Indonesian text goes through machine translation, so translation errors lower the scores.
Rare labels (for example Explicit and some tags) are predicted less reliably.
The output is a recommendation. The author should always review it before publishing.
Intended use
Helping fanfiction authors choose metadata, and non-commercial research or education. Not allowed: commercial use, redistributing the model files, or using them (or their outputs) to train generative models.
References
- Sahin, U., Kucukkaya, I. E., & Toraman, C. (2023). ARC-NLP at PAN 2023: Hierarchical Long Text Classification for Trigger Detection. CLEF 2023 Working Notes. arXiv:2307.14912
- Liu, Y., et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692
- midwestern-simulation-active. ao3_random_subset [Dataset]. Hugging Face. https://huggingface.co/datasets/midwestern-simulation-active/ao3_random_subset
- NLLB Team (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672
Model tree for ZAaraAa/fanmeta-model
Base model
FacebookAI/roberta-base