File size: 3,887 Bytes
48a842f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
license: apache-2.0
base_model: google/bert_uncased_L-8_H-512_A-8
library_name: transformers
pipeline_tag: text-classification
tags:
- social-media
- content-filtering
- engagement-bait
- distillation
- onnx
metrics:
- f1
- precision
- recall
---
# engagement-farm-classifier
A small classifier that flags engagement-farming posts on social platforms (built and evaluated on X/Twitter posts): explicit CTAs ("like if you agree", "tag someone who"), reply-bait questions, giveaways, chain posts, and low-substance filler whose main goal is farming replies and likes.
- **Base:** google/bert_uncased_L-8_H-512_A-8 (32M parameters)
- **Serving artifact:** int8 ONNX in `onnx-int8/` (42 MB)
- **CPU latency:** 1.9 ms mean, 2.7 ms p95 per post (Apple Silicon, single post)
## Labels
`0` = `genuine`, `1` = `engagement_farming`. Serving rule: softmax probability of `engagement_farming` >= 0.5 flags the post; lower the threshold to flag more aggressively (see Metrics).
## Metrics
Validation (777 posts, 189 farming, held out from training):
| threshold | precision | recall | f1 |
|---|---|---|---|
| 0.5 | 0.98 | 0.90 | 0.94 |
| 0.3 | 0.97 | 0.90 | 0.94 |
Held-out test set (273 posts collected after all training data, zero id/text overlap with training, 15 farming):
| threshold | precision | recall | f1 |
|---|---|---|---|
| 0.5 | 1.00 | 0.47 | 0.64 |
| 0.4 | 1.00 | 0.53 | 0.70 |
| 0.3 | 1.00 | 0.67 | 0.80 |
Zero false positives on the test set at every threshold. The test positives are subtle, timeline-native bait (rhetorical "how many of you" questions, greeting filler), so test recall is the realistic number for in-feed filtering. Test positives are few (15), so treat these as indicative.
## Training data
12,506 posts collected from public timelines and keyword searches. Labeled by a large LLM teacher (kimi-k3 via batched prompts), keeping only verdicts with teacher confidence >= 0.85: 7,766 posts (1,706 farming) after text dedupe. Positive class oversampled ~1:2 in the train split. Split and threshold details are in `train.py` and `eval.py`.
**No raw post corpus is redistributed.** Only scripts and weights are published; the collection and labeling pipeline is included so anyone can rebuild the dataset.
## Usage
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
name = "selftaughtdev/engagement-farm-classifier"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)
text = "like if you agree"
probs = model(**tok(text, return_tensors="pt", truncation=True, max_length=128)).logits.softmax(-1)[0]
print(probs[1].item()) # probability of engagement_farming
```
For the int8 ONNX artifact:
```python
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer
model = ORTModelForSequenceClassification.from_pretrained(name, subfolder="onnx-int8", file_name="model_quantized.onnx")
```
## Limitations
- Trained on English-language posts; other languages are untested.
- Boundary cases are genuine disagreement: a rhetorical question with substance versus the same question as pure bait. Confident-only teacher labels (>= 0.85) trim but do not remove this ambiguity.
- Innocent-looking filler (plain "good morning" posts, rhetorical questions) is the main source of false negatives.
## Regenerating the dataset and model
1. `collect_tweets.js`: paste into a browser console on the target platform, it scrolls and dedupes posts into JSON.
2. `label.py`: sends batches of 20 posts to any OpenAI-compatible teacher endpoint (`--base-url`) with parallel workers, resumable.
3. `train.py`: fine-tunes the base model (`--base`), exports fp32 and int8 ONNX.
4. `eval.py`: threshold sweeps on validation or a held-out file (`--labeled <file> --full`).
## License
Apache 2.0 (matches the base model). |