README.md

---
license: apache-2.0
base_model: google/bert_uncased_L-8_H-512_A-8
library_name: transformers
pipeline_tag: text-classification
tags:
- social-media
- content-filtering
- engagement-bait
- distillation
- onnx
metrics:
- f1
- precision
- recall
---

# engagement-farm-classifier

A small classifier that flags engagement-farming posts on social platforms (built and evaluated on X/Twitter posts): explicit CTAs ("like if you agree", "tag someone who"), reply-bait questions, giveaways, chain posts, and low-substance filler whose main goal is farming replies and likes.

- **Base:** google/bert_uncased_L-8_H-512_A-8 (32M parameters)
- **Serving artifact:** int8 ONNX in `onnx-int8/` (42 MB)
- **CPU latency:** 1.9 ms mean, 2.7 ms p95 per post (Apple Silicon, single post)

## Labels

`0` = `genuine`, `1` = `engagement_farming`. Serving rule: softmax probability of `engagement_farming` >= 0.5 flags the post; lower the threshold to flag more aggressively (see Metrics).

## Metrics

Validation (777 posts, 189 farming, held out from training):

| threshold | precision | recall | f1 |
|---|---|---|---|
| 0.5 | 0.98 | 0.90 | 0.94 |
| 0.3 | 0.97 | 0.90 | 0.94 |

Held-out test set (273 posts collected after all training data, zero id/text overlap with training, 15 farming):

| threshold | precision | recall | f1 |
|---|---|---|---|
| 0.5 | 1.00 | 0.47 | 0.64 |
| 0.4 | 1.00 | 0.53 | 0.70 |
| 0.3 | 1.00 | 0.67 | 0.80 |

Zero false positives on the test set at every threshold. The test positives are subtle, timeline-native bait (rhetorical "how many of you" questions, greeting filler), so test recall is the realistic number for in-feed filtering. Test positives are few (15), so treat these as indicative.

## Training data

12,506 posts collected from public timelines and keyword searches. Labeled by a large LLM teacher (kimi-k3 via batched prompts), keeping only verdicts with teacher confidence >= 0.85: 7,766 posts (1,706 farming) after text dedupe. Positive class oversampled ~1:2 in the train split. Split and threshold details are in `train.py` and `eval.py`.

**No raw post corpus is redistributed.** Only scripts and weights are published; the collection and labeling pipeline is included so anyone can rebuild the dataset.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "selftaughtdev/engagement-farm-classifier"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)

text = "like if you agree"
probs = model(**tok(text, return_tensors="pt", truncation=True, max_length=128)).logits.softmax(-1)[0]
print(probs[1].item())  # probability of engagement_farming
```

For the int8 ONNX artifact:

```python
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model = ORTModelForSequenceClassification.from_pretrained(name, subfolder="onnx-int8", file_name="model_quantized.onnx")
```

## Limitations

- Trained on English-language posts; other languages are untested.
- Boundary cases are genuine disagreement: a rhetorical question with substance versus the same question as pure bait. Confident-only teacher labels (>= 0.85) trim but do not remove this ambiguity.
- Innocent-looking filler (plain "good morning" posts, rhetorical questions) is the main source of false negatives.

## Regenerating the dataset and model

1. `collect_tweets.js`: paste into a browser console on the target platform, it scrolls and dedupes posts into JSON.
2. `label.py`: sends batches of 20 posts to any OpenAI-compatible teacher endpoint (`--base-url`) with parallel workers, resumable.
3. `train.py`: fine-tunes the base model (`--base`), exports fp32 and int8 ONNX.
4. `eval.py`: threshold sweeps on validation or a held-out file (`--labeled <file> --full`).

## License

Apache 2.0 (matches the base model).