HC3 Human vs. ChatGPT Detector (HW1)
Fine-tuned hc3-finetuned to classify answers from the
HC3 dataset as human (label 0) or ChatGPT (label 1).
Results (test set, 4,668 examples, balanced)
| Model | Test accuracy |
|---|---|
| Baseline: frozen sentence embeddings + logistic regression | 0.8449 (84.49%) |
| Fine-tuned transformer (this model) | 0.9754 (97.54%) |
Fine-tuned confusion matrix: all 2,334 ChatGPT answers were correctly detected; 115 of 2,334 human answers were misclassified as ChatGPT (all errors are false positives on human text).
Training setup
- Full fine-tuning, 5 epochs, AdamW, constant learning rate, no validation-based early stopping.
Limitations
- Trained and tested only on HC3 (ChatGPT, late 2022). Expect lower accuracy on other domains, newer models, or edited/paraphrased AI text.
- Errors are skewed toward flagging formal human writing as AI. Do not use this model to make decisions about individuals.
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="dsantra2/hw1-hc3-detector")
clf("Your text here")
- Downloads last month
- 26