Chongshan0Lin/hw1-hc3-detector

HW1: HC3 AI-Generated Text Detector

sentence-transformers/all-MiniLM-L6-v2 fine-tuned end-to-end for binary classification of HC3 (English) answers.

Labels: 0 = human, 1 = ChatGPT

Results (HC3 test split, 4,668 answers, balanced)

Model	Test accuracy
Baseline: frozen all-MiniLM-L6-v2 embeddings + logistic regression	0.8449
Fine-tuned (this model)	0.9852

Total misclassifications dropped from 724 (baseline) to 69 (fine-tuned). Of the fine-tuned errors, 68 are human answers predicted as ChatGPT and 1 is a ChatGPT answer predicted as human.

Training details

• Data: Hello-SimpleAI/HC3 all.jsonl (revision 4d0ff18143b5a7e1b1e79beb540c04549d1e59d3). The first nonempty human and ChatGPT answer was kept per question, and empty, duplicate or identical-answer questions were excluded. Questions were split 80/10/10 (seed 42) so each question's answers stay in one split: 37,334 train / 4,666 validation / 4,668 test answers.
• Model: AutoModelForSequenceClassification with num_labels=2 and cross-entropy loss.
• Optimizer: AdamW, learning rate 2e-5, constant schedule, batch size 32, 5 epochs, max length 256 tokens, seed 42.
• Hardware: 1× NVIDIA RTX A6000 (~1.7 min per epoch).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "Chongshan0Lin/hw1-hc3-detector"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

inputs = tok("Your answer text here.", truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    pred = model(**inputs).logits.argmax(-1).item()
print(["human", "ChatGPT"][pred])

Limitations

This model was trained for a course homework on a 2022-era benchmark. It is not a reliable detector of AI-generated text in general, and it should not be used to judge student work. It can exploit dataset artifacts, such as the tokenizer-style spacing in Reddit ELI5 human answers, length and formatting. It is biased toward predicting "ChatGPT": almost all of its errors are false accusations against human answers. It will likely not transfer to other LLMs, prompts or domains.