Lattice_Spike / examples /README.md
Md Saiful Islam Sajol
Upload Lattice_Spike project with full LFS weights
e2c9d42 verified
|
Raw History Blame Contribute Delete
970 Bytes

Tiny Token Evaluation Demo

tiny_token_eval.jsonl is a tiny fake evaluation set for smoke-testing the token-level baseline harness. It is intentionally small and not a benchmark.

Each row has:

  • prompt: the question/context.
  • answer: the generated answer to score.
  • hallucinated_spans: answer substrings that should be labeled hallucinated.
  • metadata: notes explaining the toy case.

The DGX evaluator converts hallucinated_spans into tokenizer-aligned 0/1 labels at runtime. That avoids hard-coding labels that only work for one tokenizer.

Run it with:

python scripts/eval_tiny_baselines_dgx.py \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --detector epr \
  --detector lookback \
  --device cuda \
  --dtype bfloat16

The wrapper writes token-level predictions to:

outputs/tiny_token_eval_predictions.jsonl

Use these numbers only as an end-to-end sanity check. Formal detector quality still needs a real labeled dataset.