Lattice_Spike / examples /README.md
Md Saiful Islam Sajol
Upload Lattice_Spike project with full LFS weights
e2c9d42 verified
|
Raw History Blame Contribute Delete
970 Bytes
# Tiny Token Evaluation Demo
`tiny_token_eval.jsonl` is a tiny fake evaluation set for smoke-testing the
token-level baseline harness. It is intentionally small and not a benchmark.
Each row has:
- `prompt`: the question/context.
- `answer`: the generated answer to score.
- `hallucinated_spans`: answer substrings that should be labeled hallucinated.
- `metadata`: notes explaining the toy case.
The DGX evaluator converts `hallucinated_spans` into tokenizer-aligned `0/1`
labels at runtime. That avoids hard-coding labels that only work for one
tokenizer.
Run it with:
```bash
python scripts/eval_tiny_baselines_dgx.py \
--model Qwen/Qwen2.5-0.5B-Instruct \
--detector epr \
--detector lookback \
--device cuda \
--dtype bfloat16
```
The wrapper writes token-level predictions to:
```text
outputs/tiny_token_eval_predictions.jsonl
```
Use these numbers only as an end-to-end sanity check. Formal detector quality
still needs a real labeled dataset.