traj-eval / README.md
arrdel's picture
Sync traj-eval space artifacts
e24b8cd verified
|
Raw History Blame Contribute Delete
2.91 kB
metadata
title: traj-eval
emoji: 🧭
colorFrom: yellow
colorTo: red
sdk: static
app_file: index.html
pinned: true
license: mit
short_description: Denoising trajectory evaluation for diffusion LMs
models:
  - Dream-org/Dream-v0-Instruct-7B
  - GSAI-ML/LLaDA-8B-Instruct
  - GSAI-ML/LLaDA-1.5
datasets:
  - arrdel/traj-eval-benchmarks
tags:
  - masked-diffusion
  - diffusion-language-models
  - reasoning-trajectories
  - evaluation

traj-eval: Measure reasoning in the denoising trajectory, not just final-answer accuracy

Static Hugging Face Space for traj-eval, an open-source toolkit and benchmark suite for evaluating denoising trajectories in masked diffusion language models.

Two models can have identical final-answer accuracy for very different reasons: genuine reasoning, rationalization, and oscillation. Final-step evaluation collapses the entire denoising process into a single endpoint prediction, hiding when a verdict stabilized and how the intermediate rationale developed. traj-eval instruments and scores the full step-by-step denoising trajectory so these reasoning regimes can be distinguished.

What is on this page

  1. Why accuracy lies: Plain-language definitions of genuine reasoning, rationalization, and oscillation, plus the v0.1 synthetic benchmark split (marked synthetic, not model evidence).
  2. One trajectory, two conditions: Interactive step-by-step chart over denoising steps 0 to 255 (sampled every 16 steps) for ARC-Challenge items Mercury_LBS10610 (reasoning condition) and Mercury_7175875 (direct condition) on Dream-v0-Instruct-7B (shown to illustrate the trajectory pattern under each condition, not a paired comparison on the same item).
  3. Results (64-item suites only): Verified measurements across Dream-v0-Instruct-7B, LLaDA-8B-Instruct, and LLaDA-1.5 on 64-item ARC-Challenge (direct and reasoning) and GSM8K (numeric) suites, plus the side-by-side rationalization component decomposition for Dream-v0-Instruct-7B.
  4. Metrics, install, quickstart, and caveats: Operational definitions from src/traj_eval/metrics.py, zero-weight CLI/Python quickstart (numpy only, Python >= 3.10), and explicit caveats.

Links