punyajain/epidebug-mm-v3

EpiDebug Multimodal Checkpoint v3 (CPU reasoning seed)

Philosophy

1. You cannot encode every failure case/scenario.
2. Encode physics, maths, raw intelligence, reasoning capability.
3. Train on failure data (experiments/engineering failures).
4. Multimodal user input is messy — clean/extract BEFORE the model.
5. Multimodal encoding of cleaned extracts is first-class.

Pipeline

Messy dump → Cleanse/Extract → Multimodal encode → Reason (compete hypotheses / ask follow-ups)

Architecture

• Frozen MiniLM text encoder + tabular MLP + tiny CNN
• Fusion → failure-category head + hypothesis ranking + reasoning pair head + weak rationale distill + follow-up ranking
• Collate shuffles hypotheses every batch (fixes v0 gold-at-index-0 inflation)

Best val metrics

{
  "loss": 8.894475611595071,
  "accuracy": 0.5481727574750831,
  "macro_f1": 0.5133248224375323,
  "hyp_accuracy_shuffled": 0.5780730897009967,
  "reason_accuracy_shuffled": 0.521594684385382,
  "followup_accuracy": 0.49169435215946844,
  "batch_gold_at_index0_rate": 0.22259136212624583,
  "n": 301
}

Limits (honest)

• Not a frontier reasoner; CPU seed only
• Physics priors are synthetic micro-vignettes
• Scale-up: GPU LoRA on a larger reasoning LM; keep cleanse→extract gate