| # v1.0 Verification Results |
|
|
| ## Summary |
|
|
| | Metric | Value | Threshold | Result | |
| |--------|-------|-----------|--------| |
| | retrieval_similarity | 0.7239 | >= 0.60 | PASS | |
| | tIOU | 0.6700 | >= 0.40 | PASS | |
| | mAP@0.5 | 0.5092 | >= 0.50 | PASS | |
| |
| **Overall: PASS — v1.0 ACCEPTED** |
| |
| ## Details |
| |
| - Checkpoint: `outputs/checkpoint-770` (step 770, eval_loss 0.5272) |
| - Predictions: `checkpoints/v21_semantic_boundaries/predictions.jsonl` (98 samples) |
| - Embedder: OpenAI text-embedding-3-small (production-aligned) |
| - Date: 2026-03-19 |
|
|
| ### Retrieval Similarity |
|
|
| - Mean: 0.7239 |
| - Median: 0.7355 |
| - Min: 0.4982, Max: 0.9162 |
| - Excellent (>= 0.80): 22/98 |
| - Good (>= 0.60): 87/98 |
| - Acceptable (>= 0.40): 98/98 |
|
|
| ### Temporal IoU (tIOU) |
|
|
| - Mean: 0.6700 |
| - Median: 0.6703 |
| - Predictions with timestamps: 98/98 |
| - References with timestamps: 98/98 |
|
|
| ### mAP@0.5 |
|
|
| - Mean: 0.5092 |
|
|
| ## Production Embeddings (prior run) |
|
|
| For reference, the prior eval with OpenAI text-embedding-3-small on 20 pairs showed: |
| - mean_similarity: 0.6781 |
| - production_ready: true |
|
|
| ## Failure Analysis |
|
|
| All metrics pass. v1.0 is accepted for production. |
|
|
| ## Per-Sample Scores (first 10) |
|
|
| | # | retrieval_sim | tIOU | mAP@0.5 | pred_events | ref_events | |
| |---|--------------|------|---------|-------------|------------| |
| | 0 | 0.6549 | 0.5277 | 0.4500 | 5 | 4 | |
| | 1 | 0.7739 | 0.8080 | 1.0000 | 5 | 5 | |
| | 2 | 0.6433 | 1.0000 | 1.0000 | 4 | 4 | |
| | 3 | 0.7473 | 0.6228 | 0.4500 | 5 | 4 | |
| | 4 | 0.7774 | 0.5406 | 0.2500 | 4 | 4 | |
| | 5 | 0.7924 | 0.3868 | 0.0500 | 4 | 5 | |
| | 6 | 0.7626 | 0.7779 | 0.7500 | 4 | 3 | |
| | 7 | 0.6476 | 0.8060 | 0.7500 | 4 | 3 | |
| | 8 | 0.7355 | 0.7520 | 0.6400 | 5 | 5 | |
| | 9 | 0.8859 | 0.6579 | 0.5625 | 4 | 4 | |
| |
