Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
update
MedGRPO Team
aa5db53
add leaderboard printout
MedGRPO Team
c92c494
update name
MedGRPO Team
9aa418f
update
MedGRPO Team
0562da7
improve the prompt
MedGRPO Team
758710c
update
MedGRPO Team
4752404
upload prediction only
MedGRPO Team
b28cd8f
Fix evaluate_all_pai to pass --skip-llm-judge to task main() functions
MedGRPO Team
18339c0
CRITICAL FIX: The root cause of LLM judge still running despite --skip-llm-judge flag
Problem:
- _run_task_eval() was calling module.main() without any parameters
- main() functions had no way to receive the skip_llm_judge flag
- Result: Per-dataset evaluations ran LLM judge even when flag was set
Solution:
- Add skip_llm_judge parameter to _run_task_eval()
- Pass flag via sys.argv to main() functions
- main() functions (like eval_dvc.py) now parse --skip-llm-judge from sys.argv
Flow now works correctly:
1. app.py calls evaluate_predictions.py with --skip-llm-judge
2. evaluate_predictions.py calls evaluate_all_pai.run_evaluation(skip_llm_judge=True)
3. run_evaluation() calls _run_task_eval(task, file, skip_llm_judge=True)
4. _run_task_eval() sets sys.argv with --skip-llm-judge
5. module.main() parses sys.argv and correctly skips LLM judge
Tested: Per-dataset DVC evaluations now show 'Skipping LLM judge' message
Fix eval_dvc.py main() to support --skip-llm-judge flag
MedGRPO Team
5f41159
- Add --skip-llm-judge parameter parsing in main()
- Pass skip_llm_judge to per-dataset evaluations
- Add warning message when flag is detected
- Update usage message with flag example
This fixes the issue where per-dataset DVC evaluations would
run LLM judge even when --skip-llm-judge was specified, because
main() wasn't accepting or passing the parameter.
Fix DVC evaluation to compute temporal F1 when --skip-llm-judge is set
MedGRPO Team+1: Claude Sonnet 4.5
dd1b9c6
Problem:
- When using --skip-llm-judge flag, DVC was entirely skipped
- This meant DVC_F1 (temporal F1 score) was set to 0.0
- But DVC_F1 is NOT an LLM judge metric - it's based on temporal IoU
Solution:
1. Modified evaluate_all_pai.py to NOT skip DVC task when --skip-llm-judge is set
2. Modified eval_dvc.py to accept skip_llm_judge parameter
3. When skip_llm_judge=True:
- Skip caption quality evaluation (sets caption_score=0.0, method='skipped')
- Still compute temporal F1 based on segment IoU matching
4. Fixed metric printing to extract from results['overall'] dict
5. Added skip_llm_judge parameter to print_overall_evaluation_results()
Results:
- DVC temporal F1 now computed correctly (e.g., NurViD: 0.2169)
- VS and RC still skipped when --skip-llm-judge is used (no non-LLM metrics)
- Faster evaluation without LLM API costs while preserving DVC_F1
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add semantic similarity matching for Next Action evaluation
MedGRPO Team+1: Claude Sonnet 4.5
a66b9a4
Major improvements:
- Implement semantic similarity using SentenceBERT (all-MiniLM-L6-v2)
- Add support for predefined action lists (AVOS, CholecT50, CoPESD, NurViD)
- Handle free-form actions (EgoSurgery) with dynamic action list creation
- Add fallback to 'gnd' field for ground truth (supports CholecT50)
- Add normalization and class mapping per dataset
- Compute overall accuracy across all FPS values
SA metric fix:
- Extract Overall Skill Level Accuracy (0.2437) instead of Aspect Balanced Accuracy (0.2542)
- Now matches table value of 0.244 (vs 0.2542 before)
Results:
- NAP_acc improved from 0.3384 (exact match) to 0.4074 (semantic similarity)
- Matches original evaluation (0.4045) within 0.0029
- Now consistent with original methodology
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix TAL overall metrics computation and extraction
MedGRPO Team
c8f4cad
- eval_tal.py: Add overall IoU_0.3 and IoU_0.5 metrics computation
- Compute metrics across all FPS values combined
- Match original evaluation script behavior
- Returns IoU_0.3 and IoU_0.5 keys with Recall and meanIoU
- app.py: Add IoU section tracking for metric extraction
- Track current_iou_section state (0.3 or 0.5)
- Extract meanIoU from IoU_0.3: and IoU_0.5: sections
- Support both old format (meanIoU@0.3) and new format (IoU_0.3: meanIoU:)
Results now match original script:
- TAG_mIoU@0.3: 0.2164 ✅
- TAG_mIoU@0.5: 0.1555 ✅
Use STG overall metrics instead of placeholder
MedGRPO Team
d1337a3
Fix STG evaluation to extract bbox_dict from struc_info
MedGRPO Team
0b29eca
Add placeholder for STG metrics when evaluation returns empty
MedGRPO Team
6d00cf8
Fix STG and next_action metric printing in overall mode
MedGRPO Team
049c07c
Show evaluation metrics even in silent mode (--grouping overall)
MedGRPO Team
2323031
Filter DVC/VS/RC from tasks list when skip-llm-judge is set
MedGRPO Team
2bd924c
Remove debug messages - system working correctly
MedGRPO Team
80e3e7d
Add debug inside loop to verify entry
MedGRPO Team
d53d0f7
Add debug at line 741
MedGRPO Team
d878ea1
Add debug before task list print
MedGRPO Team
c886e23
Debug before/after imports
MedGRPO Team
4e448a4
Add debug at function entry and after analyze
MedGRPO Team
c736ac8
Add explicit flush debug
MedGRPO Team
1c45d6c
update
MedGRPO Team
f0846a5
update
MedGRPO Team
1af117c
update
MedGRPO Team
a605ebb
Implement secure ground truth with prediction-only submission format
MedGRPO Team
fe743f5
- Add private HF repo loading for ground truth (load_ground_truth)
- Update validation to accept predictions-only format (id, qa_type, prediction)
- Update evaluation to merge predictions with private ground truth before evaluation
- Add huggingface_hub dependency
- Update UI instructions for new submission format
- Add merge_predictions_with_gt.py utility script
- Remove old documentation files (keep only README.md)
- Security: Ground truth stored in private repo UIIAmerica/MedGRPO-GroundTruth
Add support for pre-computed LLM judge scores
MedGRPO Team+1: Claude Sonnet 4.5
31817d3
Implement automatic detection and handling of pre-computed LLM judge scores
in raw inference results, allowing users to skip expensive LLM judge
re-evaluation while still calculating other metrics.
Changes:
evaluate_all_pai.py:
- Add --skip-llm-judge flag to skip LLM judge evaluation for DVC/VS/RC
- Modify run_evaluation() to accept skip_llm_judge parameter
- Skip DVC, VS, RC tasks when flag is set (other metrics still calculated)
- Pass flag through main() function
app.py:
- Add check_for_precomputed_llm_scores() to detect llm_judge in struc_info
- Update validate_results_file() to return has_precomputed_llm flag
- Modify run_evaluation() to pass --skip-llm-judge when detected
- Update submit_model() to use pre-computed score detection
- Update UI documentation to explain struc_info detection and time savings
- Reject pure evaluation output files (lack raw inference data)
Benefits:
- Saves 3-8 minutes by skipping LLM judge re-evaluation
- Reduces API costs for caption tasks (DVC, …
Update evaluation metrics and leaderboard display
MedGRPO Team+1: Claude Sonnet 4.5
a36b7fe
- Modified app.py to fix metric definitions and display format
- Updated eval_dvc.py and eval_tal.py for consistent metric computation
- Aligned with standard evaluation pipeline
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Complete evaluator fixes for all 8 tasks
MedGRPO Team
331979f
- Fixed TAL: Added data format handling (answer + struc_info parsing)
- Fixed STG: Added wrapper functions, uses extract_boxes and compute_iou_batch
- Fixed next_action: Added wrapper functions with numpy import
- Created eval_dvc.py: Wrapper around caption evaluator for DVC tasks
- All evaluators now have main(), group_records_by_dataset(), evaluate_dataset_X()
- Removed duplicate if __name__ blocks from all files
All 8 tasks should now work:
✅ TAL, STG, next_action, DVC, VS, RC, skill_assessment, cvs_assessment
Fix syntax errors and add TAL wrapper functions
MedGRPO Team
8c805bc
- Fixed syntax errors in eval_skill_assessment.py and eval_cvs_assessment.py (removed \n literals)
- Added main(), group_records_by_dataset(), and evaluate_dataset_tal() to eval_tal.py
- TAL evaluator now compatible with evaluate_all_pai.py
Still TODO:
- Add wrapper functions to STG and next_action evaluators
- Create/fix DVC evaluator
- Full 8-task evaluation testing
Fix evaluate_all_pai.py to use eval_caption_llm_judge
MedGRPO Team
58dd6d7
- Updated RC/VS evaluation to use eval_caption_llm_judge instead of deleted eval_rc_vs
- Directly calls evaluate_caption_task() function
- Tested successfully with DAPO model results:
- VS: 0.8476 semantic similarity
- RC: 0.6910 semantic similarity
Consolidate evaluation scripts and remove hardcoded paths
MedGRPO Team
3ea8a3a
- Moved eval_next_action, eval_stg, eval_tag from my_eval_old/ to main evaluation/
- Renamed eval_tag.py to eval_tal.py (consistent with task name)
- Replaced wrapper files with actual implementations
- Removed hardcoded absolute paths (now require command line arguments)
- Deleted empty my_eval_old/ directory
- All evaluators now self-contained in evaluation/ folder
Remove all captioning_metrics dependencies
MedGRPO Team
5a5d9ce
- Deleted eval_dvc.py and eval_rc_vs.py (both top-level and my_eval_old/)
- These files used CIDER/METEOR metrics which require JAR files
- Replaced by eval_caption_llm_judge.py (server-side LLM judge)
- Updated evaluate_all_pai.py to use new evaluator
- No more captioning_metrics dependencies in codebase
Remove unused captioning_metrics folder
MedGRPO Team
a09374a
Since we're using server-side LLM judge for caption evaluation,
the CIDER/METEOR JAR files and captioning_metrics are no longer needed.
Changes:
- Remove evaluation/captioning_metrics/ (entire folder)
- Remove nltk from requirements.txt (not needed)
- Keep eval_rc_vs.py for reference but won't be used
The leaderboard now uses eval_caption_llm_judge.py exclusively
for DVC, VS, and RC tasks with GPT-4 evaluation.
Add server-side LLM judge for caption evaluation
MedGRPO Team
da2e674
Features:
- Server-side GPT-4 evaluation for DVC, VS, RC tasks
- Ensures consistent scoring across all submissions
- Uses V4 Best5 prompts (R2, R3, R8, R5, R4)
- Fallback to semantic similarity if API key not available
- Parallel API calls (10 workers) for fast evaluation
Benefits:
- Fair comparison (no user score manipulation)
- Lower barrier (users don't need API access)
- Reproducible (same methodology for all)
- Secure (API key stored in HF Space secrets)
Changes:
- Add eval_caption_llm_judge.py with server-side evaluation
- Update requirements.txt (openai, torch)
- Remove JAR files from history (cleaned up)
- Add LLM_JUDGE_SETUP.md documentation
Cost: ~$21 per submission for caption tasks
Security: API key never exposed to users
Make evaluation scripts fully self-contained
MedGRPO Team
82f81ab
- Remove all /root/code/Qwen2.5-VL path dependencies
- Fix all eval scripts to use relative paths
- Add captioning_metrics module with all required files:
* CIDEr and METEOR metrics
* JAR files for METEOR computation
* Data files (paraphrase-en.gz)
- Update evaluate_all_pai.py to load modules from local directory
- Change default file paths to require explicit arguments
This makes the leaderboard ready for HuggingFace Spaces deployment
without external dependencies.
