🏥 MedVidBench Leaderboard

Official Rankings (Verified)

Models on this leaderboard have been independently verified by the benchmark maintainers. We evaluate top community submissions by requesting model API access and running our evaluation pipeline directly.

This ensures reproducible and trustworthy results.

How to get on the Official Leaderboard

1. Submit your model predictions via the "Community Submissions" tab
2. Top performers will be contacted by the benchmark maintainers
3. Provide model API access so we can independently verify results
4. Once verified, your model is added to the Official Leaderboard

For questions, contact us via GitHub.

Community Submissions

Community members run inference on their own machines and upload predictions via the 📤 Submit Results tab. Scores are then evaluated on our server against private ground truth.

Submit Your Model Results

Evaluation is a two-step process:

Step 1: Upload Predictions

Upload your model's predictions only on the MedVidBench test set (6,245 samples).

[
  {
    "id": "video_id&&start&&end&&fps",
    "qa_type": "tal",
    "prediction": "Your model's answer here"
  },
  {
    "id": "another_video&&0&&10&&1.0",
    "qa_type": "video_summary",
    "prediction": "The surgeon performs..."
  }
]

Required fields:

• id: Sample identifier (matches test data from HuggingFace dataset)
• qa_type: Task type (tal/stg/next_action/dense_captioning/video_summary/region_caption/skill_assessment/cvs_assessment)
• prediction: Your model's answer (text output)

Important: Submit predictions only (no ground truth needed). The server merges with private ground truth and evaluates securely.

Step 2: Run LLM Judge (Caption Metrics)

After Step 1 completes, the caption metrics (DVC_llm, VS_llm, RC_llm) will be 0.0. Run the LLM Judge here to compute them using GPT-4.1/Gemini.

• Enter the exact model name you used in Step 1
• The evaluation runs in the background -- you can close the browser and come back later
• Check progress anytime with the "Check Status" button

About MedVidBench

MedVidBench is a comprehensive benchmark for evaluating Video-Language Models on medical and surgical video understanding, introduced in the MedGRPO paper. It spans 8 tasks across 8 surgical datasets with 6,245 test samples.

Benchmark Tasks

Evaluation Metrics

• TAL (Temporal Action Localization): mAP@0.5 - mean Average Precision at IoU threshold 0.5
• STG (Spatiotemporal Grounding): mIoU - mean Intersection over Union (spatial + temporal)
• Next Action: Accuracy - Classification accuracy
• DVC (Dense Video Captioning): LLM Judge + F1 for temporal localization
• VS (Video Summary): LLM Judge - caption quality scoring
• RC (Region Caption): LLM Judge - caption quality scoring
• Skill Assessment: Accuracy - Surgical skill level classification (JIGSAWS)
• CVS Assessment: Accuracy - Clinical variable scoring

LLM Judge Details

Caption tasks (DVC, VS, RC) use GPT-4.1 or Gemini-Pro with rubric-based scoring (1-5 scale) across 5 key aspects: R2 (Relevance & Medical Terminology), R4 (Actionable Surgical Actions), R5 (Comprehensive Detail Level), R7 (Anatomical & Instrument Precision), R8 (Clinical Context & Coherence). The final score is the average across these 5 aspects.

Test Set Statistics

• Total samples: 6,245
• Source datasets: 8 (AVOS, CholecT50, CholecTrack20, Cholec80_CVS, CoPESD, EgoSurgery, NurViD, JIGSAWS)
• Video frames: ~103,742
• Training samples: 21,060

Citation

@article{su2024medgrpo,
  title={MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding},
  author={Su, Yuhao and Choudhuri, Anwesa and Gao, Zhongpai and Planche, Benjamin and Nguyen, Van Nguyen and Zheng, Meng and Shen, Yuhan and Innanje, Arun and Chen, Terrence and Elhamifar, Ehsan and Wu, Ziyan},
  journal={arXiv preprint arXiv:2512.06581},
  year={2025}
}

License

• Dataset: CC BY-NC-SA 4.0 (Non-commercial, Share-alike)
• Leaderboard Code: Apache 2.0
• Evaluation Scripts: MIT

Contact

For questions or issues, open an issue on GitHub or visit the project page.

Admin Panel

Manage both the Official Leaderboard (verified models) and Community Submissions.

Note: Admin password is set via ADMIN_PASSWORD environment variable in HuggingFace Spaces settings.