🏥 MedVidBench Leaderboard

Official Rankings (Verified)

Models on this leaderboard have been independently verified by the benchmark maintainers. We evaluate top community submissions by requesting model API access and running our evaluation pipeline directly.

This ensures reproducible and trustworthy results.

How to get on the Official Leaderboard

1. Submit your model predictions via the "Community Submissions" tab
2. Top performers will be contacted by the benchmark maintainers
3. Provide model API access so we can independently verify results
4. Once verified, your model is added to the Official Leaderboard

For questions, contact us via GitHub.

Community Submissions

Community members run inference on their own machines and upload predictions via the 📤 Submit Results tab. Scores are then evaluated on our server against private ground truth.

Submit Your Model Results

Evaluation is a two-step process:

Step 1: Upload Predictions

Upload your model's predictions only on the MedVidBench test set (6,245 samples).

[
  {
    "id": "video_id&&start&&end&&fps",
    "qa_type": "tal",
    "prediction": "Your model's answer here"
  },
  {
    "id": "another_video&&0&&10&&1.0",
    "qa_type": "video_summary",
    "prediction": "The surgeon performs..."
  }
]

Required fields:

• id: Sample identifier (matches test data from HuggingFace dataset)
• qa_type: Task type (tal/stg/next_action/dense_captioning/video_summary/region_caption/skill_assessment/cvs_assessment)
• prediction: Your model's answer (text output)

Important: Submit predictions only (no ground truth needed). The server merges with private ground truth and evaluates securely.

Step 2: Run LLM Judge (Caption Metrics)

After Step 1 completes, the caption metrics (DVC_llm, VS_llm, RC_llm) will be 0.0. Run the LLM Judge here to compute them using GPT-4.1/Gemini.

• Enter the exact model name you used in Step 1
• The evaluation runs in the background -- you can close the browser and come back later
• Check progress anytime with the "Check Status" button

About MedVidBench

MedVidBench is a comprehensive benchmark for evaluating Video-Language Models on medical and surgical video understanding, introduced in the MedGRPO paper. It spans 8 tasks across 8 surgical datasets with 6,245 test samples.

How Models Are Ranked

Models are ranked by average rank across all 10 metrics — lower average rank = better. For each metric we rank every model (1 = best; ties share the smaller rank), then average those per-metric ranks. This is robust to different metric scales (accuracy 0–1 vs. LLM-judge 1–5) and rewards models that are strong across tasks rather than exceptional on one.

Global ranking across views: the rank shown is computed against the union of all submissions (official ∪ community), so the same model gets the same rank number in either the Official or the Community table — even though each table only displays a subset of rows. The Official table omits rows from the global ranking; the rank column shows each row's position in the full ranking, not its position within the visible subset.

Tiebreakers (applied in order when two models have the same average rank):

1. Number of metrics won outright — a model that's #1 on more metrics wins over one that ties closely on many.
2. Sum of per-metric ranks — catches near-ties where the mean rounded equal.
3. Sum of normalized scores — favors the model with marginally higher absolute scores.
4. Model name alphabetical — final fallback for full determinism.

Benchmark Tasks

Evaluation Metrics

• CVS Assessment (CVS_acc): Accuracy on clinical variable scoring (Cholec80_CVS)
• Next Action Prediction (NAP_acc): Classification accuracy for the next surgical step
• Skill Assessment (SA_acc): Surgical skill level classification accuracy (JIGSAWS)
• Spatiotemporal Grounding (STG_mIoU): Mean IoU over the joint spatial + temporal region
• Temporal Action Grounding (TAG_mIoU@0.3, TAG_mIoU@0.5): Mean IoU over temporal segments, computed at two IoU thresholds (0.3 and 0.5)
• Dense Video Captioning (DVC_F1, DVC_llm): F1 over predicted vs. ground-truth temporal windows, plus LLM-judge caption quality
• Video Summary (VS_llm): LLM-judge caption quality scoring
• Region Caption (RC_llm): LLM-judge caption quality scoring

LLM Judge Details

Caption tasks (DVC, VS, RC) use GPT-4.1 or Gemini-Pro with rubric-based scoring (1-5 scale) across 5 key aspects: R2 (Relevance & Medical Terminology), R4 (Actionable Surgical Actions), R5 (Comprehensive Detail Level), R7 (Anatomical & Instrument Precision), R8 (Clinical Context & Coherence). The final score is the average across these 5 aspects.

Test Set Statistics

• Total samples: 6,245
• Source datasets: 8 (AVOS, CholecT50, CholecTrack20, Cholec80_CVS, CoPESD, EgoSurgery, NurViD, JIGSAWS)
• Video frames: ~103,742
• Training samples: 51,505

Citation

If you use our model or benchmark (MedVidBench / uAI-NEXUS-MedVLM), please cite our paper. To ensure reproducibility and acknowledge the significant investment in establishing this benchmark, please use the following official citation in any published work or public repository:

@inproceedings{su2026medgrpo,
  title     = {{MedGRPO}: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding},
  author    = {Su, Yuhao and Choudhuri, Anwesa and Gao, Zhongpai and Planche, Benjamin and
               Nguyen, Van Nguyen and Zheng, Meng and Shen, Yuhan and Innanje, Arun and
               Chen, Terrence and Elhamifar, Ehsan and Wu, Ziyan},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

License

• Dataset: CC BY-NC-SA 4.0 (Non-commercial, Share-alike)
• Leaderboard Code: Apache 2.0
• Evaluation Scripts: MIT

Contact

For questions or issues, open an issue on GitHub or visit the project page.

Admin Panel

Manage both the Official Leaderboard (verified models) and Community Submissions.

Note: Admin password is set via ADMIN_PASSWORD environment variable in HuggingFace Spaces settings.