Q: At 21.80 s, which surgical instrument is positioned within the bounding box {"bbox_2d": [675.8, 382.0, 980.5, 1000.0]} in normalized [0, 1000] format?
A: The instrument located at these coordinates is the aspirator.
Metric: Exact-match (instrument name)
Q: Are there any instruments actively interacting with the target bladder between 15.03 s and 17.03 s? If so, which ones?
A: Yes, the following instruments are interacting with bladder: forceps, scissors.
Metric: Set F1 (instrument set)
Q: Trace the trajectory of the surgical instrument scissors between 7.73 s and 15.83 s and provide the spatiotemporal coordinates for its most extreme top and bottom positions in normalized [0, 1000] format.
A: Extreme Top: {"bbox_2d": [581.0, 251.2, 909.2, 515.6]} at t=7.73 s; Extreme Bottom: {"bbox_2d": [518.5, 460.2, 735.3, 720.3]} at t=15.83 s.
Metric: Spatiotemporal error + center distance
Q: Following the completion of idling by the forceps, identify both the subsequent verb AND the subsequent target.
A: After idling, the forceps transitions to retract on the bladder.
Metric: Exact-match (verb + target)
Q: Between 21.20 s and 25.73 s, what is the observed change in the spatial relationship between the grasper and the aspirator?
A: The grasper approximately maintained its distance from the aspirator (distance change 0.02; below the 0.30 significance threshold).
Metric: Relation accuracy + numeric relative error
Q: What is the highest concurrent tool count observed throughout the segment spanning from 20.63 s to 27.17 s? Options: A) 4 B) 3 C) 5 D) 0
A: Answer: B) 3
Metric: Exact-match accuracy
Surgical video understanding is a crucial prerequisite for advancing Computer-Assisted Surgery. While vision-language models (VLMs) have recently been applied to the surgical domain, existing surgical vision-language datasets lack in capturing and evaluating complex, interleaved spatial-temporal dynamics. Creating large scale datasets that accurately represent fine-grained spatial-temporal relationships in surgical videos is challenging due to costly manual annotations or error-prone generation using large language models.
To address this gap, we introduce the SurgSTU-Pipeline, a deterministic generation pipeline featuring temporal and spatial continuity filtering to reliably create surgical datasets for fine-grained spatial-temporal multimodal understanding. Applying this pipeline to publicly available surgical datasets, we create the SurgSTU dataset, comprising 6711 video clips densely extended with 150k fine-grained spatial-temporal question-answer samples.
Our comprehensive evaluation shows that while state-of-the-art generalist VLMs struggle in zero-shot settings, their spatial-temporal capabilities can be improved through in-context learning. A fine-tuned VLM on the SurgSTU training dataset achieves highest performance among all spatial-temporal tasks, validating the dataset's efficacy to improve spatial-temporal understanding of VLMs in surgical videos.
The SurgSTU-Pipeline turns surgical videos and their instrument-localization / action-triplet metadata into structured Event Tuples, applies spatial / temporal / semantic continuity filtering, and completes predefined QA templates. Deterministically, with no LLM in the loop.
Bbox grounding, trajectories, frame-region classification, reverse grounding, and explicit refusals.
locatewindowidentify_bbox[412, 188, 591, 374] at 4.20 s?"segmentclosest[0.43, 0.58] at 4.20 s?"trajectoryrel_posrel_changelocate_by_actionlocate_by_targetrefusal_absent_instrumentFour-way and binary MCQs over instrument classes, presence, and counting — with distractors drawn from the instrument vocabulary.
ClassesExistence:globalExistence:local_instExistence:local_targetCounting:distinctCounting:specificCounting:concurrentWhat-is-doing-what at a point, plus temporal aggregation, ordering, and compositional whole-clip queries.
Bucket A — temporal aggregation
interaction_durationinteraction_countlongest_continuous_actionidle_durationBucket B — ordering
first_appearance_timelast_appearance_timefirst_actionaction_sequenceBucket C — whole-clip MCQ
most_active_instrumentmost_target_diversity_instrumentdominant_verbdistinct_targets_touchedLegacy + refusal
target_interactionaction_statusnext_actioncomparisonreverse_targetrefusal_no_specific_actionSurgSTU currently stands on publicly available surgical datasets. The cholecystectomy half builds on CholecT50 (Nwoye et al.) for instrument-verb-target triplets and on CholecTrack20 (Nwoye et al.) for instrument bounding boxes. The prostatectomy half builds on ProstaTD for instrument-verb-target annotations; ProstaTD bounding boxes are internally annotated by the authors.
@inproceedings{maack2026approach,
title={An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models},
author={Maack, Lennart and Schlaefer, Alexander},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={2945--2954},
year={2026}
}