Surgical Pretraining Improves Adapted Robotic Video Encoders

Robot-assisted surgery has far less public video data than laparoscopy, leaving many surgical foundation models trained or judged on a different visual domain. OmniRAS addresses both sides of that problem: it continues pretraining V-JEPA-2.1 encoders on a 19-source surgical catalog and pairs them with robotic-cholecystectomy annotations, fixed splits, and probe protocols. The result is not a claim that surgical pretraining universally replaces general-video representations; it is a measured advantage once the downstream encoder is allowed to adapt.

Core Contribution

The paper contributes 1B- and 2B-parameter OmniRAS encoders, continued from V-JEPA-2.1 on approximately 2,650 hours of surgical video, 51% of it robotic. Training reaches 256 compute nodes with a global batch of 6,144. The accompanying evaluation is unusually broad for this setting: two newly annotated robotic-cholecystectomy datasets, six downstream tasks, frozen and last-four-block adaptation protocols, and 254 seed-level runs, including 109 partially fine-tuned runs.

The dataset work is consequential because phase labels in surgery can be genuinely concurrent rather than simply mistimed. On a 10% YT-Chole re-annotation sample, the eleven-class phase ontology reaches mean inter-rater agreement of κ=0.807 with a ±4-second boundary tolerance. Pooling the five concurrent Calot's-triangle sub-activities raises it to 0.833, which supports the authors' argument that some residual disagreement reflects alternative valid descriptions rather than annotation noise. The paper also introduces a robotic-cholecystectomy tool–verb–target task, extending evaluation beyond coarse phase labels.

Technical Approach

OmniRAS is a continued-pretraining campaign rather than a new task head or temporal decoder. The authors initialize from Meta's V-JEPA-2.1 checkpoints, use a masked-prediction objective over 16-frame 384² clips, and compare the surgical 2B checkpoint with its raw Meta 2B starting point. Downstream probes either freeze the encoder or unfreeze its final four blocks; the latter, called FT4, is depth-matched across encoders rather than parameter-matched.

That choice matters for interpretation. Frozen probes ask how immediately accessible a representation is, whereas FT4 asks whether a useful surgical initialization can specialize. The authors find that these rankings often differ. On GraSP, the best phase- and step-recognition results occur under different adaptation settings, so the study avoids treating one regime as the definitive measure of a surgical foundation model.

The paper also adds a label-free diagnostic based on V-JEPA masked-prediction fidelity. Across 60 of 60 paired clip-level comparisons, the surgical 2B checkpoint has higher matched cosine similarity than the raw Meta checkpoint; the reported mean per-unit differences range from +0.0157 to +0.0220. This supports a shift in the pretrained predictive representation, though it does not establish a shared-space ranking between models.

Results and Analysis

The clearest gains appear after adaptation. On OmniRAS-PR, OmniRAS 2B FT4 reaches 61.11 ± 0.85 phase mAP and 67.15 ± 0.78 segmental F1@10. On GraSP, the OmniRAS 2B FT4 model reports 83.41 ± 0.63 phase mAP and 57.85 ± 1.29 step mAP, compared with TAPIS at 76.75 ± 0.74 and 38.54 ± 0.33. That is a large step-recognition margin, where workflow detail is harder to retain than phase identity.

Figure 6 shows the same pattern on OmniRAS-PR: adapted 2B OmniRAS is at the top of the displayed phase and segmentation panels, while frozen comparisons are more mixed.

On YT-Chole, the 37M-sample OmniRAS 2B model reaches 39.92 ± 0.91 IVT mAP with FT4. Across SAR-RARP50, SARAS-ESAD, GraSP, OmniRAS-PR, and YT-Chole, the authors report the strongest adapted result in each task family. The evidence is strongest for OmniRAS as an initialization for partial adaptation, not as a drop-in frozen feature extractor: the paper explicitly says frozen orderings vary by benchmark and can reverse on GraSP.

Caveats in Practice

The training catalog, sampling, and masking choices are changed together, so the study cannot attribute the transfer gain to scale, robotic-video proportion, or any single pretraining decision. The private data contribution is in-house, while public benchmarks remain necessary external anchors. Continued surgical pretraining also reduces general-video performance: the 36.86M-sample checkpoint scores 53.82 top-1 on SSv2, versus 60.27 for the raw model. For practitioners, the paper is best read as evidence for domain specialization with a measurable generality cost, not as proof that one encoder dominates across deployment settings.