Expert Gaze Improves PET/CT Lesion Segmentation

FDG-PET/CT supports cancer staging and treatment decisions, but lesion assessment remains a demanding reading task and specialist capacity is limited. Existing lesion-detection systems can mark suspicious regions, yet they rarely expose the visual evidence and workflow context behind an expert’s decision. GazeXPErT turns that context into training data: the authors record how specialists inspect PET/CT volumes while detecting and measuring lesions, then pair gaze trajectories with lesion-level annotations and decision windows.

The resulting dataset contains 346 dual-read FDG-PET/CT studies, 9,030 gaze-to-lesion trajectories, and 3,948 minutes of eye tracking sampled at 60 Hz. Its aim is narrower than replacing a reader. The paper asks whether routinely captured expert gaze supplies usable signal for segmentation, dynamic lesion localization, and inference of the reader’s intent.

Core Contribution

GazeXPErT’s central contribution is a four-dimensional representation of expert image review: three-dimensional PET/CT navigation plus time-resolved eye movement. Rather than treating an annotation as the only record of diagnostic reasoning, the dataset preserves where an expert looked, how that gaze moved through a study, and the windows in which a lesion was adjudicated.

That distinction matters for explainable clinical AI. A conventional lesion mask indicates the endpoint of a decision; a gaze-to-lesion trajectory can indicate which anatomy and image slices led to it. The authors package these data in a COCO-style format and describe a folder structure that separates imaging, annotations, and eye-tracking records. This makes the release useful for work on gaze-conditioned vision models as well as interfaces that can respond to a clinician’s ongoing search behavior.

Figure 1 shows the acquisition setup: readers view the PET/CT study in a full-screen application with patient information and a keybinding legend. It is a practical design choice, because the recorded signal comes from a reading workflow rather than an artificial fixation task.

Technical Approach

The collection protocol records gaze during tumor detection and measurement, then aligns the trajectory with lesion annotations and expert decision windows. The paper evaluates three feasibility paths. First, it adds gaze information to a 3D nnU-Net lesion-segmentation setting. Second, it tests whether gaze-trained vision models can localize lesions dynamically as the reader navigates the volume. Third, it predicts intent from raw gaze features.

For the vision experiment, the authors use a Transformer-based Gaze + Vision design. Gaze tokens derived from a trajectory are summarized into a [GAZE] token and processed jointly with a pretrained Vision Transformer. The design does not merely overlay a heatmap on an image; it gives the model a learned representation of the temporal viewing path alongside visual content. Figure 8 depicts that joint encoding path.

Results and Analysis

The clearest quantitative result is the segmentation baseline: the gaze-informed 3D nnU-Net reaches Dice 0.6819, compared with 0.6008 without gaze. That 0.0811 absolute increase is meaningful for a feasibility experiment because it suggests that reader attention contains information beyond the image input used by the comparison model. It does not establish that gaze will improve every segmentation system, but it provides direct evidence that the dataset’s signal is not inert metadata.

For dynamic localization, 74.95% of predicted gaze positions are closer to the tumor, indicating that the gaze-trained Vision Transformer learns an association between trajectory context and lesion location. The intent-prediction experiment reaches 67.53% accuracy and AUROC 0.747 from raw gaze. Those values show separable behavioral signal, although the paper’s reported metrics do not by themselves demonstrate clinical readiness or a reduction in reader error.

The evidence supports the dataset’s immediate claim: expert gaze can improve selected baseline tasks and can be modeled jointly with PET/CT imagery. The larger claim—that this will yield trusted, interactive diagnostic AI—remains prospective. The experiments establish useful signals on a defined collection, not a deployed workflow evaluation with patient or clinician outcomes.

Limits in Practice

The dataset covers 346 dual-read studies, which is substantial for eye-tracked PET/CT collection but modest relative to the variation in scanners, institutions, cancer types, and reading practices encountered in deployment. The reported segmentation, localization, and intent experiments are feasibility baselines rather than comparisons across many contemporary architectures or external sites. Gaze also records attention, not necessarily diagnostic certainty or causal reasoning: a clinician may inspect normal structures, revisit a difficult finding, or look away while deciding. Models trained on these traces need validation that they generalize across readers and improve decision support without encouraging automation bias.