Task Traction Cuts Latency for Open-World Vision

Open-world visual perception has a basic mismatch: most pixels are irrelevant to the task, but conventional pipelines still process the full field before deciding what matters. That cost becomes more visible in autonomous settings with glare, transparent obstacles, scattering, camouflage, low light, and high dynamic range. This paper introduces a neuromorphic vision system that moves part of the decision process into the sensor front end, using polarization-sensitive imaging and an RRAM array to select task-relevant visual information before downstream tracking, segmentation, and trajectory prediction.

Core Contribution

The central idea is a task traction mechanism: perception is guided by the current task rather than by uniform processing of the whole image. The system progressively selects the light-field channel, extracts a region of interest, and predicts where the target will move next. That gives the visual task a smaller and more relevant input. The paper frames this as an information bottleneck implemented in hardware, drawing an analogy to dragonfly vision: select useful cues, suppress background signals, and anticipate short-term target motion.

The work is more than a model proposal. The authors build a prototype with a 12×12 polarization-sensitive photodiode array, a neuromorphic processor, an 8×8 1T1R Ti/Pt-HfO2-Ta/Pt RRAM crossbar, and a Zynq-7020 FPGA. The RRAM devices provide analogue conductance states for gradient computation, temporal cue storage, and read/write control. The device characterization matters because the claimed latency depends on fast conductance modulation and retention rather than only software-side pruning.

Technical Approach

The system has three traction stages. In feature traction, the polarization-sensitive imager captures intensity and polarization components, and the RRAM computation region evaluates gradient entropy inside a predicted task region. The selected local light-field feature is the one with maximum gradient entropy, while a lower-dimensional global signal is kept for monitoring.

In attention traction, temporal changes in the selected feature are passed through a high-pass operation and converted into modulation pulses for the RRAM array. Conductance changes encode temporal cues; thresholding and morphological operations turn those cues into a binary task-region map. In prediction traction, the system computes motion intensity from gradients of the RRAM state in the current task region, maps that motion to a displacement, and updates the predicted region for the next frame.

Figure 3 is the most useful visual summary of the evaluation: it shows the eight scene types, the three downstream tasks, and the side-by-side accuracy and latency comparisons against RGB and SOTA baselines.

A separate lightweight monitoring path watches non-task regions through row-wise and column-wise summaries. If sustained unexpected motion appears outside the current region, the monitoring circuit can introduce additional regions of interest. This is an important design detail because a system that only follows its previous prediction would be brittle under new targets or sudden motion.

Results and Analysis

The strongest reported result is the latency reduction. Across the main open-world benchmark, the paper reports average accuracy gains over SOTA methods of 25.54% for object tracking, 37.73% for object segmentation, and 36.10% for trajectory prediction, while reducing latency by 32.69×, 29.54×, and 29.86× for the three tasks. Against a base RGB model, the average accuracy gains are larger: 724.17%, 424.47%, and 74.31%, with end-to-end processing time reduced by 6.09×, 5.26×, and 5.55×.

The hardware proof-of-concept under specular interference reports 193 μs task latency, a 4.6× reduction, and approximately 128× lower power consumption relative to the full-image baseline. The authors also report task-level gains of 214% for tracking IoU, 358% for segmentation F1, and 62% for trajectory prediction error in that prototype validation. The autonomous-driving evaluation extends the test to high dynamic range and an underground parking garage: compared with full-image light-intensity processing, tracking, segmentation, and prediction improve by 329%, 1116%, and 63%, with 4.1×, 4.6×, and 3.9× latency reductions. Against SOTA methods in the garage setting, the paper reports roughly 195% tracking improvement, 121% segmentation improvement, and an 80% prediction-error decrease, with latency reductions of 17×, 18×, and 17×.

The evidence supports the narrower claim that task-oriented information distillation can reduce visual workload while improving performance in adverse scenes. The broader claim is less fully settled. Much of the large-scale eight-scenario evaluation uses simulated RRAM arrays calibrated from measured pulse data, not the complete fabricated system operating at scale. The prototype is still small, and the paper does not establish how the approach behaves under crowded multi-object traffic, long-horizon driving stacks, or manufacturing variation beyond moderate simulated device variability. Even so, the mechanism is technically coherent: the gains come from reducing where and what the system processes, not from a larger network hidden elsewhere.