Progressive Scene Graphs Improve Dynamic Urban NeRFs

Urban driving scenes are a hard case for neural rendering: the camera moves over long distances, foreground objects move independently, and many vehicles are visible for only a few frames. A single global NeRF has too little capacity for that setting, while object-aware methods can lose quality when fast-moving objects are seen sparsely. ProSGNeRF addresses both failure modes by splitting the scene into progressively allocated local scene graphs and giving dynamic objects a foundation-model prior for shape and appearance.

Core Contribution

The paper’s central idea is to make the scene graph grow with the vehicle trajectory instead of pre-partitioning the world or forcing one fixed representation to cover the entire drive. Each local graph contains a background branch, dynamic object nodes, and a far-field branch. When the camera reaches the current graph boundary, the system creates a new local graph with overlapping frames, freezes older graph parameters, and fuses predictions in overlap regions with inverse-distance weights. That design targets the large ego-motion problem directly: capacity is spent where the car has actually driven, while adjacent graphs retain enough overlap for consistency.

Figure 2 shows the full pipeline: SAM-derived masks feed a progressive neural scene graph; background and object branches are modeled separately; DINOv2 supplies object priors; and a far-field loss regularizes sky regions.

Technical Approach

ProSGNeRF decomposes each local graph into a static background model, dynamic object nodes, and far-field handling. The background branch uses multiresolution hash encoding for spatial coordinates and Fourier encoding for view direction, then predicts density and view-dependent color through separate decoders. Dynamic objects are represented in local object-centric coordinates, using tracked 3D boxes and transformations from the world frame into each object node.

The dynamic branch is the more paper-specific part. For each object crop, the method uses DINOv2 to produce a 128-dimensional shape code and a 128-dimensional appearance code. The shape decoder predicts density, while the color decoder conditions on shape, appearance, position, direction, and object pose. The authors add frequency modulation to avoid fitting high-frequency detail too early when only sparse object views are available. The mask starts with low-frequency positional components and gradually admits higher-frequency bands over training, a practical response to the paper’s observation that fast-moving cars create sparse and noisy supervision.

Rendering combines several pieces of supervision. Static background rays use ray-plane sampling, dynamic objects use AABB ray-box intersections with 7 samples per object ray, and volume rendering accumulates color and depth over all intersected dynamic nodes plus the background. The loss includes RGB reconstruction, LiDAR point projection depth supervision, KL divergence between rendered and noisy depth distributions, and a far-field sky loss that pushes density toward zero along rays SAM marks as sky.

Results and Analysis

The quantitative case is broad: KITTI, vKITTI2, Waymo, and nuScenes, with comparisons against NeRF, NSG, SUDS, MARS, PVG, 3DGS variants, and several urban-scene baselines. On the complete KITTI evaluation, ProSGNeRF reports 30.31 PSNR, 0.931 SSIM, and 0.057 LPIPS for image reconstruction, compared with PVG at 29.37 PSNR, 0.922 SSIM, and 0.061 LPIPS. Under the 50% novel-view split on KITTI, it reports 26.55 PSNR, 0.878 SSIM, and 0.086 LPIPS, ahead of PVG at 25.12 PSNR, 0.837 SSIM, and 0.091 LPIPS.

The strongest evidence is not just the best average score, but the behavior as training views become sparse. Figure 6 plots object PSNR and LPIPS against train/test split ratio; ProSGNeRF stays above NeRF, NSG, and SUDS across the plotted range, with the gap larger when fewer views are available. That supports the authors’ claim that the DINOv2 prior and frequency schedule help with sparse dynamic-object observations rather than only improving background reconstruction.

The gains carry over to other datasets. On vKITTI2, the full reconstruction setting reaches 32.42 PSNR and 0.938 SSIM; on Waymo, it reaches 32.55 PSNR and 0.912 SSIM for reconstruction and 29.89 PSNR for novel view synthesis. On nuScenes, the method reports 29.24 PSNR, 0.878 SSIM, and 0.223 LPIPS, ahead of DrivingGaussian at 28.74 PSNR, 0.865 SSIM, and 0.237 LPIPS.

The ablations make the architecture choices more convincing. Removing the progressive scene graph drops KITTI reconstruction PSNR from 30.31 to 26.34. Using only DINOv2 still reaches 29.41 PSNR, while using frequency modulation with an MLP only gives 27.18, suggesting the foundation prior carries much of the object-quality gain and the frequency schedule refines it. The full model is also not cheap: the representative KITTI sequence trains for about 10 hours, uses 14.5 GB peak GPU memory, and renders at 3 FPS on a single NVIDIA A100.

Limitations

The method assumes dynamic objects are rigid or approximately rigid. The authors state that pedestrians and cyclists are harder because a single rigid object-centric transformation cannot fully model their motion. Quality also depends on preprocessing: SAM masks, 3D boxes, and object tracking errors can create incomplete nodes or identity breaks across local graphs. For long occlusion or re-entry, the system may initialize the object as a new node, so strict long-term identity consistency is not guaranteed.