Request-Level Sharing Cuts Recommendation Inference Cost

Large recommendation systems often score one user request against many candidate items, yet much of the model computation is repeated as if every request-item pair were independent. That design choice becomes expensive as feature-interaction modules and sequence modules grow. ROCS, short for Request-Oriented Compute Sharing, treats candidate multiplicity as a modeling constraint: compute what depends only on the request once, keep candidate-dependent paths isolated, and defer request-candidate mixing until late in the network.

Core Contribution

The paper’s main contribution is a recommendation inference pattern rather than a single model block. ROCS asks whether a model can preserve the accuracy benefits of deeper interaction and sequence architectures while moving a large fraction of work from per-candidate execution to per-request execution. The authors argue that this is a structural property of recommendation serving: retrieval and ranking systems routinely evaluate hundreds or thousands of candidates for one request, so a request-side subnetwork can be amortized across the batch of candidates.

That makes ROCS different from ordinary kernel tuning or model compression. The method changes where interaction is allowed to happen. Candidate-specific representations are protected until the late stages; request-side representations are reused; and only the final request-candidate fusion is paid per candidate. The paper supports this with public benchmark experiments, production-scale offline workloads, kernel-level latency measurements, and online deployments across Meta recommendation systems.

Technical Approach

ROCS uses two model-side mechanisms. Generalized Layer Masking enforces candidate isolation in feature-interaction architectures by controlling which layers can mix request and candidate features. This lets the model retain cross-feature capacity while preventing early layers from binding a request representation to each candidate. Deep Cross Attention extends the same idea to sequence architectures, where request-side sequence processing can otherwise dominate serving cost.

The systems part is In-Kernel Broadcast Optimization. The issue is that a shared request representation still has to be paired with many candidate representations at serving time. A naive implementation can lose the theoretical savings to memory movement and explicit broadcast operations. IKBO fuses broadcast, attention, and request-candidate computation into GPU kernels so that the shared path is not re-expanded into a costly tensor before the final computation.

The visible kernel breakdown is concrete. An incremental IKBO-LCB kernel falls from 1.944 ms in the original implementation to 1.389 ms after decomposition, 0.798 ms after memory alignment, 0.580 ms after broadcast fusion, and 0.482 ms after pipelining GEMM with broadcast plus fusing request/candidate compute. That is a 75.2% latency reduction from the baseline kernel. In the DCA comparison, IKBO-DCA reports 594 TFLOP/s throughput and 0.230 ms total/attention latency, compared with 245 TFLOP/s and 1.473/0.561 ms for TLX FA3 and 250 TFLOP/s and 1.462/0.550 ms for CuteDSL FA Hopper.

Results and Analysis

The strongest evidence comes from production deployment, where ROCS is tested under the serving constraints it was designed for. The abstract reports up to a 3× QPS improvement on retrieval models without quality degradation, and a short-form video ranking model with a 0.5% relative LogLoss improvement plus a 50% QPS gain. The online deployment section gives three additional examples: a 0.04% topline metric gain with 32% capacity savings on short-form video ranking, on-par topline quality with 38% capacity savings on ads ranking, and a 0.2% topline metric gain with 29% capacity savings on ads retrieval.

Those numbers matter because they show ROCS is not just trading model quality for cheaper inference. In the reported deployments, the method either preserves the online metric or improves it while reducing serving capacity. The gains are also heterogeneous: retrieval sees the largest QPS multiplier, while ranking shows smaller quality gains paired with meaningful capacity savings. That pattern is consistent with the method’s premise. The more candidates share one request-side computation, the more ROCS can amortize.

The public-data setup is narrower but useful for reproducibility. The appendix lists KuaiRand, KuaiVideo, and KKBox, with interaction counts of 4,369,953, 3,239,534, and 7,377,418 respectively. Public experiments are implemented in PyTorch on a single NVIDIA H100 GPU with 96 GB memory, trained with Adam at learning rate 10⁻³, batch size 65,536, and up to 100 epochs with early stopping. Hyperparameters are selected under a forward-cost budget of about 10 MFLOPs per instance for vanilla backbones, and ROCS variants are scaled so amortized per-item cost at 1,000 candidates does not exceed the vanilla budget.

Caveats in Practice

ROCS depends on request-level reuse. The paper states that its benefit diminishes when candidate multiplicity is low, including low-candidate training settings. It also relies on GPU execution support: IKBO is part of the practical result, not an incidental optimization. The main takeaway is therefore conditional but valuable: for high-candidate recommendation serving, request-level redundancy is a model-system co-design opportunity, and ROCS shows that exploiting it can buy capacity without giving up measured ranking or retrieval quality.