Residual Tail Mass Breaks Survival Score Properness

Survival models are often compared with scoring rules that reward calibrated probabilistic predictions. That premise is less clean under right censoring, because the evaluator observes a mixture of event times and censoring times rather than the event-time distribution itself. This paper studies when commonly used survival losses are proper in that observable-data setting, and the answer is conditional: the squared Brier-type rules behave well only when the tail eventually disappears.

Core Contribution

The central move is a definition of marginal properness for right-censored survival data. Instead of asking whether a score is proper for the unobserved event time $Y$ directly, the paper asks whether the expected observed loss is minimized by the true survival function $S_Y$ when the observed data are $T = \min(Y, C)$ and $\Delta = \mathbb{1}(Y \leq C)$ under independent censoring.

Under that definition, the authors separate three scoring families. The survival Brier score at a fixed time, SBS, and its integrated version, ISBS, are strictly proper when the event or censoring survival tail vanishes. When both tails retain residual mass, as in finite follow-up or cure-fraction settings, the expected SBS is minimized below the true survival probability. The right-censored log-likelihood, RCLL, is treated differently: its expected loss difference decomposes into a KL term plus a tail-mass term, which explains why it separates misspecified models better in the experiments under the regularity conditions tested.

Technical Approach

The paper’s useful result is the closed-form minimizer for the expected SBS at an evaluation time $\tau$. In the residual-tail case, the minimizer is not $S_Y(\tau)$, but a shifted value below it. In the notation used by the authors, the pointwise minimizer has the form

where $a$ is the true event survival at $\tau$, $b$ is the censoring survival at $\tau$, and $\epsilon$ is residual tail mass. The mechanism is direct: observations that remain unresolved beyond follow-up are not neutral for the squared loss; they pull the risk minimizer toward lower survival. Figure 2 shows this empirically, with negative bias growing as residual tail mass increases and with stronger effects at later evaluation times.

The same pointwise argument extends to ISBS because the integrated score minimizes the integral by minimizing each time point. Temporal integration softens the problem, but it does not remove the tail condition. If residual mass persists, the score can still prefer a misspecified survival curve that underestimates survival.

Results and Analysis

The simulations make the theory operational rather than leaving it as a pathological counterexample. The authors first generate Weibull survival and censoring curves, then estimate censoring through Kaplan–Meier in finite samples. Across $K = 10{,}000$ simulations, Table C.1 reports empirical violations for SBS and ISBS. At $n=10$, SBS violations are frequent at the 10th percentile evaluation time: 4,325 violations, or 43.25%. The median and 90th percentile time points show 6.95% and 7.87% violation rates, respectively. ISBS is less affected but not immune, with 320 violations, or 3.2%, at the same sample size.

The violation rates shrink with sample size, which matters for interpretation. By $n=100$, SBS still shows 1.86% to 2.43% violations depending on time point, while ISBS has no flagged violations in that table. At $n=1000$, late-time SBS still has 36 violations, or 0.36%, while the 10th percentile SBS and ISBS columns drop to zero. The paper’s claim is therefore not that SBS always fails in ordinary use, but that its propriety depends on a tail condition that finite follow-up can violate, with late-time evaluation especially exposed.

The misspecification experiments are more relevant for model selection. They use two simulated non-proportional-hazards tasks with Kaplan–Meier curves crossing by a binary covariate: a low-censoring task with 24% total censoring and a high-censoring task with 64% total censoring, both administratively censored at $t_\omega = 10$. Figure 5 compares excess predicted loss against mean integrated absolute error for RCLL, SBS at $t=5$, and ISBS. RCLL tracks increasing misspecification more clearly; SBS compresses differences between models and can blur the ordering; ISBS sits between the two.

Additional sensitivity plots vary test set size and prediction-grid resolution. RCLL becomes cleaner as $n_{test}$ grows from 10 to 500 and remains usable across grid sizes, while ISBS remains noisier under high censoring. Table C.2 gives a concrete high-censoring example for RCLL with $n_{test}=250$: at the full grid size $B=363$, the exact truth scores 0.679 ± 0.049, the oracle scores 0.739 ± 0.063, Cox_int scores 0.937 ± 0.068, KM scores 0.978 ± 0.071, and RSF scores 1.480 ± 0.139. That spread supports the authors’ practical reading: for these simulated tasks, RCLL is the more discriminating evaluation score, while SBS and ISBS require care near the tail.