Legal Prediction Models Exploit Tribunal Text Shortcuts
Auditing 33,158 UK employment claims shows outcome-revealing phrases can deliver 56.07 Macro-F1 by themselves, close to full-text classifiers.
Underlying Paper
Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal
Current Legal Judgment Prediction (LJP) is constrained by its reliance on post-hoc judicial materials, increasing the likelihood that models perform retrospective classification rather than true forecasting. This paper empirically investigates shortcut learning in this context by studying claim-level outcome prediction in UK Employment Tribunal (UKET) decisions. Using a corpus of 33,158 individual claims, we predict outcomes from claim texts and LLM-extracted case summaries, evaluating models ranging from interpretable TF-IDF-based classifiers to black-box LLMs. While headline predictive performance figures appear strong, we demonstrate that such performance in LJP systems trained on post-hoc judicial text can be driven by the retrospective nature of the source material. Stratifying the test data by human judgments of leakage reveals that performance increases where outcome-revealing cues are embedded in the narrative. Moreover, a model trained on just the 4% of features identified as leakage achieves high performance, outperforming human experts. These findings substantiate concerns that LJP performance may be exaggerated by linguistic artefacts. Yet this vulnerability is not fatal to the research agenda. Instead, post-hoc judgments might be treated as potentially contaminated texts, requiring active auditing. Retraining models after masking leakage features results in only a negligible reduction in Macro-F1. Hence, while models will opportunistically exploit shortcuts when available, they remain capable of extracting useful predictive signals when these artefacts are removed.
Legal judgment prediction is supposed to forecast outcomes from information available before a decision. Much of the field instead trains on post-hoc judgments: texts written after the outcome, often shaped by the judge’s reasoning and final disposition. This paper tests how much that matters in UK Employment Tribunal claim-level prediction, where each case may contain several separate claims with different outcomes.
The authors’ answer is measured but uncomfortable. Models can predict claim outcomes from short claim descriptions and case summaries, but part of that performance comes from retrospective language that gives away the result. The paper’s contribution is not another leaderboard; it is an audit of where the signal comes from.
Core Contribution
The study reframes Legal Judgment Prediction as a contamination problem. Instead of treating leakage as a dataset-level defect, the authors look for outcome-revealing cues inside otherwise usable post-hoc text, then ask three questions: do models perform better on leaky claims, do high-weight features contain such cues, and does performance survive when those cues are removed?
The dataset contains 13,253 UKET judgments and 33,158 individual claims. A large LLM annotation pass supplies 32,030 training labels, while 1,128 test claims are annotated by a human expert as ground truth. Two trained legal annotators also provide matched-input human baselines using the same restricted inputs as the models, rather than the full judgment text. That design matters because it avoids comparing models with humans who saw different evidence.
Technical Approach
The evaluated systems span interpretable and black-box settings: TF-IDF logistic/SVM-style classifiers over unigram, bigram, and trigram features; a frozen DistilBERT sentence-embedding model classified with a linear SVM; and DeepSeek used as a zero-shot and few-shot generative classifier. The outcome space is collapsed to Win, Loss, and Other, with Macro-F1 used because the class distribution is uneven: 39.5% Win, 48.8% Loss, and 11.7% Other.
The diagnostic pipeline has two parts. First, the test set is split into clean and leaky subsets using human judgments of whether the model inputs reveal the answer. Second, the authors audit the top 5,000 trigram features with an LLM, producing a contaminated feature set of 200 trigrams, or 4% of the feature space. A law professor and a law PhD candidate then blind-check a stratified sample of 30 features; they agree on the binary leakage label 73% of the time, while the LLM agrees with the PhD candidate 67% and with the professor 53%.
Figure 2 shows the central diagnostic pattern: most models score higher on claims marked as leaky than on claims without apparent leakage, with gaps from +0.05 to +0.11 for several main systems. The unigram TF-IDF model is the exception at -0.01, which fits the mechanism: many leakage cues are multi-word phrases such as “judgment for claimant” or “claimant failed to.”
Results and Analysis
Headline performance looks respectable before the audit. The best supervised systems reach about 0.623 Macro-F1, above the zero-shot and few-shot LLMs at 0.47 and 0.54, and above the matched-input human benchmark at roughly 0.51. Those numbers alone would suggest that simple text classifiers can beat constrained human readers on this UKET task.
The leakage analysis changes the interpretation. In the TF-IDF trigram model, leakage features receive larger coefficients and heavier tails than ordinary legal language. Figure 3 shows that the model is not merely exposed to contaminated features; it assigns some of them high decision weight. The authors cite examples including “unauthorised deduction from” for Win predictions and “claimant failed to” for Loss predictions.
The intervention experiments are the strongest part of the evidence. A leakage-only model trained only on the 200 contaminated trigrams reaches 56.07 Macro-F1, well above naïve baselines and close to full-text models. When the standard trigram model is tested after masking those trigrams, Macro-F1 drops from 62.27 to 60.85, showing some inference-time reliance. But when the model is retrained from scratch with leakage features removed, it still reaches 61.62. That is the paper’s most useful result: post-hoc tribunal text is contaminated, yet not empty of legally relevant signal.
Limits and Takeaway
The evidence supports a narrower claim than “legal prediction works.” It supports the claim that common LJP evaluations can overstate forecasting ability when they rely on post-decision texts, and that leakage audits should accompany headline Macro-F1 scores. The remaining signal after retraining is encouraging, but it does not establish performance on genuine pre-decision filings such as ET1 claim forms or ET3 responses. The paper also notes a separate risk for LLM predictors: public judicial decisions may appear in pretraining data, making apparent prediction hard to separate from memorization or temporal lookahead bias.
Evidence Box
strongKey Claims
- •Post-hoc tribunal judgments contain outcome-revealing shortcut cues
- •Standard LJP models exploit leakage when it is available
- •Useful predictive signal remains after masking identified leakage features
- •Claim-level UKET prediction gives a finer diagnostic setting than case-level labels
Key Results
- •33,158 individual claims from 13,253 UKET judgments
- •Best supervised systems reach up to 0.623 Macro-F1, versus about 0.51 for matched-input human benchmarks
- •200 contaminated trigrams represent 4% of the audited feature space and yield 56.07 Macro-F1 alone
- •Masking leakage trigrams lowers Macro-F1 from 62.27 to 60.85 at test time, while retraining after removal reaches 61.62
Limitations & Caveats
- •Evaluation still uses post-hoc judicial texts rather than genuine pre-decision filings
- •Leakage-feature validation sample covers 30 trigrams, with 73% binary agreement between human reviewers
- •LLM predictors and annotators may face memorization or temporal lookahead bias on public judgments
- •Contaminated feature set targets high-signal trigrams and may miss subtler leakage cues
