Code Transformations Detect LLM-Written Python Across Models

Detecting generated code is harder than detecting generated prose because code has less harmless variation: many tokens are fixed by syntax, tests, or APIs. Existing text watermarking methods try to bias token choices during generation, but that requires access to the model’s decoding process and can interfere with code quality. This paper proposes ACW, a post-processing watermark that treats code transformations as the carrier rather than token probabilities.

The result is a watermarking scheme that is model-agnostic, training-free, and usable with closed-source LLMs. The cost is a narrower technical scope: the implementation evaluated here targets Python, and its reliability depends on a hand-built library of transformations that preserve semantics and remain detectable after rewriting.

Core Contribution

ACW’s central idea is to watermark code by applying selected transformations that should not change behavior: refactoring, operand reordering, and formatting changes. The watermark is not a visible string or a learned signature. It is the pattern of which applicable transformations have been applied.

That distinction matters. A detector can later re-run the same ordered transformation library on a candidate program. If applying a selected transformation changes the code, ACW treats that as evidence the transformation was absent; if it does not change the code, the transformation may already have been applied. The paper makes this workable by requiring transformations to be semantic-preserving and idempotent, so repeated application converges rather than corrupting the program.

Figure 1 shows the full pipeline: LLM code generation, transformation-set selection, watermark embedding, and detection by reapplying the applicable transformations.

Technical Approach

The method uses 46 Python transformations divided into three groups. Refactoring rules include removing unnecessary else branches, converting loops to list comprehensions, removing redundant casts, merging comparisons, and inlining trivial return variables. Reordering rules canonicalize commutative expressions or comparisons using a stable hash. Formatting rules add blank lines or normalize whitespace and bracket alignment.

Embedding first identifies which transformations are applicable to a generated code snippet, sorts them by a secret ordering derived from the watermark, and applies the first $n$ transformations. Identification repeats the applicability analysis and checks whether the selected transformations have already taken effect. The authors also discuss the chance of false identification for naturally written code: if a random transformation appears to have been applied with probability under 50%, six independent selected transformations would put that probability below 1.6%.

The experiments use function-level code from MBPP, APPS, and HumanEval generated by GPT-4, GPT-4o, and Qwen2.5-Coder-14B-FP16, plus project-level code from the SRDD prompt dataset generated with GPT-4 and GPT-4o. Baselines include WLLM, WaterMax, SWEET, and STONE, reimplemented or adapted where necessary.

Results and Analysis

On discriminability, ACW is much stronger than the token-watermark baselines in this code setting. Across the reported datasets it achieves accuracy above 97%, true positive rate above 94%, false positive rate below 1%, precision above 99%, and F1 above 97%. On MBPP-Qwen, for example, ACW reports 98.81% accuracy, 97.63% TPR, 0% FPR, 100% precision, and 98.80% F1. The closest baselines are far behind on the same setting: SWEET reaches 75.64% accuracy and STONE 70.84%, while WLLM and WaterMax perform worse.

The efficiency result is the paper’s most practical advantage. ACW adds less than 0.1 seconds per function-level sample on all datasets; on MBPP-Qwen it takes 0.03 seconds, compared with 1.10 seconds for the original generation. It also adds no GPU memory overhead. By contrast, WaterMax takes 9.78 seconds on MBPP-Qwen, SWEET takes 6.03 seconds, and the token-based methods require tens of gigabytes of GPU memory in the reported setup.

Utility is mostly preserved because ACW rewrites code after generation rather than changing the model’s token distribution. On MBPP-Qwen, pass rate drops from 63.12% to 62.29%, a 0.83-point degradation. On GPT-generated project code from SRDD, the reported pass rate remains 100%. In a larger project-level check using 507 Python projects and 50,700 unit tests, 93.31% of system tests and 99.20% of unit tests pass before watermarking, and the results remain unchanged after watermarking.

Resilience and Caveats

The attack experiments support the claim that ACW survives common code edits better than generation-time token watermarks. Under default and maximum code optimizations, the paper reports TPR and one-minus-FPR mostly near the unattacked results while pass rates are largely preserved. Figure 6 gives the default optimization case, where MBPP-GPT-4 keeps 97.02% TPR and 99.74% one-minus-FPR, with pass rate unchanged at 70.21%.

The weaker point is adaptive modification. When a CodeT5+ attacker is trained to reverse watermarked code, MBPP-GPT-4 TPR falls from 97.94% to 71.56%, while pass rate drops from 70.21% to 63.74%. That is still useful signal, but it shows the watermark is not immune to an attacker who knows the mechanism and can learn inverse rewrites. The paper is best read as evidence that transformation-based code watermarking is a practical default for Python LLM outputs, not as a complete authorship proof under a determined adversary.