Does an AI watermark affect code quality?

Google's SynthID watermark, covered in "AI watermarks aren't that invisible, and they hurt output", left each text just as good as before but made repeated requests converge. Twenty emails came back with one subject line.

Code is a tougher test. It gives a model far fewer choices than prose, and lots of people assume code can't carry a watermark at all. I tested the same watermark against code and posed four questions. Will it introduce bugs? Does it alter the complexity or design? Do repeated requests still reach the same result? And is the watermark detectable in the code?

The testing process

I used the same setup as before. Gemma 4 26B ran on my Mac with the open-source SynthID code and my own keys, at temperature 1, top-k 64 and top-p 0.95, with thinking turned off. I ran each request three ways, with no watermark, with key K and with a second key, K2. The first attempt used the same random draws across all three columns. The next attempt used a different set, and so on.

I ran two sets of tasks.

• 164 HumanEval+ problems, with five tries per problem. HumanEval+ takes the original HumanEval functions and adds tougher tests, which catches minor bugs.
• Six tasks, each with 20 attempts, split between two languages. The Python jobs are a cache with expiry, an interval set, the cheapest path across a grid and tracking down a bug. The Rust jobs are an expression parser and a k-way merge. Their test suites include speed checks.

I ran every program in a sandbox without network access. The complexity metrics came from radon and complexipy, and I used ruff and clippy for linting.

Code quality

Key K scored 2.4 points lower on HumanEval+ than the no-watermark runs, while key K2 scored 1.6 higher. Two keys moving the opposite way is just noise. Across both keys the average was 90.3% versus 90.7%, with a 95% band running from 2.2 points worse to 1.3 better. Line count, branching and linter findings all stayed within a tenth of each other. The design tasks and the Rust compile errors showed no pattern either.

That matches how SynthID works. When you average over all the possible keys, the odds stay exactly the same, so the model has no reason to prefer worse code. Earlier watermarks raised the odds for a fixed share of tokens, which hurt code and were hard to spot in it, since code leaves so few choices (Lee et al.).

Variation between tries

Twenty caches without a watermark used the reentrant lock 12 times and the plain lock 8 times. Key K used the reentrant lock every time and key K2 used the plain lock every time. This matches the subject line from the text test. Each key picks one of the model's own answers and keeps returning it.

When the same problem was attempted twice on HumanEval+, the code matched 37% of the time without a watermark and 72 to 77% of the time with one.

The effect isn't just about names and spacing. After renaming variables and dropping comments, five tries still produced 2.3 different programs without the key and only 1.5 with it.

On the six design tasks, the keys gave 21 and 18 designs. No watermark gave 32. Randomly shuffling the same tries almost never produces a gap like that (p below 0.01).

The second attempt counts

People correct a wrong answer by asking again. This works only when the next answer actually changes.

HumanEval/120 wants the k largest numbers in sorted order. Without a watermark, one in five attempts grabbed the last k items of the sorted list, which fails when k is 0. Key K picked that version every time. Key K2 never did.

def maximum(arr, k):
    return sorted(sorted(arr)[-k:])

def maximum(arr, k):
    return sorted(sorted(arr, reverse=True)[:k])

The pattern shows up across the benchmark. Without a watermark, 13 problems pass on some tries and fail on others, so asking again can fix them. With a key there are 8 and 6, and the key settles each one. Of those 13, key K made 5 always pass and 4 always fail, and key K2 made 9 always pass and 1 always fail. Key K also lost 6 problems that the plain model solved at least once in five tries. Shuffling the same tries at random loses about one, with p below 0.001.

The watermark doesn't make code worse. It brings the attempts closer together. Whether that helps or hurts depends on which answer your key lands on, one problem at a time.

Why it happens

SynthID scores each candidate using the key plus the four tokens that came before it. A request always starts from the same spot, so two runs face the same options and get the same scores, which means the same candidate wins. Without the key, only the dice decide. When two tries reach a real decision at the same point, they choose the same token 65% of the time without a watermark and 94% of the time with one.

I ran the same four tasks again, but this time I used a fresh key for each request. The share fell back to 66%. The design tasks came back with 10 designs this time, compared to 11 without a watermark. Reusing a single key is what drives the convergence, not the watermark itself. A key per request costs something too. The detector then has to know the key for each request or go look for it.

Can a watermark be found in code?

The idea that code can't be watermarked is partly true. A short function gives the key almost nothing to work with. A HumanEval+ answer has a median of 78 tokens, and in a typical one the model has a real choice at only 3 of them. Under 300 tokens, just 1% of watermarked answers get flagged. Long answers are another story. Over 1,000 tokens, 70% are flagged, and no answer without a watermark was flagged at any length.

The signal hides in the comments and in the text around the code. Comments score about 3 points per 100 tokens, while code scores under half a point, barely above the noise level of unmarked code. Remove the comments from those long answers and the flagged share drops from 64% to 6%.

I asked the model for code with no comments and no docstrings, adding that instruction to all six design tasks. It left the mark in place. The model obeyed the rule on the short tasks, and those were rarely flagged in the first place. On the two hardest tasks it kept its reasoning in comments, about 100 lines each on the interval set, and those answers stayed flagged in 16 to 20 of 20 tries.

Across the board, 89 of the 240 watermarked answers got flagged, versus 81 when comments were allowed. Long code can carry the mark by itself. When no comments were present, 9 of the 20 Rust parsers from key K were flagged on the code alone. Key K2's were never flagged.

Limits

• Like the text test, I ran this with my own keys, the published algorithm and one model set to temperature 1 and top-p 0.95. I didn't test a lower temperature, and Google doesn't say whether it assigns the same key to every request.
• I tried each HumanEval+ problem five times and used six design tasks. The retry data comes from 13 problems, and the key-per-request check covers four tasks.
• HumanEval has been around long enough to end up in the model's training data, and the test only repeated the exact same prompts.

References

1. Dathathri, S. et al. Scalable watermarking for identifying large language model outputs. Nature 634 (2024).
2. Hugging Face. Introducing SynthID Text, the open-source implementation used here.
3. Liu, J. et al. Is Your Code Generated by ChatGPT Really Correct? NeurIPS 2023. The source of HumanEval+.
4. Chen, M. et al. Evaluating Large Language Models Trained on Code. 2021. The source of HumanEval.
5. Lee, T. et al. Who Wrote this Code? Watermarking for Code Generation. ACL 2024.