Confidence interval
A range around a result showing how far it could be from the truth just from the luck of what got sampled. Fewer samples, wider range. It covers bad luck only: a biased study can have a narrow interval around the wrong answer.
An agent doesn't give the same answer twice, so one or two good runs tell you very little. A prompt that passed 15 times out of 15 could still fail one run in five. This is a daily tool when agents do the work, and not only something for reading papers.
Say it to your agent
“You ran this prompt 3 times and it passed 3 times, which proves very little. Run it 30 times, each in a fresh session, count how many pass tests/holdout.test.ts, and give me the Wilson 95% interval. Don't call it reliable unless the low end is above 90%.”