The same input can give different output. Dice are the obvious cause. Even with the dice removed, a hosted model can vary, because of details such as how requests get batched together on the server.
An agent's success at a task is a rate, not a yes or a no, so one good run shows little and so does one bad run. Whatever must hold every time has to be checked by something that gives the same answer every time.
Say it to your agent
“Run this eval 20 times and report the pass count, not one result. If it is under 18 of 20, list the cases that failed, and don't call it fixed.”