Speed is only one question. The other is what happens to the person doing the work.
Skill atrophy
Losing a skill, or never building it, because something else does the work for you. With agents, the worry is that you can ship code you can't explain or debug, and that you don't notice until the agent gets stuck and so do you.
Directing an agent well depends on the skill it might erode: you can only catch what you are able to read. You don't feel this happening, since the work still gets done. It comes up in interviews that take the agent away, which some do on purpose.
Say it to your agent
“Explain the approach and ask me what I'd do before you write anything. After you write it, quiz me on three things: why this line exists, what happens if this input is null, and where I'd look first if it failed. Don't give me the answers until I've tried.”