Chapter 18: The proof

How strong is this evidence

Also called Evidence hierarchy

A rough ranking of how far a kind of study lets you believe its claim: randomised trials near the top, then comparisons over time, then surveys and benchmarks, then single stories. A ranking only makes sense for one question at a time.

An agent summarising research gives a survey and a trial the same confident sentence, so the ranking has to come from you. And the person across the table has heard all the loud numbers. You can't check every study, but you can find out what kind it is in a minute.

Say it to your agent

“You cited a study for that. Tell me what kind it is: randomised trial, comparison over time, survey, vendor benchmark or single story. Give me the sample size, the date of the tools it tested, and the sentence in the source. If you haven't opened the source, say so.”

The workshop

Name a word, or describe what you mean in everyday language.