Anyone who takes a college critical thinking course learns about the fallacies or common errors in reasoning. One fallacious form of scientific reasoning is confirmation bias. That is, when someone collects evidence, they immediately interpret it in a way that confirms their prior beliefs or theories.
Many of us are prone it, but what is especially upsetting is when the media make this mistake. It might even rival fake news!
Several news outlets picked up on a recent study authored by a Harvard researcher in public health named Ashish Jha. Jha’s study collected and analyzed data from 1.5 million hospitalized Medicare patients. He discovered a marginal but still statistically significant gap between medical results delivered by male and female doctors.
Can you guess which doctors obtained the best outcomes for their patients? You guessed it: women.
On his blog, Jha clearly explains how the slight statistical difference amounts to a much larger effect:
“The first question everyone will ask is whether the size of the effect matters. [ …. T]he effect size is modest, but important. If we take a public health perspective, we see why it’s important: Given our results, if male physicians had the same outcomes as female physicians, we’d have 32,000 fewer deaths in the Medicare population. That’s about how many people die in motor vehicle accidents every year. Second, imagine a new treatment that lowered 30-day mortality by about half a percentage point for hospitalized patients. Would that treatment get FDA approval for effectiveness? Yup. Would it quickly become widely adopted in the hospital wards as an important treatment we should be giving our patients? Absolutely. So while the effect size is not huge, it’s certainly not trivial.
The media craze over this study was astounding. And the titles of the news articles commenting on it were equally sensational.
James Hamblin’s article in the Atlantic: “Evidence of the Superiority of Female Doctors.”
Julia Belluz’s contribution to Vox: “Want to save 32,000 lives a year? Get male doctors to practice more like women.”
David Shaywitz of Forbes exposed the confirmation bas in his article appropriately titled “When Science Finally Confirms Your Cherished Beliefs – Worry.”
Although the science might be good, what worried Shaywitz was how so many seasoned journalists received the conclusion as joyous confirmation of with what they already knew to be true. The conclusion also supported a social justice issue they already championed: pay equity between men and women.
Even Jha acknowledges that the study’s conclusion helps women doctors make the case that they deserve salaries on par with their male counterparts (for the outcomes they deliver are even better than the men’s).
Before this devolves into a war of the sexes though, let’s concede that pay equity is a good thing. Still, as skeptical consumers of scientific research, we should be wary when someone—member of the media, doctor, researcher or other—immediately seizes upon a study’s results and presses an interpretation that supports a belief, theory or cause they already hold dear.
And ask yourself this, as Shaywitz does:
[C]ould a paper with the opposite conclusion to Jha’s (i.e. same paper with the genders reversed) ever be published? Would anyone in academia be foolish enough to try to publish such result–and would such a manuscript ever make it into a peer-reviewed publication, and similarly celebrated by the media?
Cleverly interpreting evidence so that it confirms prior beliefs is an easy mistake to make. But when scientific researchers and the media do it in order to popularize a study’s conclusions, then we have to put on our critical thinking caps, identify the fallacy, and push back against this faulty mode of reasoning.