“The majority of published studies are likely to be wrong”

A growing number of analyses – think, for example, of Philip Tetlock’s Expert Political Judgment and N.N. Taleb’s Fooled by Randomness – have in recent years thrown light on what expertise really is, and what it can achieve.  It is not infallibility, of course.  Nor is it  prophetic prowess – the world-class political experts tracked by Tetlock were dreadful failures at predicting events, and the market players skewered by Taleb succeeded hugely until, even more hugely, they crashed and burned.

Here is a WSJ review (via A&E) of a new addition to the literature, a book by David Freedman with the indicative if unsubtle title of  Wrong: Why Experts Keep Failing Us – And How to Know When Not to Trust Them.  We know where Freedman stands, even before reading the first page.

To ask why the experts fail is to tackle the wrong problem.  Why should it be otherwise?  Expertise – whether scientific or of any other kind – doesn’t pertain to a person or to a study but to a process rooted in trial and error.  And as in genetic evolution, most trials – most experiments, most research studies – end in error and failure.

This is inherently the case, but Freedman observes that it is compounded by the pressure to call attention to one’s findings.  Hence the need to make exaggerated claims which the media echo chamber will then validate.

. . . the current market creates the wrong kinds of incentives for doing good research or admitting failure. Novel ideas and findings are rewarded with grants and publication, which lead to academic prestige and career advancement. Researchers have a vested interest in overstating their findings because certainty is more likely than equivocation to achieve all of the above. Thus the probability increases of producing findings that are false. As the medical mathematician John Ioannidis tells Mr. Freedman: “The facts suggest that for many, if not the majority of fields, the majority of published studies are likely to be wrong.”

The problem is that the media tend to validate these findings before they have been properly interpreted, qualified, tested, and either refuted or replicated by other experts. And once a lousy study gets public validation— think of Andrew Wakefield’s claim about autism and vaccination—it can prove almost impossible to invalidate.

I have touched on this subject before.  Life is confusing, and human nature demands reasons and explanations.  The news media, which by its own ideology considers itself an explainer of complex matters to simple minds, trots out its pet experts – scientists and researchers who build sweeping generalizations on slender evidentiary foundations.  Everything is explained.

But, unfortunately, there is no class of wise men on which we can rely for our answers.  There is only trial and error.  For the process to work at all, every assertion must be open – to examination, to repetition, and finally to criticism by experts and lay persons alike.

A system powered by high rates of failure will accumulate knowledge very slowly.  That should be a source of humility for the human race as a whole, but most particularly for those who claim a special expertise.


One Response to “The majority of published studies are likely to be wrong”

  1. Great stuff my friend! “The majority of published studies are likely to be wrong” vulgar morality was a wonderful read. Love reading stuff about this.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: