Wednesday 26 October 2016

Can we trust science when we know it's often wrong?

Patricia Casey

Published 08/12/2015 | 02:30

Former doctor Andrew Wakefield.
Former doctor Andrew Wakefield.

'A lot of what is published is incorrect," according to Richard Horton, editor of the world's oldest and most prestigious medical journal, The Lancet. He was speaking at a symposium at the Wellcome Trust in London in early April this year and he wrote in The Lancet on April 11 2015: "The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness."

  • Go To

This is not the first time that questions have been asked about the validity of research published in journals claiming scientific respectability. In 2013 a spoof paper written by journalist John Bohannon of the reputable journal Science, using a false name and fake address (Ocorrafado Cobange of Wassee Institute of Medicine), was submitted to 304 open access journals. The "papers" were slightly different from each other but they were all to the same theme of anti-cancer drugs.

It was written in such a manner that the flaws, in the text and in the graphs were gross and obvious to even the most inexperienced reviewer in the field. By implication these papers should have been rejected at first sight. Yet more than half of the journals accepted the spoof papers for publication.

And of course there was the international furore over the MMR vaccine almost two decades ago. Andrew Wakefield was a British paediatrician who published a paper in The Lancet, claiming that the vaccine caused autism or some form of this condition.

At the time the media were widely sympathetic to this view also and vaccinations against MMR dropped significantly, leading to a rise in measles and mumps and death in some cases.

At a Medical Council hearing in London in 2010 Wakefield was struck off the medical register because he was deemed to have acted improperly. The Lancet then retracted the paper completely.

Another researcher from Tufts University in Boston, John Loannidis raised similar concerns in PLoS (a medical peer-reviewed journal) in 2005 when he said that "It can be proven that most claimed research findings are false".

And in May 2015 Science retracted a paper claiming that a relatively brief conversation with a canvasser who identified themselves as gay and as an advocate for gay marriage could persuade voters in California to become more supportive of gay marriage.

Now Science has revisited the issue and this time is focussing on the psychological sciences in a paper published on August 28 2015. The lead author was Dr Brian Nosek from the University of West Virginia. The team of 270 investigators worked on the Reproducibility Project.

They attempted to replicate 100 studies published in three psychology journals using more complex statistics on the same data set as those used by the original researchers. They found that, depending on the particular test being compared, the replicability varied from roughly one third to two thirds.

In other words in up to 65pc of studies the results, on repeating the analyses, differed from those originally described.

What are the reasons for this? After all, since the Enlightenment, science is regarded as the font of true knowledge, yet the editor of The Lancet writes about science taking "a turn towards darkness".

One explanation is that many journals only publish papers with positive findings even though negative or unexpected findings are just as valuable. This is known as publication bias.

Moreover, in behavioural science there is a large volume of data and by dredging it is possible to obtain some positive results while ignoring those that are negative. This assists in confirming the researchers' own biases and is known as confirmation bias.

In the US particularly many studies in psychology and the social sciences are conducted on "college kids" and the findings do not generalise to the rest of the population, making replication almost impossible when similar methods are used but on other populations.

Deliberate fabrication is also a possibility. An anonymous poll four years ago showed that 15pc of social psychologists admitted using a variety of practices from over-massaging their data to fabricating it, while 30pc reported they had seen others also doing this.

For Horton the explanation for the problems with replication is a combination of flaws in the research design/methods and researcher bias.

Whatever the most common reason for the non-reproducibility of research, these findings are truly alarming and they raise the ugly prospect that instead of seeking truth some research may be facilitating subversion.

And remember, you don't know what you think you know any more!

Health & Living

Read More

Promoted articles

Editors Choice

Also in Life