The Peer Review Fallacy

Certain academics today use peer-reviewed studies in the same way theologians in the past would use scripture, as a means of shutting down rational debate and narrowing the breadth of public discourse. If the peer-reviewed study says X, it must be true, especially if it has been blind peer-reviewed and published by a prestigious outlet, or so the all too familiar argument goes. In this piece, I will argue that merely quoting a peer-reviewed study (blind or otherwise) is a poor way of making an argument and a single study is not ‘the science’. Furthermore, the whole peer-review process is deeply flawed, subject to bias and open to manipulation.

In 2018, philosopher Peter Boghossian and two collaborators published the results of an audacious experiment testing the credibility of peer review in certain academic fields. They created 20 totally fictional papers featuring fake data, absurd hypotheses, and nonsensical arguments. These bogus papers deliberately flouted scholarly conventions and standards. Topics included rewriting Mein Kampf in feminist language and likening dog parks to rape culture. Remarkably, 7 of the 20 fake papers were accepted by peer-reviewed journals related to fields like gender studies, race studies and queer studies. Others were under review or requested revisions. Many of the papers were also praised by reviewers for excellence and insight, despite the fact that they were simply made up

There is a long, and not so illustrious, history of flawed studies passing peer review and being published in reputable journals. These include:

·        In 2011, the journal Science published a paper by social psychologist Diederik Stapel that claimed meat eaters are more selfish and less social than vegetarians. However, investigators later found Stapel guilty of research fraud in dozens of papers, including this one.

·        In 1998, The Lancet published Andrew Wakefield's infamous study linking autism to the MMR vaccine. The tiny sample size, biased selection, and unsupported conclusions should have raised reviewer concerns. It took over a decade before the paper was retracted.

·        In 1989, the Journal of Electroanalytical Chemistry published a paper on Cold fusion theory which was based on tentative evidence and was soon thoroughly discredited. Theoretical impossibility and experimental design problems were overlooked.

Social dynamics dictate that once a group of people start working together on a common endeavour in a specific field, a common culture develops and acts as a form of social glue. This common culture, which carries its own norms and values, also acts to keep dissenting or heterodox voices out, since they threaten the in-group solidarity required to make the working environment free of tensions and schisms. In other words, a common outlook creates cohesion and, therefore, people who work together are incentivised to maintain it and repel threats to it. This dynamic alone casts doubt on the ability of reviewers, who are often part of a larger collective with a common culture, to be entirely objective in analysing new and novel ideas.

According to recent research, 8 in 10 academics in the UK identify as left wing whilst the numbers in the US and Canada are quite similar. Other studies have found that between 18-55% of academics would discriminate against a right-wing applicant for a job or a grant. This pervasive bias has led to a culture in which right-leaning academics often self-censor and claim to be working in a hostile environment in which their views are seen as dangerously wrong. Thus, it could be credibly argued that the culture wars have come to academia and this affects the ability of academics to act objectively.

Furthermore, data from personality research suggests that people with common traits, that are largely part of our genetic inheritance, are likely to be drawn to certain professions which exploit those traits. For example, those drawn to the creative arts are more likely to be high in openness whilst those in banking are more likely to be high in conscientiousness. These traits are also correlated with political proclivities and, thus, any particular field of expertise is highly likely to be dominated by those of a similar political persuasion. This, again, increases the chances of biases and entrenched points of view playing a role in reviews, especially if the subject is of cultural or political significance.

At a more individual level, reviewers are often chosen based on their expertise in the field of study and, whilst this may seem like a sensible thing to do, it can also open the door to strong biases impacting the feedback. The more expert one is, the more they are likely to have very strong views and an interest in defending them. In practise, this means they are unlikely to respond favourably to conclusions that contradict their own published research or the consensus of the academic peer group they belong to.

There is evidence to suggest that prestigious researchers tend to attract preferential treatment whilst theoretically novel papers struggle to pass the gatekeepers. This is perhaps because reviewers do not want to be seen as being opposed to someone who has power and clout in the sector they are active in, since that may negatively impact their future prospects. They also do not want to adopt positions that will alienate their peers and so one eye is firmly focused on playing the game to get ahead. Whilst a blind review process, in which author names are concealed, may go some way towards solving this problem, in more niche areas reviewers can often guess the author from the content.

Papers that are aligned with prevailing social trends are also more likely to receive positive feedback. This is, perhaps, because most of us are taken in by cultural winds and form our opinions based on what those around us believe and say. Accepting widely held views is also a means of developing social bonds and being accepted by others. Reviewers are, after all, human and prone to the same inclinations as the rest of us and, thus, the peer review process will always reflect that to some extent.

In my view, where the peer-review process becomes completely nonsensical is when a paper that is based on textual analysis claims to have passed it, which is sometimes the case in political science. Textual analysis, in layman terms, refers to reading books and drawing conclusions based on what authors have written. In other words, there is no experimentation or what most of us consider ‘research’ as such. In this way, politically biased academics get to publish papers that reflect their views and claim they are credible pieces of research because they have passed a peer review process that is not worth the paper it is written on.

It must also be stressed that rejecting a conclusion that is based on a peer-reviewed study is not the same as rejecting the science, as is commonly argued. Science is a process in which conclusions are drawn and facts are arrived at through multiple studies, replication, discourse and analysis over many years. A single study, or even a single set of studies, can be deeply flawed, misquoted or simply not cognisant of all the possible variables. Research is highly complex and often the range of factors we can control, and phenomenon we can measure, is limited.

A better alternative to the peer review process would be an open source review, in which papers are published on an open forum and those who have an interest in the field are invited to offer feedback and ratings. This way a much larger number of people can participate in the process and the opportunities for gatekeepers with biases are decreased. In this ‘Rotten Tomatoes’ style feedback system, credibility can be built on a wide range of feedback amalgamated to give an overall score (using an AI powered algorithm) and a list of key strengths and drawbacks. Of course, those who are qualified in the subject at hand are more likely to be interested and able to offer more detailed responses, whilst bad reviews based on a poor understanding of the subject would be easy to detect.

So the next time someone quotes you a study supporting a conclusion you disagree with and tells you that it must be true because it has been peer-reviewed, use the words of the late Christopher Hitchens and ask – “I am still waiting to hear your argument”.