Peer Review — The Need for Change by Jon Stamford
Twenty years ago I was interviewed for a lectureship at a London teaching hospital.  A week before the interview, the secretary to the committee told me that the panel would most likely wish to discuss one of my publications.  But which one, I mused - perhaps my comparison of striatal and limbic dopamine release?  Or a recently published investigation of antidepressant metabolites?

No sirree - none of those. It was a letter I had recently written, almost as a throwaway, in Trends in Pharmacological Sciences, bemoaning the vicissitudes of anonymous peer review (Stamford, 1988).

Don't get me wrong - I have nothing against peer review per se. I have been reviewer, author and editor and am certainly not advocating its abolition.  Nobody wants that - a recent report by Mark Ware reported no support for the proposition that peer review was completely unnecessary (Ware, 2008).  Peer review has been the backbone of critical scientific appraisal for as long as there have been journals.  Despite this, there is a surprising lack of data demonstrating that peer review actually works.   A 2002 study published in JAMA concluded that "editorial peer review, although widely used, is largely untested and its effects are uncertain" (Jefferson et al, 2002). A Cochrane review in 2007 also emphasized the paucity of data supporting editorial review as a mechanism to ensure quality (Jefferson et al, 2007).

Let's not go there.  Peer review remains the norm and it is popular among authors.  For instance, 90% of all authors felt that peer review had improved their last published paper.  Of particular note, 64% report that the peer review of their last paper had identified scientific errors in the manuscript.  Almost all agreed with the statement that "peer review improves the quality of a published paper" (Ware, 2008).  So peer review is a good thing.  But that's not the issue.

The tenor of my 1988 letter was that anonymous (blinded) peer review was open to abuse, whether wilful or accidental, on the part of reviewers.  Scientists have long complained when articles have been rejected for seemingly specious reasons, or grant applications given short shrift by unsympathetic reviewers. Meadows (1998) reported that one of the biggest concerns among authors were referee bias, real or perceived plagiarism, and delaying publication by tardy reviewing (Rowland, 2002).

Science is an honest even noble endeavour and its business should be conducted openly and honestly.  The review procedure should be no different.  And yet the conduct of peer review seems so often to fall short of the ideal.  A report on the peer review process would likely end with the comment, "could do better"

My suggestion in 1988 was to remove the cloak of anonymity from reviewers.  Apart from lengthy historical precedent, the rationale for anonymity was unclear to me.  Surely, reviewer comments should be open and attributable.  The principal advantage of this, as I saw it, was to raise the level of debate, to discourage snide remarks, and to banish the use of theatrical asides masquerading as insights, a feature recognised by journal editors (Smith, 1999). Not a new idea I agree - even then - but the response at the time suggested I had touched a nerve.

That was in 1988.  So where do we stand 20 years on?  Certainly the issue has not gone away.  The number of editorials discussing peer review in the biomedical press is gradually but steadily increasing (see figure).  For journal editors, the issue is straightforward and hinges on quality.  It can be couched in the form of a simple question: Do anonymous reviewers provide higher quality reviews than assessors whose identity is made known to the authors? 

Apparently not. Godlee et al (1998) had 221 reviewers comment on the same paper, and found it didn't matter whether reviewers were anonymous or not.  Both groups spotted similar numbers of errors.  That conclusion seems to be borne out by an article in the BMJ at the same time (van Rooyen et al, 1999) where manuscripts received at the BMJ editorial office were assigned to either anonymous or identified reviewers.  Independent assessors graded the quality of the ensuing reviews.  To cut a long story short, they found no difference in the quality of reviews by anonymous or identified reviewers.  Nor was there any difference in recommendation.  Identified or anonymous reviewers rejected and accepted similar numbers of papers (see figure).  They even took the same amount of time over the reviews.

So if the editors are happy, what about the authors?  One recent study found a strong author preference for double blind peer review (Ware, 2008).  Although this is another way of levelling the playing field, in practice it is only used by a handful of journals - it is editorially demanding and rarely completely successful.  Too often, authors fail to blind their identity adequately, thereby subverting the process. In my own experience as a reviewer, I have usually been able to identify the authors of blinded manuscripts.

That just leaves us with open peer review. When given a choice between open peer review and blinded review, more than half of the authors preferred identified reviewers (van Rooyen et al, 1998). In the author questionnaire, of 346 responses received, 192 (55%) were in favour of reviewers being identified while 90 (26%) were against. Moreover authors felt that the introduction of open peer review by the BMJ would not influence the decision on whether to submit to the journal. In fact the BMJ concluded that open peer review is feasible in a large medical journal and would not be detrimental to the quality of the reviews.  It doesn't get any clearer than that.

If authors prefer open peer review and it does not reduce scientific quality, what is the stumbling block?  Mainly it seems to be the reviewers - unmasking makes them less likely to participate.  Of the 44 BMJ reviewers who declined to review when told that their names would be passed to the author, 9 cited opposition in principle to open peer review as their main reason for declining (van Rooyen et al, 1999).  In another study, open peer review was an active discouragement for many reviewers with 49% saying that disclosing their name to the author would make them less likely to review (Ware, 2008).

I think that's a bit feeble. If reviewers are reluctant to stand by their comments, perhaps they shouldn't be making them.  Unsigned poison pen reviews and innuendo have no place in scientific review. In any case, the better journals will find a way round this reluctance. The adoption of open review by the BMJ has not brought the journal to its knees.

Perhaps the strongest argument in favour of open peer review is transparency.  Blind peer review makes little allowance for conflicts of interest.  Although reviewers are often asked to declare any conflicts of interest, this is not well policed.  Moreover, authors may well feel that a given reviewer has more conflict of interest than has been declared.  This industry is increasingly scrupulous about disclosures, yet peer review seems to slip through this net.

Ultimately the argument is ethical. If you stand in judgement you should stand openly.

• Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998 Jul 15;280(3):237-40.
• Jefferson, T., Alderson, P., Wager, E. and Davidoff, F. (2002) Effects of Editorial Peer Review: A Systematic Review. JAMA 287: 2784-2786.
• Jefferson, T., Rudin, M., Brodney Folse, S. and Davidoff, F. (2007) Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews. Available at www.cochrane.org/reviews/en/mr000016.html (accessed January 2009)
• Meadows, A.J. (1998) Communicating Research. San Diego, CA: Academic Press.
• Rowland, F. (2002) The Peer Review Process: a report to the JISC Scholarly Communications Group, Available www.jisc.ac.uk/uploaded_documents/rowland.pdf (accessed January 2009).
• Smith R. Opening up BMJ peer review. BMJ. 1999 Jan 2;318(7175):4-5.
• Stamford JA. Peer review: time for a change? Trends Pharmacol Sci. 1988 Jul;9(7):234-5.van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ. 1999 Jan 2;318(7175):23-7.
• Ware M. Peer review: benefits, perceptions and alternatives. PRC Summary Papers 4, 2008 Publishing Research Consortium

Over the last 30 years, Jon Stamford has held research, teaching and editorial positions in academia, industry and medical publishing. He has co-authored 3 books, and published nearly 100 papers. The opinions expressed here are his own and reflect past experience not current employment.