It is widely assumed that the peer review process improves the article that is eventually published. This concept is taken largely as an article of faith except for one miniscule question posed to a range of published international academics as part of a large survey in 2009; in this survey, 91% of respondents, none of whom were medical professionals, agreed with the idea that the revisions made to their article as a result of the conventional peer review process improved the end product. Of note, most of this subjective sense of manuscript improvement was felt by authors to be within the discussion section. The response to this one question represents the only evidence of which I am aware that demonstrates peer review resulting in better published papers.
Obviously one cannot simply dismiss the findings in the above study out of hand. However, it seems to be worth asking whether or not peer review has the same advantages outlined in the above survey when considered in the rich world of observational studies that describes so much of medical publishing. Moreover, is an improvement in a article’s discussion section significant enough to warrant all the added time and effort on the part of the very busy physician authors who write most medical articles and who represent some of the busiest individuals on our planet?
As an editor of a journal myself (Cureus) I cannot help but ask these questions. I also pose this line of inquiry now, however, due to a recent discussion with a colleague who is a very senior and internationally respected neurosurgeon. He is about to publish an article in a well-known neurosurgery journal having now jumped through the usual reviewer hoops requiring several revisions and months of effort. I asked this colleague whether his article was any better after all this effort. His simple response: “NO!” In fact, in my own academic career, which involves more than 150 peer-reviewed articles, I find it difficult to recall a single instance where I felt the final published article was meaningfully improved through the peer review process.
Oh, I jumped through plenty of hoops, like routinely needing to add additional references so that my paper would cite the reviewer’s own publications, but sadly I have found far too much of the peer review process to be merely an exercise in power and vanity. With so many diseases so poorly treated, and so much clinical research that needs to (or could) be published, is all this inefficiency justifiable? It is for this very reason that we at Cureus emphasize a much more efficient post-publication “peer review” process, or as we term it, SIQ (Scholarly Impact Quotient). I would very much like to hear from any readers of this blog who have published in conventional peer-reviewed journals – did the review process improve the ultimate published article?
The Cureus Blog 
I have published a number of peer-review articles (http://orcid.org/0000-0002-6921-9567). I would agree with Dr. Adler that the benefit is not always worth the delay and hassle. I also would say that most of the criticisms of my eventually published papers represented either reviewer misunderstandings or subjective judgments of expected impact. Publications in (I think) PLOS ONE have reviewed evidence showing that high-impact journals disproportionately favor findings that eventually prove false (because many true findings aren’t newsworthy). On the other hand, I am not sure I would take the argument as far as Dr. Adler does. I’ll make two points.
First point: I think review can improve papers. We recently had a paper accepted in Annals of Neurology. I had wanted to publish it in the easiest possible venue a year ago because I didn’t want it to take so long, I didn’t want to be “scooped,” and I didn’t really care where it was published because people find articles on PubMed or Google these days anyway. However, after several revisions I do believe the paper is much clearer and easier to read, and I imagine it will get a lot more attention than it would in a lesser-known journal.
Second point: Some papers deserve to be rejected. I recently reviewed a couple of papers for a well-known neurology journal. Each described an industry-funded study in the most effusive of terms. The articles were good for the respective companies, but only because the weaknesses of the study were underplayed and the strengths were overstated. I would not be ashamed to sign a post-publication review to that effect, but why would I?
In short, I see serious flaws in peer review, and I am eager to explore new publishing models like that of Cureus (or PeerJ, or F1000Research), but I think peer review has benefits that we will have to be careful to replace rather than discard.
Kevin….perhaps my arguments suffer from a touch of editorial flourish, but I am happy that I at least provoked this response from you! 🙂 I certainly agree that peer review is in theory a very useful tool for improving scientific papers. In fact it seems inconceivable that I personally would ever want to “publish” important ideas without subjecting them to some form of peer review, with this current blog post being an exception. Many times this type of peer review for a scientific article can be (and often is) my next door colleague or a friend at another institution. I take their “informal” criticisms very seriously, mostly because I know they have no axe to grind and that their primary interest is helping me to tell my story in the best possible manner.
In contrast to the above, the criticisms I have tended to get through the journal peer review system have been far less useful in part because it is sometime hard for me to discern when their critique is grounded in objective advice and when it is grounded in an intellectual competitor’s own vanity. Having said such things, I would definitely agree (with you) that the majority of the hundreds of journal reviewers who have had “a go” at me throughout my career, have usually made a few helpful criticisms, and when they have done so I have almost always included their suggestions into my published papers and that my paper was improved through this process. I am truly grateful for such advice. Nevertheless, when it comes to addressing secondary (and more often politically motivated) reviewer suggestions, it is typically quite a dance with the editor (and the anonymous group of reviewers) to get a paper published without having to waste too much time; I sense you know this feeling yourself.
Within Cureus we have tried to structure the peer review process so that the role of reviewers is to provide constructive suggestions but leave it at that. Since Cureus authors are free to reject the criticims of reviewers, as long as they make a good faith effort to explain why they made such choices, there is no incentive for reviewers to make authors jump through needless hoops. I sense that this is increasingly also the philosophy of PLoS, a journal I respect a lot. So I wish to thank you for your feedback visa vi my blog post. I also welcome any future advice you might have for our growing journal and the peer review philosophy we try to embrace. Thanks!