Does Peer Review = Better Science?

It is widely assumed that the peer review process improves the article that is eventually published. This concept is taken largely as an article of faith except for one miniscule question posed to a range of published international academics as part of a large survey in 2009; in this survey, 91% of respondents, none of whom were medical professionals, agreed with the idea that the revisions made to their article as a result of the conventional peer review process improved the end product. Of note, most of this subjective sense of manuscript improvement was felt by authors to be within the discussion section. The response to this one question represents the only evidence of which I am aware that demonstrates peer review resulting in better published papers.

Obviously one cannot simply dismiss the findings in the above study out of hand. However, it seems to be worth asking whether or not peer review has the same advantages outlined in the above survey when considered in the rich world of observational studies that describes so much of medical publishing. Moreover, is an improvement in a article’s discussion section significant enough to warrant all the added time and effort on the part of the very busy physician authors who write most medical articles and who represent some of the busiest individuals on our planet?

As an editor of a journal myself (Cureus) I cannot help but ask these questions. I also pose this line of inquiry now, however, due to a recent discussion with a colleague who is a very senior and internationally respected neurosurgeon. He is about to publish an article in a well-known neurosurgery journal having now jumped through the usual reviewer hoops requiring several revisions and months of effort. I asked this colleague whether his article was any better after all this effort. His simple response: “NO!” In fact, in my own academic career, which involves more than 150 peer-reviewed articles, I find it difficult to recall a single instance where I felt the final published article was meaningfully improved through the peer review process.

Oh, I jumped through plenty of hoops, like routinely needing to add additional references so that my paper would cite the reviewer’s own publications, but sadly I have found far too much of the peer review process to be merely an exercise in power and vanity. With so many diseases so poorly treated, and so much clinical research that needs to (or could) be published, is all this inefficiency justifiable? It is for this very reason that we at Cureus emphasize a much more efficient post-publication “peer review” process, or as we term it, SIQ (Scholarly Impact Quotient). I would very much like to hear from any readers of this blog who have published in conventional peer-reviewed journals – did the review process improve the ultimate published article?

Finding a Doctor

Finding the right doctor is a challenge for everyone. Believe it or not, even physicians are challenged.

I love to ask all types of patients and referring doctors how they go about finding a good doctor. After a confused stare and a lot of stammering I get a series of answers. In the case of physicians making a referral it generally boils down to the old tried and true three “A’s”: “Available,” “Affable” and “Ability.” I am embarrassed to acknowledge that this little list describes the thoughts of referring doctors in DESCENDING order of importance! Although “ability” is often judged through the lens of medical education and any prior firsthand results that were witnessed, many (most?) times the final choice is dictated by vague rumors or even gossip. The only exception to this practice is when referring physicians have access to, and opt to actually read, outcome studies published by the potential specialist in question.

Ultimately, if there is a ground truth in the referral business, as exemplified when physicians themselves need a great doctor, the decision is based on peer-reviewed publications. It’s clear that subjecting one’s clinical outcomes to the peer review process remains society’s single best tool for revealing physician competency, and as medicine becomes more specialized, this is ever truer.

How do patients go about this same decision when trying to locate the right specialist for them? Sometimes the choice has, for better or worse, been removed by insurance plan networks or geographical constraints. However, when true life or death decision making is involved, there is oftentimes some flexibility within the system, and in such cases patients are much more determined to find excellence; if the risk of dying or ending up disabled at the hands of one doctor is 2-3X greater than another, only the most desperate or unsophisticated patients would ignore such knowledge. This notion is nice in theory, but where can a patient access such information? When it comes to finding the best doctor, the typical resources that patients can use to make their decision are little better than throwing darts at a dartboard!

Yes, patients can check to see if their physicians have been sued in the malpractice lottery or if the physician has complied with some mindless government-collected quality metrics. Yes, they can rely on rumors, innuendo, ZocDoc, Angie’s list, glossy advertising pages in airline magazines etc., however, do any of these measures reflect the type of competency that truly matters when selecting a doctor, the consequences of which involve life and death? Not from my perspective. In my opinion, the single best tool for assessing a specific doctor’s ability continues to be the same resource used by smart referring physicians: published, peer-reviewed outcome studies.

Historically, patients have not had access to the peer-reviewed medical literature. Fixing this situation is one of the primary rationales for Cureus’ existence. In the modern world of internet connectivity, I would think such validated health information should be a virtual human right. By the same token, I think the onus should be on ALL physicians to contribute to the medical literature, and in doing so, demonstrate their competency within a peer-reviewed environment. If you really are a good specialist, then prove it in some tangible way! Billboards, magazine ads, fancy brochures or websites, broadcast advertising and sundry PR campaigns are not evidence of your competence.

Once upon a time it was not easy to publish in the medical literature, but Cureus is doing its best to remove publication barriers to the greatest extent possible. Therefore, it is time that ALL physicians, (yes, including you guys in private practice), prove some of your skills through a validated process like publishing in journals. Going forward it will be my provocative advice to all patients in need of complex healthcare to trust no physician who has not demonstrated some competency within the peer-reviewed medical literature!! In turn I urge all physicians to use Cureus to demonstrate your hard-earned knowledge to the world, including the lessons imparted to you through patient complications, which will be the topic of a future blog by me.

The VA’s Revelation: No Surprise to the Physicians in the Trenches

In a classic movie line from Casablanca, the anything but naive Captain Renault proclaims “I’m shocked, shocked to find that gambling is going on in here”, shortly before being handed a wad of money (his winnings) by a casino worker.

And so when I hear politicians proclaim to be “shocked” about the atrocious treatment of veterans within the Veterans Affairs hospital system, my only response is that I am more than shocked that they are shocked! The type of care that has engendered such public and congressional outrage was the norm throughout my 7-year tour of duty as a staff neurosurgeon almost 2 decades ago inside a flagship VA hospital. With rare exception (spinal cord injury, rehab for brain trauma), patient care inside the VA system I knew, was totally second rate with customer service on par with the California Department of Vehicles. As a young and newly minted neurosurgeon, I yearned to practice the kind of high quality care I delivered as a resident, but the opposing bureaucratic forces within the VA system were omniscient. Throughout so much of my VA experience I felt like a sordid part of an uncaring bureaucracy and wanted more than anything else to move onto a better job, which in time is what happened. One particular event more than any now shapes my perspective of the VA.

Aspiring to be a young Turk neurosurgeon delivering state-of-the-art care for malignant brain tumors, I implemented, after a lot of sweat and tears, an immune therapy program inside my VA hospital, which at the time was quite an avant-garde approach to this dreadful disease. Despite some interesting preliminary results in a Phase 1 study, a few patients had problems with brain swelling. However, the VA hospital I was in had no equipment for monitoring intracranial pressure (ICP). To remedy this shortcoming I filed the appropriate paperwork to request an ICP monitoring device as part of the capital budget process. Despite a considerable effort my plea fell on deaf ears.

Months later, within a day of the government’s fiscal year being closed, I got a frantic call from the chief of staff’s office informing me that there were unspent funds in the VA system and I had a few hours to get my request for an ICP monitor into the system….which of course I did. But to no avail, I did not get my device and I was forced to manage my patients much like a third world country neurosurgeon. The following year I again submitted my request through the capital budgeting process, and low and behold, when that failed again, I got a second panicked call in the whee hours of the fiscal year suggesting yet another hurried opportunity to procure the object of my desire before the VA funds disappeared into some fiscal black hole. Yet it was Deja vu all over again and the funds were never to come my way. Believe it or not this cycle repeated itself a third year, and soon I would be yet another three-time VA loser in the budget cycle.

Not long after this farce had ended, I found deliverance at the university mother ship. Now free of my VA purgatory, I felt only eternal gratitude. About a year later, while in the middle of an operation, my pager went off and I instructed the circulating nurse to answer it. With the phone put to my ear, I heard a voice on the line tell me he was the manager of the loading dock of the VA hospital and he had just received a shipment for me: a Camino ICP monitor. Almost 10 years later I do wonder if my $30,000 unused monitor isn’t still sitting somewhere on that loading dock…..along with a long line of neglected military veterans.

Yes it is true that the recent “shocking” revelations about the VA system are almost 2 decades removed from my personal experience. As a simple physician in the trenches, I cannot say for sure how best to care for today’s military veterans. However my experience suggests that a radical decentralization might improve both morale and the quality of care.

The Gross Inefficiencies & Politics of Peer Review

A junior colleague described to me a review he was recently asked to perform for one of the leading journals in neurosurgery. After a careful review he reported that the paper “sucked” and he recommended that it not be published. A few months later a revised paper was sent back to him, despite his rejection, leading him to again recommend that the paper be rejected. The next communication was a letter from the editor-in-chief asking my colleague to write a commentary of the paper in question which was to be published alongside the now twice-rejected article. Eschewing a discussion of whether this paper merited ultimate publication or not, I do question the entire peer-review charade that this vignette exposes. The reality is that too often reviewers have little say in whether a paper gets published or not. Instead journal politics and the discretion of the editor-in-chief is transcendent throughout the peer-review process. My lament is that if the editor-in-chief is intent on publishing a paper from the start, why not say so and at least ensure that the process is efficient; this is of course the Cureus process of peer review!

What the above story calls into question is just what exactly constitutes peer review? What is its function? The stalwart champions of peer review argue that it alone uniquely enforces scientific quality. The “acceptance” of one’s scientific work by potential critics and academic rivals, represents the highest measure of personal and scientific integrity. It is hard to argue with such a high-minded proposition, but is it grounded in reality?

In my own, and many of my colleague’s considerable experience, the realities of peer review are nothing like the above idealistic notions bandied about by proponents. Gross inefficiency is often embraced in the name of scientific integrity by rivals who are in no rush to see the accomplishments of scholarly competitors lauded. Many reviews are little more than cursory reads by time-harried reviewers resulting in a thumbs up or down. Ironically when all is said and done this may not be the worst outcome!

Too frequently reviews become meaningless intellectual battles, which at their core are little more than exercises in power and ego. These many skirmishes are often expressed in the to-and-fro volleys between reviewers and authors centering around secondary or even tertiary measures of paper quality. Time-wasting reviewer obsessions with incautious conclusions, inadequate words of self-criticism (at times almost Marxist in nature), failure to reference important (reviewer?) publications, by the letter ethical processes, inconsequential statistical measures, etc. often mask the importance, and strip away the beauty, of science. Given human nature and a subjective system that deliberately selects reviewers from the pool of intellectual competitors, is it not too surprising that baser self-interested emotions too often trump fairness, or at least efficiency? There has to be a better way! Because we at Cureus believe there is, we invite you to learn more about our peer review process.

Time Magazine Tackles Controversial Surgery for Addiction

Cur&#275us published a study from China on a controversial surgery for opiate addiction back in June — the scientific paper has just now begun to spark intense debate in the scientific community after drawing attention in a November English journal publication.

Time Magazine Neuroscience Journalist, Maia Szalavitz has written an insightful piece highlighting the study on Nucleus Accumbens Surgery for Addiction — pointing out the risks versus the push for innovation.

“The surgery is actually performed while patients are awake in order to minimize the chances of destroying regions necessary for sensation, consciousness or movement. Surgeons use heat to kill cells in small sections of both sides of the brain’s nucleus accumbens.” Maia Szalavitz wrote for Time.

John Adler, MD, Professor of Neurosurgery, Stanford University and Editor-in-Chief at Cur&#275us spoke to Time Magazine about the controversy surrounding this risky surgery.

Even though Dr. Adler is not endorsing the procedure, he is convinced the surgery can “provide valuable information about how the nucleus accumbens works, and how best to attempt to manipulate it.”

“I do think it’s worth learning from,” he says. “As far as I’m concerned, ablation of the nucleus accumbens makes no sense for anyone. There’s a very high complication rate. [But] reporting it doesn’t mean endorsing it. While we should have legitimate ethical concerns about anything like this, it is a bigger travesty to put our heads in the sand and not be willing to publish it,” Dr. Adler told Maia Szalavitz.

It’s important to note that Dr. Adler collaborated with Chinese researchers on their publication and he is listed as a co-author. When the study was originally published in the free Cur&#275us journal library back in June, Dr. Adler spoke to the value of the research for neurosurgeons.

“Although this procedure has been used in 1000s of patients in China yet never reported in English until now, other scientific journals and their reviewing processes made it exceedingly difficult to publish in a traditional peer reviewed journal.” Dr. Adler wrote.

“I expect many physicians, especially neurosurgeons, involved in treating behavioral diseases or addiction, to be CURIOUS about the findings contained within this paper!” he added.

In a passionate post entitled “Forbidden Science – Addressing Behavior With Surgery”, Dr. Adler called out the ongoing challenge of publishing such “taboo clinical studies” in peer reviewed medical journals.

“The team of Chinese researchers who conducted this study have personally reported to me that despite the truly ground breaking and novel nature of the experience they report, their work has more than once been rejected by major English medical journals in the past.” Dr. Adler wrote.

You can read the entire Time Magazine article yourself and then jump headlong into the controversy surrounding this not so new bit of “forbidden science”.

Should the process of publishing a paper trump the power of scientific ideas within?

Henry_Jacob_Bigelow_c1854 — Henry Jacob Bigelow

Last week the New England Journal of Medicine announced that a survey of readers selected the 1842 publication of a paper by Henry Jacob Bigelow, reporting the first use of anesthesia, as the single most important paper in the 200 year old history of the journal.

Given the future implications of what was being reported in this paper, especially as we now look back over the medical landscape 150 years later, I cannot disagree with the selection. However, what I do find more than a little bit ironic is that such a paper, if submitted today, would stand almost no chance of being published in the NEJM, or virtually any so-called high impact journal.

As the Bigelow paper wended through the review process, this in-hindsight completely transformational paper, would be rightly criticized (and surely killed) as being little more than anecdote.

There is certainly no randomized blinded trial with rigorous statistical measures that ensures the quality of findings live up to the highest “scientific standards”. Moreover, lacking an IRB stamp of approval would certainly invalidate the paper for publication right out of the gate.

So I ask, have the standards of the NEJM and nearly all modern journals become so obtuse and elevated that the findings they now report are irrelevant to truly novel clinical innovation?

Also last week, I was advised by one of the journals for which I have long served as an editor, that future observational studies, if they are to even be considered for publication, must now assiduously adhere to STROBE guidelines.

Such STROBE guidelines represent the consensus of a 2004 big funded workshop, i.e. self appointed committee, composed of editors and sundry luminary academics who, presumably with the best of intentions, felt compelled to improve the reporting of observational studies.

But what started as well meaning “guidelines” have (perhaps not surprisingly) now morphed into a compulsory directive enforced by the editor in chief. Will the quality of the journal articles now being published under the STROBE guidelines truly benefit from this new set of regulations? Surely the editor in chief and politically-connected self-appointed STROBE experts would argue yes, but notably these sacrosanct “guidelines” have themselves never been subjected to any empirical test.

What I am quite confident about is that the journal in question has put up yet another barrier to authors reporting potentially interesting, and even ground breaking, clinical studies. As a result is it possible that that today’s equivalent to the good Dr. Bigelow’s reporting of anesthesia will go undiscovered because of the sheer hassle of publishing in the peer reviewed literature?

Is it possible that STROBE processes might drive some of today’s clinical innovators to just throw up his or her hands and say no thank you to reporting their findings? This leads to the bigger question, should the process of publishing a paper trump the power of scientific ideas within?

Readers of Cur&#275us are well aware of our Journal’s bias in this regard!

A New Age Of Peer Reviewed Scientific Journals

Dr. John Adler — We can do much better by authors, reviewers and certainly patients. – John Adler MD

John Adler, MD, Professor of Neurosurgery, Stanford University; Editor-in-Chief, Cur&#275us, has published a new editorial on the variety of issues associated with the centuries-old industry of peer reviewed scientific publication.

In today’s internet age, can the status quo peer review system provide value to our society – or is there a better way for “establishing scientific validity” and dissemination of knowledge?

Dr. Adler highlights some of the serious hurdles with current traditional peer review (i.e. fraud, and reviewer bias) and proposes to re-imagine peer review to become more suitable to our internet age. In a digital platform, the space limits within paper journals are gone. Without having to ration the space, there is no need to “kill” papers and artificially limit the number of papers published in a given month.

And to assess the quality of these papers Adler proposes crowd sourcing. To tap into the collective intelligence via Cur&#275us’ Scholarly Impact Quotient (SIQ). The SIQ is an “evolving, yet enduring reflection of a paper’s true scientific impact.”

Dr. Adler founded Cur&#275us “to address the challenges I have observed first-hand as an editor of numerous journals and an academic physician who has published and reviewed for years. We can do much better by authors, reviewers and certainly patients. This is the mission of Cur&#275us.”

Source: A new age of peer reviewed scientific journals – John R. Adler Jr

The Cureus Blog

Medical publishing powered by community

Tag: John R. Adler MD