Scholarly Roulette: Impact Factor & Scientific Quality

A friend and colleague referred me to a recent publication that represents a major indictment of the peer-reviewed journal cartel.

This article summarizes a blog published by what is arguably a Cureus competitor, the open access-publishing house, Frontiers, which in this study could find no correlation between rates of rejection and journal impact factor.

Just for argument’s sake, let’s assume that impact factor does correlate with scientific quality. So if this study is correct, it means by extension that the rate of rejection also doesn’t correlate with quality. In turn this means that the standard peer review process (which totally dominates the journal industry) has no bearing on article quality. In other words, getting published, or not, appears to be a random event.

Accepting this fact (parenthetically one which I already subscribe to) means that despite all the process and rituals of scientific journal publishing, and the huge importance of this matter to society and academic tenure decisions, there is currently no objective index for assessing research article quality. Of course if one believes that impact factor is not a measure of quality, then the above random publication process is even more random than I implied in my argument! Based on this observation, I would like to sarcastically suggest we should rename this current process “scholarly roulette.”

Clearly there are huge shortcomings to the Frontiers study, such as selection bias and the non-medical nature of many scholarly journals. However, to a huge research community that has invested so much of their lives into the existing peer review process these type of findings must be dispiriting. Nevertheless, this current set of observations does resonate with our philosophy at Cureus, where we subscribe to the belief that as long as the medical science within is credible, there is no reason not to publish an article. In other words, we believe that we are, on average, no better equipped to discern quality than any other journal.

A big part of our confidence stems from Cureus’ unique post-publication SIQ scoring system. Although we are no smarter in finding scientific quality up front, we do believe that it is quite possible for the medical community-at-large to find quality over time, post publication. Nevertheless, to make this a reality, every reader has a role to play, which of course includes you!

The Anecdote is the Antidote for What Ails Modern Medical Science

Today I read with great interest an article in Vox.com entitled: “Fleming’s discovery of penicillin couldn’t get published today. That’s a huge problem.” As a long time academician and Co-Editor-in-Chief of Cureus, I could not agree more! Not only would Fleming’s discovery go unreported, but Jenner’s discovery of the first small pox vaccine or Connecticut dentist Crawford Long’s discovery of ether anesthesia would also be dismissed as trivial curiosities and anecdotes.

In a rapidly changing world of medical science, the process of peer-reviewed publishing has never been more important for validating medical knowledge, yet at the same time the arcane processes of peer-reviewed journals have never been more at odds with potential societal benefits. The altruism that is supposed to drive the publication of scientific research has been almost entirely co-opted by the peculiar needs of academic promotion and tenure, as well as the pecuniary demands of the scholarly publishing industry; the public good of medical knowledge has been reduced to a mere after-thought by both academia and the publishing industry. These two interest groups only seek seemingly important “big science,” and actually decry any role for mere clinical observation, which Cureus has in the past and the Vox authors now in the present both refer to, in a somewhat ironic, self-deprecating manner, as “small science.”

Despite the above dire (my word) situation in medical publishing, just maybe the proverbial pendulum is about to swing the other way. For example, the conventional wisdom is that we are entering an era of “personalized medicine,” a concept which at its core recognizes the unique genomic, proteomic, metabolomics and demographic, etc. profile of every patient. Embodied within this concept is the understanding that each of us will as patients undergo a unique clinical experience, not necessarily predicted by a priori knowledge, and that such observations will need to be documented if we are to build new scientific understandings. Assuming that to be true, the future of medical publishing may well be found in observational “small science.” If this is true, the important medical publishing of the future may be about to shift from “the few” elites reporting big science, to the power of the many, within the trenches of medicine, collectively reporting their humble clinical observations? For the record, Cureus aspires to lead this transition into the new world of small, yet very powerful, medical science.

Intellectual Fascism

I have been Co-Editor-in-Chief of Cureus for about three years and in this time I’ve learned a lot about how peer-reviewed journals function. It’s often said that in water polo the real game happens beneath the surface of the water. Similarly with journals, sometimes the serious action goes on behind the scenes. From its inception, Cureus was designed to minimize the role of politics in scientific publishing by way of its post-publication SIQ scoring process. Despite these ambitions, politics have occasionally crept into our efforts to publish great medical science. So it was with one recent article, and boy did the Cureus editorial staff learn a lot through this experience!

The article in question was written by several very accomplished clinical neuroscientists and involved a complex intersection of multiple scientific fields. Despite being evaluated by three reviewers, a clear error was noted in the published article by a reader; the error was of a political nature and not scientific in the least, but still an unambiguous error. An erratum was being prepared when a big hullabaloo broke loose in social media. Two individuals, whose specialty overlapped the erroneous article, attacked the article for its political misstatement, and by extension, Cureus’ journalistic integrity for missing this error during our pre-publication review process.

I immediately invited these critics to set the record straight via our liberal comment and scoring processes, but in a series of personal (and necessarily confidential) emails, the critics refused, insisting on remaining anonymous. Over the next several days they recruited a chorus of similarly-minded colleagues who insisted that the article in question represented serious scientific misconduct and demanded it be retracted… period! In light of these demands, Cureus, by virtue of its status as a peer-reviewed journal, was obligated to investigate under ICJME (International Committee of Medical Journal Editors) guidelines.

I personally oversaw the investigation, which started by recruiting seven truly world-class domain experts, who after reading both the original and the proposed corrected manuscripts, were to advise me; I deliberately included a couple of researchers suggested by the critics of the article. In parallel, I stumbled upon the existence of a listserv community of likeminded researchers including the two critics, whose major modus operandi is to fiercely act en-mass, hyena-like, oftentimes via social media, when certain partisan political issues arise, such as the article Cureus had unwittingly published.

If ever I witnessed intellectual fascism, this was it; the only thing missing was a goose-stepping mustached man. However, this was also to be a moment of truth for our young journal. Pending the advice of the seven domain experts, would Cureus stand up for open scientific discourse? Or would we join the ranks of cowering researchers?

After almost one month of analysis by the aforementioned unimpeachable panel of experts, some of whom are at the very pinnacle of their respective fields, it was determined that the Cureus article had erred badly (yet seemingly inadvertently) in misstating a political reality. However, the science itself was credible. As a result, an erratum addressing the erroneous facts was published and the original article retracted.

Ultimately this experience reminds me, and by virtue of this blog should remind all readers, that standing up for open and honest scientific discussion, devoid as much as possible from political considerations, is a constant struggle even in our supposedly democratic world. In fact I invite the very critics of the article in question to now publish their own scientific concepts with Cureus, which perhaps might even refute the published paper. The function of journals is not to anoint “scientific truths,” but to provide a forum for scientific truths to be discovered, and refuted.

We at Cureus are especially fortunate to have such a liberal post-publication comment and scoring system. This process, which is available to everyone, is intended to provide a voice for even the most contrarian scientific ideas. The power of the Cureus community-at-large remains a great potential strength. Please don’t be afraid. Step up and use it!

Banished From Cureus: Introducing a New Cureus Editorial Policy

Founded with the belief that far too many credible physician and allied medical specialists are disenfranchised by a publishing system driven largely by money and academic promotion, Cureus has, from its inception, bent over backwards to remove barriers to medical publication. Whenever possible, we have always tried to give each author the benefit of doubt throughout the entire review and publishing cycle. In the process of being so liberally minded, however, Cureus has attracted a handful of prospective authors that seek to take advantage of our generosity. In particular this group of authors has either failed to read closely Cureus’ author instructions or chosen to not follow its unambiguous dictates. This is especially apparent in the areas of copy editing and formatting, for even the simplest little requirements, such as bracketing reference citations.

Why is this handful of authors so sloppy? Maybe having been schooled in the practices of other journals they assume some faceless (yet compensated) copy editing team will clean up their carelessness? Of course other journals will clean up your articles, but it’s going to cost you; for such copy editing services an author must give up either all copyrights or pay many hundreds, if not thousands, of dollars for open access. One might think the generosity of Cureus’ follow-the-rules “self-serve” model for publishing, which on average requires about 30 minutes of extra work, would be appropriately valued by all authors. Sadly this is not always the case.

Therefore, Cureus has hereby implemented a new policy of banning (“blocking”) an article from any chance of being published, regardless of the quality of science, if an author fails (for whatever reason) to twice submit a manuscript that complies with our clearly published author guidelines. Moreover any author that submits two unique articles that fall short in this way will be forever banished from Cureus. There are plenty of other journals that are happy to copyedit sloppy work. Authors who produce such manuscripts are invited to send their articles to those publishers.

In Cureus’ community of trust and mutual respect, there is no room for users who fail to follow the rules. But if you are among the vast majority of users who comply with our guidelines, I promise that you will be amply rewarded for your consideration and cooperation. Thank you.

The Value of “Small Science”

“Let me tell you about an interesting case.”

“I’m having trouble with a patient, can I get some advice?”

“Help, my patient is dying on me and I cannot for the life of me figure out what’s wrong.”

“While caring for this patient, I learned something kind of cool last week.”

Have you or a colleague uttered such things in the past few days, weeks or months? Throughout my own clinical practice in an academic setting these types of utterances happened on a daily basis, if not many times a day, albeit sometimes merely under my own breath.

It is a fact that we physicians, even the smartest among us, still have a lot to learn, and vice versa, have a lot to teach through such experiences. Our clinical practices are sometimes influenced (usually for the better) by prominent, well-funded, randomized clinical trials, but more often it is the humble practical knowledge learned from one another that separates the satisfactory from the great physician. In my chosen specialty of neurosurgery, I have observed that there is not two or three bits of knowledge that make for a great operation. Instead the best surgeons have a grab bag of literally thousands of largely undocumented tricks (patient selection, choice of instrument, anatomical insights, manual skills, techniques, etc.), which make for their success. Much of this knowledge continues to get acquired the old fashion way – via trial and error in the trenches of medicine. Amazingly, in a world of more than 5,000 medical journals we all too often find ourselves repeating one another’s mistakes and relearning lessons previously learned by others. Why is this?

I believe that the above situation stems in part from a medicine-wide failure to formally acknowledge the true value of practical knowledge, or what we at Cureus like to refer to as “small science.” In many ways this is illustrated by how most medical journals see their mission, especially those with a coveted high impact factor. For example, the Instructions for Authors section in JAMA is almost half the length (in words) of Joseph Conrad’s the Heart of Darkness. Filled with complex guidelines for statistical processes and data reporting, the focus is on academic researchers who themselves are focused on climbing the ranks of academia as much as they are the knowledge at stake. The complexity of such processes, and even the associated financial costs in open access journals, intimidates too many of the busiest practicing physicians who have amazing clinical experience and insights but lack the time and arcane knowledge of contemporary journal publishing processes. As an Editor-in-Chief of Cureus this strikes me as a tragedy; some of the most knowledgeable clinicians have no forum for passing on their hard-won insights.

Our mission at Cureus is to use technology and a new philosophy of post-publication peer review to strike a better balance between process and the more efficient reporting of valuable clinical science. Our goal is to make it easier than ever for busy physicians in the trenches of medical practice to document the important things they learn on a near continual basis. Ultimately if some clinical observation is important enough for an overworked physician to invest time in writing up, we at Cureus are delighted to help with the task.

Measles, the Anti-Vaccine Crowd and the Peer-Reviewed Article Partially to Blame

A big measles outbreak in the US is generating considerable news of late. Much of the blame, rightfully so in my opinion, is centered on parents who have refused to vaccinate their children out of ill-founded paranoia. Most of their anti-vaccine “thinking,” if you can call it that, is grounded in muddled anti-science. However, one of the intellectual pillars of the anti-measles movement is rooted in peer-reviewed science. And not just any science, but none other than the acclaimed high impact journal Lancet. To my way of thinking this is a big part of the problem.

Andrew Wakefield’s much ballyhooed and eventually discredited 1998 Lancet article, concluded that childhood vaccines were a possible cause for autism. Parents of autistic children seized upon this possibility, especially A-list celebrity Jenny McCarthy, invoking the Wakefield article as proof of their fears. Unable to weigh the scientific merits for themselves, the fanatical anti-vaccine public at large ascribed considerable veracity and power to the Wakefield article in large part because of its publication in Lancet; although undoubtedly this article also reinforced a narrative they wanted to believe. Like most of its peers within the “luxury” journal domain, Lancet revels in every opportunity to burnish its reputation for publishing “important,” high quality science and in doing so, laughing all the way to the bank. It is impossible to believe that had the Wakefield article been published in the Burmese Journal of Gastroenterology it would have gotten nearly the same zealous respect it did in Lancet.

Don’t get me wrong. I don’t fault Lancet for publishing a bad paper; such risk is intrinsic to publishing a peer-reviewed journal. However I do take issue with the reputation Lancet seeks to foster around impact factor and by intimating that their peer review process results in a meaningfully more valid scientific publication; the Wakefield article being a perfect case in point of why this thinking is flawed. Any scientist with integrity knows that the proper perspective for approaching all scientific publications involves a substantial dose of skepticism until the findings within have been replicated, oftentimes more than once.

Despite its subsequent retraction, Wakefield anti-vaccine believers refuse to relent, claiming even today that allegations of scientific misconduct are a conspiracy at the highest levels of science. If the Wakefield article had instead been refuted by a democratic barrage of post-publication critique and scoring, ala SIQ, it would be much harder for the anti-vaccine fanatics to keep believing in discredited science. That said, as long as luxury journals are perceived as having a uniquely rigorous peer review system, and therefore uniquely truthful, the public health will be disserved as we are currently witnessing in this measles outbreak. We at Cureus continue to believe that there is a better process via more rigorous post-publication review, as implemented in our SIQ. Give it a chance – you’ll be surprised.

An Introduction to Cureus’ First Student Ambassador

The processes of peer-reviewed medical science have been around a very long time with the New England Journal of Medicine even passing its 200th anniversary last year. It, like many other old journals, has become venerated for the important science reported over the centuries, and in conjunction, there has evolved the aura of scholarly respect and status. In fact the prestige of a journal, perpetuated by medical school tenure processes, and as quantified by Impact Factor, is roughly proportional to the age of the journal. Roughly speaking, old journals are prestigious, while new journals tend not to be; academic medicine tends to be a very tradition-leaning and status-conscientious community! Despite its relative youth, Cureus’ leadership team also includes a number of senior academics, some of whom have five decades of medical publishing under their belts. Our advisor and former editor at JAMA, George Lundberg, is a perfect example; experience always has a lot to teach us all.

Despite medicine’s veneration of history, it cannot be denied that the future belongs to the young, and just possibly, they might choose to evolve peer-reviewed journalism in new directions. In the past decade, youthful exploits have truly transformed human culture and commerce, with Facebook, Google, Uber and AirBnB being only a few cases in point. In many ways the pace of social change seems to be accelerating on the backs of the young. To date, youthful exuberance has not been part of science, however, Cureus is hoping to change that. As we aspire to be a new concept in peer-reviewed medicine, we would like to introduce Cureus’ first medical student ambassador, Paul Windisch, from the University of Munich. Paul’s role at Cureus will be to help introduce our tools for publishing peer-reviewed science to a new generation of physicians and simultaneously provide our team with a deeper insight to the needs and aspirations of younger doctors.

When stepping into this new role, Paul was quick to encounter a little bit of skepticism among fellow students. Their concern was whether or not publishing an article in a newer journal like Cureus, and thereby not publishing in a more conventional journal, might negatively impact the career of an aspiring young academic; many professors have stressed the importance of publishing in the most highly-regarded journals. My counterargument is that by all means it’s good to publish in prestigious journals, but do not let that dissuade one from publishing in Cureus as well. To publish in Lancet or Nature or NEJM is a very time-consuming process with much of the effort being expended throughout the rationing process of responding to multiple rejections, each one eating up precious time that might have gone into publishing yet additional papers. The most successful scholarly careers in medicine tend to combine important papers in highly-regarded journals AND much more numerous peer reviewed articles in lesser-known journals. Quality (or at least the perception of such) is important, but so is the QUANTITY of one’s scholarly work as one seeks to become known and rise through the academic ranks. As the maxim “publish or perish” clearly implies, it is important to publish frequently as one seeks to build an academic reputation. Need I remind you, Cureus makes that process easier than ever. Just maybe when Cureus has enough of its own storied history in the future, the aura of our articles will bring the cachet of academic tradition as well!