The false idea that only the “top journals” publish the important stuff

From the former Editor-in-Chief of BMJ:

The naïve concept that the “top journals” publish the important stuff and the lesser journals the unimportant is simply false. People who do systematic reviews know this well. Anybody reading only the “top journals” receives a distorted view of the world. Unfortunately many people, including most journalists, do pay most attention to the “top journals.”

References:

Richard Smith: Scrap peer review and beware of “top journals”. BMJ Blogs.

Google ranks the top 100 journals and NEJM is no more the "top dog" according to G Scholar Metrics for Publications http://goo.gl/unTjs

Comments from Google Buzz (March 22, 2010):

Heidi Allen - Love the fact that the BMJ have put out so bold a statement

Neil Mehta - Very provocative. Agree that Peer review is mostly a joke and a lottery. Unfortunately we still live in a world of publish or perish. We need to have something like a YouTube of papers. As more people read these and rate them, the cream will rise to the top. The raters should also be rated and that should also count towards academic promotion.

Lakshman Swamy - Neil-- I agree, but it should be a closed system. Anyone can read, but not anyone can rate-- have to have credentials of some sort. Can't risk creating another place for opinions to become stronger than data.

Ves Dimov, M.D. - On a more basic level, this is also a simple appeal to all physicians to read more than the top 5 journals. Wait... Most physician don't even read the top 5 journals.. :)

Social networks help in some way though - I tend to have a closer look at articles linked by the people I follow.

Vamsi Balakrishnan - @Neil Mehta A friend and myself actually shared that same idea about science, about publishing and its current state. We had a grandiose goal of starting a publishing site that would be more about sharing ideas than anything else (www.sciencetrader.com/).

When I think about publishing in famous journals being important, I think about the famous "cloner" who published in Nature (http://en.wikipedia.org/wiki/Hwang_Woo-Suk); if peer-review was successful, I doubt that he'd have been published his fabricated data. But, once you're successful and thus renowned, no one would have the courage to question you.

Kind of a similar situation is found with Einstein back in the day. When he was working on TOE (theory of everything; a unified grand scheme for how the universe works) he denied the possibility that quantum mechanics could rule the microcosm simply because of his belief that regardless of the evidence, "God does not play dice with the universe" (or something close to that). The macrocosm followed his rules exactly as he predicted, so the microcosm should do the same. No one questioned him since...it's Einstein. He's probably right. (He wasn't); I think Dr. Feynman said in his book (Surely You're Joking Dr. Feynman) it took another ~10 years before Einstein admitted he might be wrong...

Anyway, I diverge.

A last example that's more healthcare-related would be the scientist who developed valproic acid as a treatment for seizures. This scientist, Eymarde, made 30-40 drugs...every drug he made was GOLD, meaning they ALL worked. No one had as much success as he did in creating seizure medications. He could have said anything he wanted about seizures medications since he was the expert, and it was taken as the truth. Unlike most scientists out there though, he knew he was willing to admit that he's not perfect and that no one could make this many drugs work so well. So, he developed controlled studies...testing every component of the medication, only to find that the vector he used to dissolve his "drugs" in was the actual active component with anti-convulsive properties. A modern scientist...would not do anything. Publish or perish. If you just retracted every paper you've written for the last x amount of years...it'd be pretty hard to get future funding, even if you were being honest. This Eymarde did however respect his trade and his peers, so he retracted every one of his papers. After this bit of honest science, he was forgotten to the world and ridiculed.

In a completely circuitous way through three different example, I'm trying to say that I agree with you that peer review, and generally the scientific process now due to peer review is broken. Changes are needed.

Neil Mehta - Vamsi,
thanks for those examples. I was not familiar with the Einstein or the Eymarde examples - I want to publish my case series that suggests famous scientists with last names starting with E tend to make errors that they own up to several years later. Do you think that I should submit to top tier journals? Suggest Vamsi for peer review? Just kidding. Enjoyed reading your comments.
Look forward to more conversations

Vamsi Balakrishnan - Oh, wow, I didn't even notice the common endings. I went searching for a third "E" named person...but couldn't find one that is...on par with the Woo-Suk one.

If I write a book, it's going to be a collection of stories about famous scientists and the things that they...endured and the follies they've made.

It will be entitles, "Famous Scientists: The Things They Endured, and the Follies They've Made."

Aidan Finley - http://www.nature.com/nature/history/timeline_1950s.html The Krebs Cycle paper was rejected by Nature for not being impactful.

http://www2.uah.es/jmc/ai56.pdf (article reviewing other major rejected discoveries, including the MRI). Nature editorial: http://www2.uah.es/jmc/

Ves Dimov, M.D. - "The Krebs Cycle paper was rejected by Nature for not being impactful."

Isn't this just amazing? Reality is truly stranger than fiction.... :)

Aidan Finley - To be fair, I think everyone recognizes it was a mistake.

Related reading:

Nature rejected/declined Krebs cycle paper in 1937 - there were too many submissions - but few remember them now... http://goo.gl/mcxPd

The grossest failures of peer review (PDF) http://goo.gl/3ovkn

Image source: 2009 cover of the journal BMJ, low resolution, fair use.