|
http://www.digibio.com/archive/SomethingRotten.htm
Something Rotten at the Core of Science?
by David F. Horrobin
Abstract
A recent U.S. Supreme Court decision and an analysis of the peer review
system substantiate complaints about this fundamental aspect of scientific research.
Far from filtering out junk science, peer review may be blocking the flow of innovation
and corrupting public support of science.
The U.S. Supreme Court has recently been wrestling with the issues of the acceptability
and reliability of scientific evidence. In its judgement in the case of Daubert
v. Merrell Dow, the court attempted to set guidelines for U.S. judges to follow
when listening to scientific experts. Whether or not findings had been published
in a peer-reviewed journal provided one important criterion. But in a key caveat,
the court emphasized that peer review might sometimes be flawed, and that therefore
this criterion was not unequivocal evidence of validity or otherwise. A recent
analysis of peer review adds to this controversy by identifying an alarming lack
of correlation between reviewers' recommendations.
The Supreme Court questioned the authority of peer review.
Many scientists and lawyers are unhappy about the admission by the top legal
authority in the United States that peer review might in some circumstances
be flawed [1]. David Goodstein, writing in the Guide to the Federal Rules of
Evidence - one of whose functions is to interpret the judgement in the case
of Daubert - states that "Peer review is one of the sacred pillars of the
scientific edifice" [2]. In public, at least, almost all scientists would
agree. Those who disagree are almost always dismissed in pejorative terms
such as "maverick," "failure," and "driven by bitterness."
Peer review is central to the organization of modern science. The peer-review
process for submitted manuscripts is a crucial determinant of what sees the
light of day in a particular journal. Fortunately, it is less effective in blocking
publication completely; there are so many journals that most even modestly competent
studies will be published provided that the authors are determined enough. The
publication might not be in a prestigious journal, but at least it will get
into print. However, peer review is also the process that controls access to
funding, and here the situation becomes much more serious. There might often
be only two or three realistic sources of funding for a project, and the networks
of reviewers for these sources are often interacting and interlocking. Failure
to pass the peer-review process might well mean that a project is never funded.
Science bases its presumed authority in the world on the reliability and objectivity
of the evidence that is produced. If the pronouncements of science are to be
greeted with public confidence - and there is plenty of evidence to suggest
that such confidence is low and eroding - it should be able to demonstrate that
peer review, "one of the sacred pillars of the scientific edifice,"
is a process that has been validated objectively as a reliable process for putting
a stamp of approval on work that has been done. Peer review should also have
been validated as a reliable method for making appropriate choices as to what
work should be done. Yet when one looks for that evidence it is simply not there.
Why not apply scientific methods to the peer review process?
For 30 years or so, I and others have been pointing out the fallibility of peer
review and have been calling for much more openness and objective evaluation
of its procedures [3-5]. For the most part, the scientific establishment, its
journals, and its grant-giving bodies have resisted such open evaluation. They
fail to understand that if a process that is as central to the scientific endeavor
as peer review has no validated experimental base, and if it consistently refuses
open scrutiny, it is not surprising that the public is increasingly skeptical
about the agenda and the conclusions of science.
Largely because of this antagonism to openness and evaluation, there is a great
lack of good evidence either way concerning the objectivity and validity of
peer review. What evidence there is does not give confidence but is open to
many criticisms. Now, Peter Rothwell and Christopher Martyn have thrown a
bombshell [6]. Their conclusions are measured and cautious, but there is
little doubt that they have provided solid evidence of something truly rotten
at the core of science.
Forget the reviewers. Just flip a coin.
Rothwell and Martyn performed a detailed evaluation of the reviews of papers
submitted to two neuroscience journals. Each journal normally sent papers out
to two reviewers. Reviews of abstracts and oral presentations sent to two neuroscience
meetings were also evaluated. One meeting sent its abstracts to 16 reviewers
and the other to 14 reviewers, which provides a good opportunity for statistical
evaluation. Rothwell and Martyn analyzed the correlations among reviewers' recommendations
by analysis of variance. Their report should be read in full; however, the conclusions
are alarmingly clear. For one journal, the relationships among the reviewers'
opinions were no better than that obtained by chance. For the other journal,
the relationship was only fractionally better. For the meeting abstracts, the
content of the abstract accounted for only about 10 to 20 percent of the variance
in opinion of referees, and other factors accounted for 80 to 90 percent of
the variance.
These appalling figures will not be surprising to critics of peer review, but
they give solid substance to what these critics have been saying. The core system
by which the scientific community allots prestige (in terms of oral presentations
at major meetings and publication in major journals) and funding is a non-validated
charade whose processes generate results little better than does chance.
Given the fact that most reviewers are likely to be mainstream and broadly supportive
of the existing organization of the scientific enterprise, it would not be surprising
if the likelihood of support for truly innovative research was considerably
less than that provided by chance.
Objective evaluation of grant proposals is a high priority.
Scientists frequently become very angry about the public's rejection of the
conclusions of the scientific process. However, the Rothwell and Martyn findings,
coming on top of so much other evidence, suggest that the public might be right
in groping its way to a conclusion that there is something rotten in the state
of science. Public support can only erode further if science does not put its
house in order and begin a real attempt to develop validated processes for the
distribution of publication rights, credit for completed work, and funds for
new work. Funding is the most important issue that most urgently requires opening
up to rigorous research and objective evaluation.
What relevance does this have for pharmacology and pharmaceuticals? Despite
enormous amounts of hype and optimistic puffery, pharmaceutical research is
actually failing [7]. The annual number of new chemical entities submitted for
approval is steadily falling in spite of the enthusiasm for techniques such
as combinatorial chemistry, high-throughput screening, and pharmacogenomics.
The drive to merge pharmaceutical companies is driven by failure, and not by
success.
The peer review process may be stifling innovation.
Could the peer-review processes in both academia and industry have destroyed
rather than promoted innovation? In my own field of psychopharmacology, could
it be that peer review has ensured that in depression and schizophrenia, we
are still largely pursuing themes that were initiated in the 1950s? Could peer
review explain the fact that in both diseases the efficacy of modern drugs is
no better than those compounds developed in 1950? Even in terms of side-effects,
where the differences between old and new drugs are much hyped, modern research
has failed substantially. Is it really a success that 27 of every 100 patients
taking the selective 5-HT reuptake inhibitors stop treatment within six weeks
compared with the 30 of every 100 who take a 1950s tricyclic antidepressant
compound? The Rothwell-Martyn bombshell is a wake-up call to the cozy establishments
who run science. If science is to have any credibility - and also if it is to
be successful - the peer-review process must be put on a much sounder and properly
validated basis or scrapped altogether.
David F. Horrobin, a longtime critic of anonymous peer review. heads Laxdale
Ltd., which develops novel treatments for psychiatric disorders. In 1972 he
founded Medical
Hypotheses, the only journal fully devoted to discussion of ideas in medicine.
References
1. Daubert
v. Merrel Dow Pharmaceuticals 509 U.S. 579 (1993), 509, 579.
2. Goodstein, D. 2000. How Science Works. In U.S. Federal Judiciary
Reference Manual on Evidence, pp. 66–72.
3. Horrobin, D.F. 1990. The philosophical basis of peer review and the suppression
of innovation. J. Am. Med. Assoc. 263:1438–1441.
4. Horrobin, D.F. 1996. Peer review of grant applications: A harbinger for
mediocrity in clinical research? Lancet 348:1293-1295.
5. Horrobin, D.F. 1981-1982. Peer review: Is the good the enemy of the best?
J. Res. Commun. Stud. 3:327–334.
6. Rothwell, P.M. and Martyn, C.N. 2000. Reproducibility
of peer review in clinical neuroscience: Is agreement between reviewers any
greater than would be expected by chance alone? Brain 123:1964–1969.
7. Horrobin, D.F. 2000. Innovation in the pharmaceutical industry. J. R.
Soc. Med. 93:341–345.
Llinks
International
Congress on Biomedical Peer Review and Scientific Publication - articles
and abstracts from the third congress, held in 1997. The fourth congress will
be held in September 2001.
Peer-Review Practices
at EPA - a section of the 2000 NAS report Strengthening Science at the
U.S. Environmental Protection Agency: Research-Management and Peer-Review Practices,
which discusses the strengths and limitations of the process.
Can Peer Review Help
Resolve Natural Resource Conflicts? - suggests that a modified form
of peer review could be useful in policy-related decisions.
Evidence and Expert Testimony
- includes many online references for scientific evidence.
Peer
Review Articles - an annotated bibliography covering scientific peer
review and its relevance to judicial proceedings.
Related HMS Beagle Articles:
Top Ten Reasons
Against Peer Review and Top
Ten Reasons For Peer Review - arguments both humorous and serious.
Anatomy of a
Rejection - strategies for improving the outcome of the peer review
process.
[All emphasis added]
Source
TOP
| 
