FOUR CONJECTURES ON IMPACT FACTORS
Or why impact
factors
correlate with ignorance, lack of originality, obsolescence and fraud.
Followed
by :
-
a
quotation and an
anecdote
-
the
original French
version of the text
-
a link to
the Spanish
version, by Eduardo Mizaji
-
a link to
an analysis
by Antoine Danchin
Jacques Ninio
Laboratoire de
Physique
Statistique, Ecole Normale SupŽrieure,
24 rue Lhomond,
75231 Paris
cedex 05, France
Each
statistical
indicator entails biases which need to be identified and corrected.
However, in
the evaluation of scientific research, popularity tests are used as
substitutes
for quality tests, a practice which penalises our most original
productions.
I criticized in
the past the
use of citation counts in the evaluation of the quality of a scientific
work.
Now the citation count criterion is neglected to the benefit of an even
less
pertinent one, the impact factors (1) of the journals in which a
scientist
publishes his work. What does the impact factor really measure ? I
propose
here four conjectures which stem from my experience in life sciences.
Conjecture
1 : The
impact factor of a journal is directly correlated with the incompetence
of
those who cite it.
When a scientist
writes an
article in his domain of research, but needs for this article to
mention a
result or an idea from another domain, he will first look into the
journals
which are the most accessible to him, thus the generalist journals
which are
received in his laboratoryÕs library. In a field in which the
researcher is not
competent, it is easier for him to quote Nature or Science, than to
perform a
meticulous bibliographic inquiry. My conjecture is that the hierarchy
of impact
factors reflects above all the statistical weight of the quotations
made by the
non-specialists.
Conjecture
2 :
Impact factors are directly correlated with lack of originality.
Impact factors
are computed
on the basis of the citations received by published articles during the
year of
their publication and the following year. An article which is ahead of
its time
and starts being cited ten years after its publication, reduces the
impact
factor of the journal in which it is published. An article in an
emergent field
which is of concern to only a small community of researchers is rarely
quoted.
Thus, it pulls the impact factor downwards. Knowing that,
the editor which is obsessed by the
impact factor of his journal rejects innovating, avant-garde articles.
Once the
subject takes roots, the editor makes up for his past neglect and
publishes as
a big revolution, the first article on the subject which is submitted
to him
emanating from a suitable source (excluding, e.g., laboratories from
Mediterranean Europe).
Conjecture
3 : The
impact factor of a journal is inversely correlated with the longevity
of the
articles it publishes.
When the high
impact factor
journals publish their articles, they succeed in general to fool the
non-specialists. However, after a few years, the really important
contributions
become known through the quotations made by real specialists. Then,
these
founding articles become also quoted by the non-specialists. On this
account,
impact factors and half-lives should be anti-correlated.
Conjecture
4 : The
impact factor of a journal is directly correlated with its rate of
fraudulent
articles.
In high impact
factor
journals, articles are sorted by editors, who essentially determine the
windÕs
direction from their wet finger. Most of the articles are eliminated
before
having even a chance to be sent to the reviewers. Knowing that the
crucial stage
of selection is under the control
of non-specialists, the authors have a tendency to overstate the
significance
of their work, and present it in a dishonest fashion. This system works
to the
benefit of cheaters.
Concluding
remarks.
While scientific
research is
now carried out in the context of a merciless world, it is curious that
science
administrators, which are in their majority under the influence of
financial
and marketing ideologies, judge
our work as though it were unfolding in a different world, governed by
pure
altruistic behaviour : A world in which our worse competitors,
upon
reading a breakthough manuscript which relegates their past and present
work
into obsolescence, will lean down
and recommend the publication of the manuscript with high priority.
Science
administrators would do a good job by studying seriously the
distorsions which
are introduced by the so-called objective criteria of scientific
evaluation.
-------------------------------------
(1) The impact
factor was
launched by Eugene Garfield and his Institute for Scientific
Information (ISI).
The Science Citation Index, invented by Eugene Garfield, is an
amazingly
powerful bibliographic scientific tool. However, the bibliometric
indexes which
are derived from the SCI have disastrous
effects. They reduce science to marketing.
================================
Bonus for the
faithful
readers :
A quotation
From the
concluding section
of my article in Ç Biochimie È [1]
Ç An
article by FaxŽn, Kirsebom and Isaksson [50] played an extremely
important role
in this work. I will stress here the qualities of this article, because
it has
all the attributes which would make it unpublishable to-day in the
so-called
"high impact factor journals". First of all, it was deeply honest.
Instead of making a narrow selection of their experimental
observations, and
using such a selection to promote simple-minded theories that would
have
appealled to a broad audience of researchers from other fields, they
laid down
all the results of their genetic experiments on the table. Next, part
of the
data was in line with current theories, others were not, but the
authors
interpreted their work in an extremely modest way. This attitude would
make it
impossible to "sell" the work as a revolution in the field. Above
all, the work was extremely systematic. It covered,
in a consistent way nonsense suppression, codon
context effects, streptomycin effects, ribosomal mutations, and
anticodon
modifications. All the clues were there, but hard to decipher. After
all, the
ribosome is one of the most complex achievements of molecular
evolution. While
progressing in the comprehension of the ribosomal puzzle, I often
returned to
the work of FaxŽn et al. [50] to determine if at this point their data
became
more intelligible. While the proposals made in this review do not
explain in
detail all their results, they provide at least sufficient leads to
envision
tentative explanations. È
[1] Ninio, J.
(2006)
Multiple stages in codon-anticodon recognition : double-trigger
mechanisms
and geometric constraints. Biochimie 88, 963-992.
[50] FaxŽn, M.,
Kirsebom,
L.A. and Isaksson, L.A. (1988) Is efficiency of suppressor tRNAs
controlled at
the level of ribosomal proofreading in vivo? J. Bacteriol. 170, 3756-3760.
An anecdote
On the
calamitous management
of science in a European country.
A few years ago,
a team
of scientists from Shanghai
University, in the PeopleÕs Republic of China made public their personal ranking of the worldÕs top scientific
institutions. The ranking was based on criteria such as the number of
publications in high impact factor journals or the number of patents
emanating
from the institutions, the number of Nobel awards, etc. As far as I
know, they
did not apply even the most obvious corrective factors, such as (i)
dividing
the output buy the number of employees, (ii) relating the output to the
budget.
Taking such factors into account, one would have had a better idea of
the
efficiency and productivity of the employees.
How do you think
that the
science administrators from this European country reacted ?
Instead of
throwing the Shanghai study into the waste basket, they took it
seriously. In
order to improve the ranking of their countryÕs institutions in the
ShanghaiÕs
list, they decided to regroup the institutions by twos or threes under
a same
name . As long as this was just a game of changing names, the
consequences were
not too deleterious. Unfortunately, the administrators decided that
there
should be a real reorganization of the institutions. So, for the last
two years
a large number of the leading researchers of the country have abandoned
their
scientific work, and are elaborating instead ambitious
Ç federative È projects, which would tie
several institutions under a same banner, and they are attempting to
redefine
hundreds of research projects carried out in the allied institutes, so
as to
make them fit into an administratively coherent framework. By the time
they
will have finished their restructuration projects, no doubt that
another
administration, issued from the latest political elections, will push
forward a
new strategic plan, requiring different modes of reorganization.
What intrigues
me most, in
this state of affair, is why there is no resistance to the
administrationÕs
idiotic pressures ? Apparently, most researchers in the country of
which I
am speaking are coward opportunists, they are convinced that it is
better to
comply superficially with the demands from the administration. They
believe that
once they have complied, and have consequently received funding and
support
from the authorities on the basis of fake projects, they will be able
to carry
out real, innovative scientific projects. In my experience, the
pressure and
the controls are such that there is less and less room these days for
real
scientific projects.
I am also
intrigued by the
fact that the science administrators do not even realize that their
policy is
self-defeating. For if the
Shanghai game is taken seriously in other countries, what will prevent
the
institutions of the other
countries to play the same administrative regrouping game ? What
can
prevent Harvard University and the MIT to be united into a larger
administrative entity, the Harmit university ? Should we then
reply by
uniting Paris, Hamburg and Cambridge into a Ç Campaham È
research
mega-center ? Once you start thinking in these terms, you begin to
see
more clearly the motivations behind the Shanghai list. The authors of
the list
are not absolute cretins, as you might have thought initially. They are working for the glory of their
country. Remember that China has more than a billion inhabitants, which
is more
than three times the whole European community. Thus any ranking of
scientific
output, which is related to global production, and not to productivity
per
employee should, in the long term, turn to the advantage of China or
India.
================================================================
QUATRE
CONJECTURES SUR LES FACTEURS D'IMPACT
Jacques Ninio
Tout
indicateur statistique est affectŽ de biais qu'il convient de repŽrer
et
corriger. Or, en Žvaluation de la recherche, les tests d'audience sont
pris
pour des tests de qualitŽ, ce qui pŽnalise nos travaux les plus
originaux, au bŽnŽfice
de nos concurrents Žtrangers.
Dans
les pages jaunes d'une VRS
d'autrefois, j'avais attirŽ l'attention sur les risques d'utilisation
des
citations dans l'Žvaluation scientifique (avril 1978, pages 26-28).
Aujourd'hui, les citations reues par un auteur sont elles-mmes
dŽlaissŽes au
profit d'un critre encore moins pertinent, celui du Ò facteur
d'impact Ó des revues dans lesquelles il publie. Que mesure ce
facteur ? Je propose ici quatre conjectures, issues de mon
expŽrience dans
les sciences de la vie:
Conjecture
numŽro 1 : le facteur d'impact d'une revue est directement corrŽlŽ
ˆ
l'incompŽtence de ceux qui la citent.
Un
chercheur qui, pour les besoins d'un article de sa spŽcialitŽ, Žprouve
le
besoin de mentionner un rŽsultat ou une idŽe relevant d'une autre
discipline,
ira d'abord puiser dans les revues les plus accessibles, donc les
revues gŽnŽralistes
reues dans son laboratoire. Dans une discipline o il n'est pas
compŽtent, il
lui est plus commode de citer Nature que de faire une enqute bibliographique
minutieuse. Ma conjecture est
que la hiŽrarchie des facteurs d'impact reflte d'abord le poids des
citations
faites par les non-spŽcialistes.
Conjecture
numŽro 2 : le facteur d'impact est directement corrŽlŽ avec
l'absence
d'originalitŽ.
Le
facteur d'impact est calculŽ sur les citations recueillies par les
articles
d'une revue l'annŽe mme de leur publication et l'annŽe suivante. Un
article en
avance sur son temps, qui commence ˆ tre citŽ dix ans plus tard
abaisse donc
le facteur d'impact de la revue. Un article dans un secteur novateur,
qui
n'implique et n'attire l'attention que d'une petite communautŽ de
chercheurs
est, de ce fait, peu citŽ, et tire le facteur d'impact vers le bas.
Sachant
cela, l'Žditeur obsŽdŽ par le facteur d'impact de sa revue refuse les
articles
novateurs ou d'avant-garde. Une fois que le sujet a commencŽ ˆ
diffuser, l'Žditeur
se rattrape en publiant comme une grande nouveautŽ, le premier article
qui lui
est soumis en provenance d'une Žquipe gŽographiquement correcte (donc
Asie,
Afrique et Europe mŽditerranŽenne exclues).
Conjecture
numŽro 3 : le facteur d'impact d'une revue est inversement corrŽlŽ
avec la
longŽvitŽ des articles qu'elle publie.
Au
moment o les revues ˆ grand facteur d'impact publient leurs articles,
elles rŽussissent
en gŽnŽral ˆ illusionner les non-spŽcialistes. Au bout de quelques
annŽes, les
apports rŽels sont mieux cernŽs et l'on cite alors les vŽritables
articles
fondateurs. De ce fait, impact et temps de demi-vie devraient tre
anti-corrŽlŽs.
Conjecture
numŽro 4 : le facteur d'impact d'une revue est directement corrŽlŽ
ˆ son
taux d'articles frauduleux.
Dans
les revues ˆ grand facteur d'impact, les articles soumis sont triŽs par
les Žditeurs,
avant d'tre Žventuellement envoyŽs aux rapporteurs les plus
compŽtents. L'Žvaluation
par les pairs est, pour l'essentiel, remplacŽe par le jugement au doigt
mouillŽ
par l'Žditeur. Sachant qu'une prŽsŽlection sera faite par un
non-spŽcialiste,
les auteurs ont tendance ˆ survaloriser leur travail, et en faire une
prŽsentation
malhonnte. Ce systme joue au bŽnŽfice des fraudeurs.
En
forme de conclusion
De
manire gŽnŽrale, alors que nous oeuvrons dans un monde concurrentiel
sans
merci, je comprends mal qu'en France, au lieu d'tre jugŽs sur notre
travail,
nous soyons de plus en plus jugŽs sur la place qu'accordent ˆ notre
travail nos
pires concurrents. Les administrateurs de la recherche feraient oeuvre
utile en
Žtudiant avec sŽrieux la nature et l'Žtendue des distorsions que
vŽhiculent les
critres soi-disant objectifs.
Texte paru dans
"La Vie
de la Recherche Scientifique" (VRS),
mars
2004, vol. 305, page 33
Jacques
Ninio
LPS,
ENS, 24 rue Lhomond, 75231 Paris cedex 05
Site
web: http://www.lps.ens.fr/~ninio
=================================================
LINKS:
Spanish translation, by Eduardo Mizraji:
click here for the
pdf version
A
detailed analysis of the uses and abuses of impact factors by Antoine
Danchin:
http://www.pasteur.fr/recherche/unites/REG/bibliometry.html
====================================================