FOUR CONJECTURES ON IMPACT FACTORS
Or why impact factors correlate with ignorance, lack of originality, obsolescence and fraud.
Followed by :
- a quotation and an anecdote
- the original French version of the text
- a link to the Spanish version, by Eduardo Mizaji
- a link to an analysis by Antoine Danchin
Laboratoire de Physique Statistique, Ecole Normale Suprieure,
24 rue Lhomond, 75231 Paris cedex 05, France
Each statistical indicator entails biases which need to be identified and corrected. However, in the evaluation of scientific research, popularity tests are used as substitutes for quality tests, a practice which penalises our most original productions.
I criticized in the past the use of citation counts in the evaluation of the quality of a scientific work. Now the citation count criterion is neglected to the benefit of an even less pertinent one, the impact factors (1) of the journals in which a scientist publishes his work. What does the impact factor really measure ? I propose here four conjectures which stem from my experience in life sciences.
Conjecture 1 : The impact factor of a journal is directly correlated with the incompetence of those who cite it.
When a scientist writes an article in his domain of research, but needs for this article to mention a result or an idea from another domain, he will first look into the journals which are the most accessible to him, thus the generalist journals which are received in his laboratoryÕs library. In a field in which the researcher is not competent, it is easier for him to quote Nature or Science, than to perform a meticulous bibliographic inquiry. My conjecture is that the hierarchy of impact factors reflects above all the statistical weight of the quotations made by the non-specialists.
Conjecture 2 : Impact factors are directly correlated with lack of originality.
Impact factors are computed on the basis of the citations received by published articles during the year of their publication and the following year. An article which is ahead of its time and starts being cited ten years after its publication, reduces the impact factor of the journal in which it is published. An article in an emergent field which is of concern to only a small community of researchers is rarely quoted. Thus, it pulls the impact factor downwards. Knowing that, the editor which is obsessed by the impact factor of his journal rejects innovating, avant-garde articles. Once the subject takes roots, the editor makes up for his past neglect and publishes as a big revolution, the first article on the subject which is submitted to him emanating from a suitable source (excluding, e.g., laboratories from Mediterranean Europe).
Conjecture 3 : The impact factor of a journal is inversely correlated with the longevity of the articles it publishes.
When the high impact factor journals publish their articles, they succeed in general to fool the non-specialists. However, after a few years, the really important contributions become known through the quotations made by real specialists. Then, these founding articles become also quoted by the non-specialists. On this account, impact factors and half-lives should be anti-correlated.
Conjecture 4 : The impact factor of a journal is directly correlated with its rate of fraudulent articles.
In high impact factor journals, articles are sorted by editors, who essentially determine the windÕs direction from their wet finger. Most of the articles are eliminated before having even a chance to be sent to the reviewers. Knowing that the crucial stage of selection is under the control of non-specialists, the authors have a tendency to overstate the significance of their work, and present it in a dishonest fashion. This system works to the benefit of cheaters.
While scientific research is now carried out in the context of a merciless world, it is curious that science administrators, which are in their majority under the influence of financial and marketing ideologies, judge our work as though it were unfolding in a different world, governed by pure altruistic behaviour : A world in which our worse competitors, upon reading a breakthough manuscript which relegates their past and present work into obsolescence, will lean down and recommend the publication of the manuscript with high priority. Science administrators would do a good job by studying seriously the distorsions which are introduced by the so-called objective criteria of scientific evaluation.
(1) The impact factor was launched by Eugene Garfield and his Institute for Scientific Information (ISI). The Science Citation Index, invented by Eugene Garfield, is an amazingly powerful bibliographic scientific tool. However, the bibliometric indexes which are derived from the SCI have disastrous effects. They reduce science to marketing.
Bonus for the faithful readers :
From the concluding section of my article in Ç Biochimie È 
Ç An article by Faxn, Kirsebom and Isaksson  played an extremely important role in this work. I will stress here the qualities of this article, because it has all the attributes which would make it unpublishable to-day in the so-called "high impact factor journals". First of all, it was deeply honest. Instead of making a narrow selection of their experimental observations, and using such a selection to promote simple-minded theories that would have appealled to a broad audience of researchers from other fields, they laid down all the results of their genetic experiments on the table. Next, part of the data was in line with current theories, others were not, but the authors interpreted their work in an extremely modest way. This attitude would make it impossible to "sell" the work as a revolution in the field. Above all, the work was extremely systematic. It covered, in a consistent way nonsense suppression, codon context effects, streptomycin effects, ribosomal mutations, and anticodon modifications. All the clues were there, but hard to decipher. After all, the ribosome is one of the most complex achievements of molecular evolution. While progressing in the comprehension of the ribosomal puzzle, I often returned to the work of Faxn et al.  to determine if at this point their data became more intelligible. While the proposals made in this review do not explain in detail all their results, they provide at least sufficient leads to envision tentative explanations. È
 Ninio, J. (2006) Multiple stages in codon-anticodon recognition : double-trigger mechanisms and geometric constraints. Biochimie 88, 963-992.
 Faxn, M., Kirsebom, L.A. and Isaksson, L.A. (1988) Is efficiency of suppressor tRNAs controlled at the level of ribosomal proofreading in vivo? J. Bacteriol. 170, 3756-3760.
On the calamitous management of science in a European country.
A few years ago, a team of scientists from Shanghai University, in the PeopleÕs Republic of China made public their personal ranking of the worldÕs top scientific institutions. The ranking was based on criteria such as the number of publications in high impact factor journals or the number of patents emanating from the institutions, the number of Nobel awards, etc. As far as I know, they did not apply even the most obvious corrective factors, such as (i) dividing the output buy the number of employees, (ii) relating the output to the budget. Taking such factors into account, one would have had a better idea of the efficiency and productivity of the employees.
How do you think that the science administrators from this European country reacted ? Instead of throwing the Shanghai study into the waste basket, they took it seriously. In order to improve the ranking of their countryÕs institutions in the ShanghaiÕs list, they decided to regroup the institutions by twos or threes under a same name . As long as this was just a game of changing names, the consequences were not too deleterious. Unfortunately, the administrators decided that there should be a real reorganization of the institutions. So, for the last two years a large number of the leading researchers of the country have abandoned their scientific work, and are elaborating instead ambitious Ç federative È projects, which would tie several institutions under a same banner, and they are attempting to redefine hundreds of research projects carried out in the allied institutes, so as to make them fit into an administratively coherent framework. By the time they will have finished their restructuration projects, no doubt that another administration, issued from the latest political elections, will push forward a new strategic plan, requiring different modes of reorganization.
What intrigues me most, in this state of affair, is why there is no resistance to the administrationÕs idiotic pressures ? Apparently, most researchers in the country of which I am speaking are coward opportunists, they are convinced that it is better to comply superficially with the demands from the administration. They believe that once they have complied, and have consequently received funding and support from the authorities on the basis of fake projects, they will be able to carry out real, innovative scientific projects. In my experience, the pressure and the controls are such that there is less and less room these days for real scientific projects.
I am also intrigued by the fact that the science administrators do not even realize that their policy is self-defeating. For if the Shanghai game is taken seriously in other countries, what will prevent the institutions of the other countries to play the same administrative regrouping game ? What can prevent Harvard University and the MIT to be united into a larger administrative entity, the Harmit university ? Should we then reply by uniting Paris, Hamburg and Cambridge into a Ç Campaham È research mega-center ? Once you start thinking in these terms, you begin to see more clearly the motivations behind the Shanghai list. The authors of the list are not absolute cretins, as you might have thought initially. They are working for the glory of their country. Remember that China has more than a billion inhabitants, which is more than three times the whole European community. Thus any ranking of scientific output, which is related to global production, and not to productivity per employee should, in the long term, turn to the advantage of China or India.
QUATRE CONJECTURES SUR LES FACTEURS D'IMPACT
Tout indicateur statistique est affect de biais qu'il convient de reprer et corriger. Or, en valuation de la recherche, les tests d'audience sont pris pour des tests de qualit, ce qui pnalise nos travaux les plus originaux, au bnfice de nos concurrents trangers.
Dans les pages jaunes d'une VRS d'autrefois, j'avais attir l'attention sur les risques d'utilisation des citations dans l'valuation scientifique (avril 1978, pages 26-28). Aujourd'hui, les citations reues par un auteur sont elles-mmes dlaisses au profit d'un critre encore moins pertinent, celui du Ò facteur d'impact Ó des revues dans lesquelles il publie. Que mesure ce facteur ? Je propose ici quatre conjectures, issues de mon exprience dans les sciences de la vie:
Conjecture numro 1 : le facteur d'impact d'une revue est directement corrl l'incomptence de ceux qui la citent.
Un chercheur qui, pour les besoins d'un article de sa spcialit, prouve le besoin de mentionner un rsultat ou une ide relevant d'une autre discipline, ira d'abord puiser dans les revues les plus accessibles, donc les revues gnralistes reues dans son laboratoire. Dans une discipline o il n'est pas comptent, il lui est plus commode de citer Nature que de faire une enqute bibliographique minutieuse. Ma conjecture est que la hirarchie des facteurs d'impact reflte d'abord le poids des citations faites par les non-spcialistes.
Conjecture numro 2 : le facteur d'impact est directement corrl avec l'absence d'originalit.
Le facteur d'impact est calcul sur les citations recueillies par les articles d'une revue l'anne mme de leur publication et l'anne suivante. Un article en avance sur son temps, qui commence tre cit dix ans plus tard abaisse donc le facteur d'impact de la revue. Un article dans un secteur novateur, qui n'implique et n'attire l'attention que d'une petite communaut de chercheurs est, de ce fait, peu cit, et tire le facteur d'impact vers le bas. Sachant cela, l'diteur obsd par le facteur d'impact de sa revue refuse les articles novateurs ou d'avant-garde. Une fois que le sujet a commenc diffuser, l'diteur se rattrape en publiant comme une grande nouveaut, le premier article qui lui est soumis en provenance d'une quipe gographiquement correcte (donc Asie, Afrique et Europe mditerranenne exclues).
Conjecture numro 3 : le facteur d'impact d'une revue est inversement corrl avec la longvit des articles qu'elle publie.
Au moment o les revues grand facteur d'impact publient leurs articles, elles russissent en gnral illusionner les non-spcialistes. Au bout de quelques annes, les apports rels sont mieux cerns et l'on cite alors les vritables articles fondateurs. De ce fait, impact et temps de demi-vie devraient tre anti-corrls.
Conjecture numro 4 : le facteur d'impact d'une revue est directement corrl son taux d'articles frauduleux.
Dans les revues grand facteur d'impact, les articles soumis sont tris par les diteurs, avant d'tre ventuellement envoys aux rapporteurs les plus comptents. L'valuation par les pairs est, pour l'essentiel, remplace par le jugement au doigt mouill par l'diteur. Sachant qu'une prslection sera faite par un non-spcialiste, les auteurs ont tendance survaloriser leur travail, et en faire une prsentation malhonnte. Ce systme joue au bnfice des fraudeurs.
En forme de conclusion
De manire gnrale, alors que nous oeuvrons dans un monde concurrentiel sans merci, je comprends mal qu'en France, au lieu d'tre jugs sur notre travail, nous soyons de plus en plus jugs sur la place qu'accordent notre travail nos pires concurrents. Les administrateurs de la recherche feraient oeuvre utile en tudiant avec srieux la nature et l'tendue des distorsions que vhiculent les critres soi-disant objectifs.
Texte paru dans "La Vie de la Recherche Scientifique" (VRS),
mars 2004, vol. 305, page 33
LPS, ENS, 24 rue Lhomond, 75231 Paris cedex 05
Site web: http://www.lps.ens.fr/~ninio
LINKS: Spanish translation, by Eduardo Mizraji:
A detailed analysis of the uses and abuses of impact factors by Antoine Danchin: