FOUR CONJECTURES ON IMPACT FACTORS

 

Or why impact factors correlate with ignorance, lack of originality, obsolescence and fraud.

 

 

 

Followed by :

 

-       a quotation and an anecdote

-       the original French version of the text

-       a link to the Spanish version, by Eduardo Mizaji

-       a link to an analysis by Antoine Danchin

 

 

 

                            Jacques Ninio

 

Laboratoire de Physique Statistique, Ecole Normale SupŽrieure,

24 rue Lhomond, 75231 Paris cedex 05, France

 

Each statistical indicator entails biases which need to be identified and corrected. However, in the evaluation of scientific research, popularity tests are used as substitutes for quality tests, a practice which penalises our most original productions.

 

I criticized in the past the use of citation counts in the evaluation of the quality of a scientific work. Now the citation count criterion is neglected to the benefit of an even less pertinent one, the impact factors (1) of the journals in which a scientist publishes his work. What does the impact factor really measure ? I propose here four conjectures which stem from my experience in life sciences.

 

Conjecture 1 : The impact factor of a journal is directly correlated with the incompetence of those who cite it.

 

When a scientist writes an article in his domain of research, but needs for this article to mention a result or an idea from another domain, he will first look into the journals which are the most accessible to him, thus the generalist journals which are received in his laboratoryÕs library. In a field in which the researcher is not competent, it is easier for him to quote Nature or Science, than to perform a meticulous bibliographic inquiry. My conjecture is that the hierarchy of impact factors reflects above all the statistical weight of the quotations made by the non-specialists.

 

Conjecture 2 : Impact factors are directly correlated with lack of originality.

 

Impact factors are computed on the basis of the citations received by published articles during the year of their publication and the following year. An article which is ahead of its time and starts being cited ten years after its publication, reduces the impact factor of the journal in which it is published. An article in an emergent field which is of concern to only a small community of researchers is rarely quoted. Thus, it pulls the impact factor downwards. Knowing that,  the editor which is obsessed by the impact factor of his journal rejects innovating, avant-garde articles. Once the subject takes roots, the editor makes up for his past neglect and publishes as a big revolution, the first article on the subject which is submitted to him emanating from a suitable source (excluding, e.g., laboratories from Mediterranean Europe).

 

Conjecture 3 : The impact factor of a journal is inversely correlated with the longevity of the articles it publishes.

 

When the high impact factor journals publish their articles, they succeed in general to fool the non-specialists. However, after a few years, the really important contributions become known through the quotations made by real specialists. Then, these founding articles become also quoted by the non-specialists. On this account, impact factors and half-lives should be anti-correlated.

 

Conjecture 4 : The impact factor of a journal is directly correlated with its rate of fraudulent articles.

 

In high impact factor journals, articles are sorted by editors, who essentially determine the windÕs direction from their wet finger. Most of the articles are eliminated before having even a chance to be sent to the reviewers. Knowing that the crucial stage of  selection is under the control of non-specialists, the authors have a tendency to overstate the significance of their work, and present it in a dishonest fashion. This system works to the benefit of cheaters.

 

 

Concluding remarks.

 

While scientific research is now carried out in the context of a merciless world, it is curious that science administrators, which are in their majority under the influence of financial and marketing ideologies,  judge our work as though it were unfolding in a different world, governed by pure altruistic behaviour : A world in which our worse competitors, upon reading a breakthough manuscript which relegates their past and present work into obsolescence, will  lean down and recommend the publication of the manuscript with high priority. Science administrators would do a good job by studying seriously the distorsions which are introduced by the so-called objective criteria of scientific evaluation.

 

-------------------------------------

 

(1) The impact factor was launched by Eugene Garfield and his Institute for Scientific Information (ISI). The Science Citation Index, invented by Eugene Garfield, is an amazingly powerful bibliographic scientific tool. However, the bibliometric indexes which are derived from the SCI  have disastrous effects. They reduce science to marketing.

 

  ================================

 

Bonus for the faithful readers :

 

A quotation

From the concluding section of my article in Ç Biochimie È [1]

 

            Ç An article by FaxŽn, Kirsebom and Isaksson [50] played an extremely important role in this work. I will stress here the qualities of this article, because it has all the attributes which would make it unpublishable to-day in the so-called "high impact factor journals". First of all, it was deeply honest. Instead of making a narrow selection of their experimental observations, and using such a selection to promote simple-minded theories that would have appealled to a broad audience of researchers from other fields, they laid down all the results of their genetic experiments on the table. Next, part of the data was in line with current theories, others were not, but the authors interpreted their work in an extremely modest way. This attitude would make it impossible to "sell" the work as a revolution in the field. Above all, the work was extremely systematic. It  covered, in a consistent way nonsense suppression, codon context effects, streptomycin effects, ribosomal mutations, and anticodon modifications. All the clues were there, but hard to decipher. After all, the ribosome is one of the most complex achievements of molecular evolution. While progressing in the comprehension of the ribosomal puzzle, I often returned to the work of FaxŽn et al. [50] to determine if at this point their data became more intelligible. While the proposals made in this review do not explain in detail all their results, they provide at least sufficient leads to envision tentative explanations. È

 

[1] Ninio, J. (2006) Multiple stages in codon-anticodon recognition : double-trigger mechanisms and geometric constraints. Biochimie 88, 963-992.

 

[50] FaxŽn, M., Kirsebom, L.A. and Isaksson, L.A. (1988) Is efficiency of suppressor tRNAs controlled at the level of ribosomal proofreading in vivo? J. Bacteriol. 170, 3756-3760.

 

 

An anecdote

On the calamitous management of science in a European country.

 

A few years ago, a team of  scientists from Shanghai University, in the PeopleÕs Republic of China made public their personal  ranking of the worldÕs top scientific institutions. The ranking was based on criteria such as the number of publications in high impact factor journals or the number of patents emanating from the institutions, the number of Nobel awards, etc. As far as I know, they did not apply even the most obvious corrective factors, such as (i) dividing the output buy the number of employees, (ii) relating the output to the budget. Taking such factors into account, one would have had a better idea of the efficiency and productivity of the employees.

 

How do you think that the science administrators from this European country reacted ? Instead of throwing the Shanghai study into the waste basket, they took it seriously. In order to improve the ranking of their countryÕs institutions in the ShanghaiÕs list, they decided to regroup the institutions by twos or threes under a same name . As long as this was just a game of changing names, the consequences were not too deleterious. Unfortunately, the administrators decided that there should be a real reorganization of the institutions. So, for the last two years a large number of the leading researchers of the country have abandoned their scientific work, and are elaborating instead  ambitious Ç federative È projects, which would tie several institutions under a same banner, and they are attempting to redefine hundreds of research projects carried out in the allied institutes, so as to make them fit into an administratively coherent framework. By the time they will have finished their restructuration projects, no doubt that another administration, issued from the latest political elections, will push forward a new strategic plan, requiring different modes of reorganization.

 

What intrigues me most, in this state of affair, is why there is no resistance to the administrationÕs idiotic pressures ? Apparently, most researchers in the country of which I am speaking are coward opportunists, they are convinced that it is better to comply superficially with the demands from the administration. They believe that once they have complied, and have consequently received funding and support from the authorities on the basis of fake projects, they will be able to carry out real, innovative scientific projects. In my experience, the pressure and the controls are such that there is less and less room these days for real scientific projects.

 

I am also intrigued by the fact that the science administrators do not even realize that their policy is self-defeating. For if  the Shanghai game is taken seriously in other countries, what will prevent the institutions of  the other countries to play the same administrative regrouping game ? What can prevent Harvard University and the MIT to be united into a larger administrative entity, the Harmit university ? Should we then reply by uniting Paris, Hamburg and Cambridge into a Ç Campaham È research mega-center ? Once you start thinking in these terms, you begin to see more clearly the motivations behind the Shanghai list. The authors of the list are not absolute cretins, as you might have thought initially.  They are working for the glory of their country. Remember that China has more than a billion inhabitants, which is more than three times the whole European community. Thus any ranking of scientific output, which is related to global production, and not to productivity per employee should, in the long term, turn to the advantage of China or India.

 

 

    ================================================================

 

 

           QUATRE CONJECTURES SUR LES FACTEURS D'IMPACT

 

 

                                 Jacques Ninio

 

 

 

Tout indicateur statistique est affectŽ de biais qu'il convient de repŽrer et corriger. Or, en Žvaluation de la recherche, les tests d'audience sont pris pour des tests de qualitŽ, ce qui pŽnalise nos travaux les plus originaux, au bŽnŽfice de nos concurrents Žtrangers.

 

Dans les pages jaunes d'une VRS d'autrefois, j'avais attirŽ l'attention sur les risques d'utilisation des citations dans l'Žvaluation scientifique (avril 1978, pages 26-28). Aujourd'hui, les citations reues par un auteur sont elles-mmes dŽlaissŽes au profit d'un critre encore moins pertinent, celui du Ò facteur d'impact Ó des revues dans lesquelles il publie. Que mesure ce facteur ? Je propose ici quatre conjectures, issues de mon expŽrience dans les sciences de la vie:

 

Conjecture numŽro 1 : le facteur d'impact d'une revue est directement corrŽlŽ ˆ l'incompŽtence de ceux qui la citent.

 

Un chercheur qui, pour les besoins d'un article de sa spŽcialitŽ, Žprouve le besoin de mentionner un rŽsultat ou une idŽe relevant d'une autre discipline, ira d'abord puiser dans les revues les plus accessibles, donc les revues gŽnŽralistes reues dans son laboratoire. Dans une discipline o il n'est pas compŽtent, il lui est plus commode de citer Nature que de faire une enqute bibliographique minutieuse. Ma conjecture est que la hiŽrarchie des facteurs d'impact reflte d'abord le poids des citations faites par les non-spŽcialistes.

 

Conjecture numŽro 2 : le facteur d'impact est directement corrŽlŽ avec l'absence d'originalitŽ.

 

Le facteur d'impact est calculŽ sur les citations recueillies par les articles d'une revue l'annŽe mme de leur publication et l'annŽe suivante. Un article en avance sur son temps, qui commence ˆ tre citŽ dix ans plus tard abaisse donc le facteur d'impact de la revue. Un article dans un secteur novateur, qui n'implique et n'attire l'attention que d'une petite communautŽ de chercheurs est, de ce fait, peu citŽ, et tire le facteur d'impact vers le bas. Sachant cela, l'Žditeur obsŽdŽ par le facteur d'impact de sa revue refuse les articles novateurs ou d'avant-garde. Une fois que le sujet a commencŽ ˆ diffuser, l'Žditeur se rattrape en publiant comme une grande nouveautŽ, le premier article qui lui est soumis en provenance d'une Žquipe gŽographiquement correcte (donc Asie, Afrique et Europe mŽditerranŽenne exclues).

 

Conjecture numŽro 3 : le facteur d'impact d'une revue est inversement corrŽlŽ avec la longŽvitŽ des articles qu'elle publie.

 

Au moment o les revues ˆ grand facteur d'impact publient leurs articles, elles rŽussissent en gŽnŽral ˆ illusionner les non-spŽcialistes. Au bout de quelques annŽes, les apports rŽels sont mieux cernŽs et l'on cite alors les vŽritables articles fondateurs. De ce fait, impact et temps de demi-vie devraient tre anti-corrŽlŽs.

 

Conjecture numŽro 4 : le facteur d'impact d'une revue est directement corrŽlŽ ˆ son taux d'articles frauduleux.

 

Dans les revues ˆ grand facteur d'impact, les articles soumis sont triŽs par les Žditeurs, avant d'tre Žventuellement envoyŽs aux rapporteurs les plus compŽtents. L'Žvaluation par les pairs est, pour l'essentiel, remplacŽe par le jugement au doigt mouillŽ par l'Žditeur. Sachant qu'une prŽsŽlection sera faite par un non-spŽcialiste, les auteurs ont tendance ˆ survaloriser leur travail, et en faire une prŽsentation malhonnte. Ce systme joue au bŽnŽfice des fraudeurs.

 

En forme de conclusion

De manire gŽnŽrale, alors que nous oeuvrons dans un monde concurrentiel sans merci, je comprends mal qu'en France, au lieu d'tre jugŽs sur notre travail, nous soyons de plus en plus jugŽs sur la place qu'accordent ˆ notre travail nos pires concurrents. Les administrateurs de la recherche feraient oeuvre utile en Žtudiant avec sŽrieux la nature et l'Žtendue des distorsions que vŽhiculent les critres soi-disant objectifs.

 

 

 

Texte paru dans "La Vie de la Recherche Scientifique" (VRS),

mars 2004, vol. 305, page 33

 

 

Jacques Ninio

LPS, ENS, 24 rue Lhomond, 75231 Paris cedex 05

Site web: http://www.lps.ens.fr/~ninio

 

 =================================================

 

LINKS: Spanish translation, by Eduardo Mizraji:

click here for the pdf version

 

A detailed analysis of the uses and abuses of impact factors by Antoine Danchin:

http://www.pasteur.fr/recherche/unites/REG/bibliometry.html

 

====================================================