A More Comprehensive Index in the Evaluation of Scientific Research: The Single Researcher Impact Factor Proposal
Gianluca Castelnuovo*, 1, Daniel Limonta 2, Luis Sarmiento 3, Enrico Molinari 4
Identifiers and Pagination:Year: 2010
First Page: 109
Last Page: 114
Publisher ID: CPEMH-6-109
Article History:Received Date: 24/12/2009
Revision Received Date: 27/5/2010
Acceptance Date: 6/6/2010
Electronic publication date: 5/11/2010
Collection year: 2010
open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.
Good alternatives to the Impact Factor (IF) algorithm are needed. The Thomson IF represents a limited measure of the importance of an individual article because 80% of a journal's IF is determined by only the 20% of the papers published. In the past few years, several new indexes has been created to provide alternatives to the IF algorithm. These include the removal of self citations from the calculation of the IF using the Adjusted IF, Index Copernicus initiative and other modifications such as the Cited Half-Life IF, Median IF, Disciplinary IF, and Prestige Factor. There is also the Euro-Factor, born in Europe to avoid the strong US centrality, and the English language basis of the Thomson database. One possible strategy to avoid "IF supremacy" is to create a new index, the Single Researcher Impact Factor (SRIF), that would move the evaluation from the power of scientific journals to the quality of single researchers. This measure can take into account the number and quality of the traditional publications and other activities usually associated with being a researcher, such as reviewing manuscripts, writing books, and attending scientific meetings. Also, in funding policy, it might be more useful to consider the merits, contributions, and real impact of all the scientific activities of a single researcher instead of adding only the journals' IF numbers. The major aim of this paper is to propose and describe the SRIF index that could represent a novel option to evaluate scientific research and researchers.
Today the assessment of scientific productivity and quality in different emerging and established scientific areas is a very controversial issue. The measure of quality is used to decide whether or not a scientist is promoted to principal investigator, obtain a better position in a department, earn a PhD degree, is given faculty tenure and is awarded an important research funding. Indeed, the evaluation of the scientific work has become a serious daily practice in basic and clinical settings [1-5].
The evaluation of the researcher scientific quality is a recognized difficulty that has not historically received a solution in a standard way. The ideal aim of scientific results assessment is to evaluate if the research is published in the international literature. The research work should be submitted for critical revision by experts in the specific area, peer reviewing, that according to established regulations state the quantitative and qualitative significance of a certain researcher work. However, the so-called peer review is often conducted by scientific committees with generic competence rather than by recognized experts of the specific theme and the referees may be uninformed [1, 6]. In addition, it may be painful for the authors, mainly the youngest ones, because of the referee demands . These scientific assessment groups have the tendency of using secondary criterions as the “raw” calculation of publications number, prestige of journals in which authors publish, reputation of authors and institutes involved in the research, and estimated importance and relevance of the research field .
In this work we will comment initially the essential information and some observations on the use and abuse of the journal Impact Factor (IF), the most extended and controversial index of international scientific prestige. This introductory debate will enable us to present some current proposals that address the dilemma of scientific quality assessment.
JOURNAL IMPACT FACTOR DEFINITION
The IF was created by Eugene Garfield and Irving H. Sher (first mentioned in 1955) to take action on the claiming from the international scientific community to have a simple method to compare journals [8-10]. This number, IF, is obtained after the following calculation:
IF (2007) = Citations in 2007 (in journals indexed by Thomson Scientific) of articles published in the journal in 2005–2006 divided by the number of “citable” articles by Thomson Scientific that were published in 2005–2006 by the journal.
The citations records are obtained from the Science Citation Index (SCI), an electronic database that records the scientific citations in the reference lists of journals articles.
It is important to bear in mind that the legitimate use, for which the IF was created according to the original intentions of Garfield, was to help libraries in the selection of journals to purchase, assist the authors in the selection of journals to send their own works, and provide an index of journals prestige in which the authors have published.
In this regard, Garfield [8-10] stated that the concept of IF has suffered an evolution during time, acquiring by mistake the double meaning of impact of the journal and impact of the author. In fact, in the last years the IF in the European region has always been more used as an index of quality and scientific excellence to receive research projects and funding, and recognition and prestige in the best research institutions [4, 11]. However, Garfield stated that the IF is mainly a useful index to estimate the expected frequency of citations in comparison to the actually obtained , although a strong correlation exists between an elevated IF and the journal prestige in the current scientific field .
Before we extend the analysis of this index, we have to consider that the IF has become important for the evaluation of research activities in a lot of disciplines, and reached a growing influence in the scientific community. In this regard, the increased use/abuse of the IF might determine the destiny of single researchers, departments and even academic institutions [4, 13-15].
ALTERNATIVES TO THE JOURNAL IMPACT FACTOR
Because the IF is derived from citations to all articles in a journal, this number is not statistically representative of single articles. Actually, the greatest part of citations are usually associated to one small part of total articles published by a journal, therefore the IF does not seem to be an appropriate measure of citations obtained by a “regular” article. Indeed, the 80% of a journal's IF is determined by only the 20% of the papers published. Therefore, a high IF doesn't assure a high productivity and scientific excellence of a single researcher and represents a limited measure of the importance of an individual article [1, 16-20].
Other problems (limitations) associated to the use of the IF as qualitative measure of the author’s scientific achievements include that books are not included in the database as a source for citations, and the database has the bias of English language exclusively. We mention in this context the article: “Does the impact factor kill the German language?” and consequently is dominated by American publications. On the other hand, limited fields of research do not tend to have journals with high IF .
In the past few years, several new indexes have been created to provide alternatives to the IF algorithm. These include the removal of self citations from the calculation of the IF which improve the calculation of the index . A more actual image of the influence of a journal in its scientific field, rather than among all the journals in general, might be a further variant of the impact factor, defined as Scope-Adjusted Impact Factor . The calculation of this index predominantly keeps in mind the thematic area and determines a more different journal classification in comparison to generic original IF which is not “repaired¨. Coelho and colleagues , support a reasonable index, notably as the IF traditional analysis through the Kruskal-Wallis that demonstrates it is not possible to compare different disciplines using the IF without adjustments.
Other diverse alternative indexes for the IF recently reviewed by Dong and colleagues , might be commented such as the Cited Half-Life IF (CHAL-IF), introduced by Sombatsompop, Markpin and Premkamolnetr , the Median IF (MIF) proposed by Rousseau , the Disciplinary IF (DIF) proposed by Hirst  and also selected by Pudovkin and Garfield  and the Journal Performance Indicator (JPI) http://scientific.thomson.com/products/jpi/. Among these proposals, it is particularly significant the European-Factor (EF)  which tries to solve the linguistic bias (absolute presence of English language) and territorial (USA-centrality) of the traditional IF system. This index is supported by VICER Publishing http://www.vicer.org/.
Van Leeuwen and Moed  have developed an alternative system, for the journal impact measurement, named Journal to Field Impact Score (JFIS). This index includes research articles, technical notes, editorial letters and reviews either in the numerator or in the denominator of the equation. Moreover, the JFIS takes into account the mean citations regarding the specific discipline that is considered.
Consequently, there has been an increase in the comprehensive systems for assessment of scientific productivity impact that also take into consideration the development in the web (in some cases exclusively from the web) of some journals, for instance: the External Web IF (WIF), defined by the number of external pages (not belonging to the site in evaluation) containing a link to the data in the Web .
Fava and colleagues  has discussed very often the IF alternatives topic and proposed the citations analysis as a suitable substitute, affirming that the only reliable system to evaluate the researcher impact are the actual citations, which is possible through the Web of Science from the Institute for Scientific Information (ISI). Therefore, the IF has importance for verifying how much the researchers have contributed to a journal and their development during years; however this index doesn't express a judgment of scientific significance.
At present, three electronic scientific databases provides a measure of an individual’s citation rate: Index Copernicus Scientists , Scopus  and Thomson-Reuters’ Researchers ID . Remarkably, Index Copernicus Scientists is the only one which calculates an individualized impact factor .
The Index Copernicus Scientists http://www.index-copernicus.com/info.php?id=1 besides providing scientists with global scientists networking and international research collaboration, present a multi-parameter career assessment system which analyses the researcher individual profile . This goal is accomplished by a uniform scoring system that evaluates the achievements of scientists in three areas of professional activity: research potential [R], teaching potential [T] and administration experience [A]. This Index Copernicus Scientists database may be useful for making decisions for scientific degree promotion purposes, and searching for the best candidates to lead a research group.
Scopus (http://www.scopus.com/), although limited to articles published after 1995, is the world's largest abstract and citation database of peer-reviewed literature and quality web sources [35, 37] and its Scopus Citation Tracker provides a simple way to investigate citations in a number of ways www.info.scopus.com/ctracker. One manner is to evaluate the real-time citation data of articles and authors. Very recently, Scopus announced that the h-index will be also incorporated http://info.scopus.com/news/press/pr_ 140407.asp. It has been postulated previously that H-index may quantify the scientific output of the researchers  and in a more recent work is discussed its power to predict future scientific achievements . Interestingly, Scopus database was used very recently to distinguish the different productivity and visibility of Cuban neuroscientists .
The ResearcherID is a website http://www.researcherid. com/ where researchers can register for a unique researcher ID number to avoid the frequent difficulty of author misidentification. In this way the researcher build their publication list using Web of Science and citations counts will be automatically updated in order to generate citation metrics. The h-index for ResearcherID participants is also calculated .
Remarkably, it was discussed very recently the need for speed the personal IF assessment in which the subject of calculation is the scientist and not the journal. This measure would really provide a more accurate measure of an individual’s citation rate .
THE SINGLE RESEARCHER IMPACT FACTOR (SRIF) PROPOSAL
One possible strategy to avoid "IF supremacy" is to propose a new index, the Single Researcher Impact Factor (SRIF) that would move the evaluation from the power of scientific journals to the quality of single researchers [41, 42]. In comparison with the Index Copernicus scientific approach previously described , the SRIF is similarly focused on the single researcher, but it is not, just dedicated to the scientific “impact factor” not considering the complex (not only scientific) researcher individual profile for making decisions and searching for the best candidates to lead a research group.
This measure can take into account the number and quality of traditional publications and other activities usually associated with being a researcher, such as reviewing manuscripts , write a book, and participating in scientific meetings. Additionally, in funding policy, it might be more useful to consider the merits, contributions, and real impacts of all the scientific activities of a single researcher, instead of measuring or adding only the IF numbers.
We will take into account some aspects of the suggestions done previously by Laudanna and colleagues  to propose this impact factor integrative system, the SRIF, to asses in a better way the scientific productivity. In fact, more than an index, we propose a more comprehensive system of evaluation.
We will consider several kinds of scientific products and scientific activities:
- Publications (journal articles, books, oral and poster presentations in scientific meetings). These publications might be in press or in electronic-digital format.
- Products, that are elaborated as software, CD-ROM, videos, databases, etc.
- Activities, or all those reported scientific activities that do not necessarily correspond to the production of manuscripts or other elaborated products such as scientific positions or positions in conferences organization, participation in journal editorial boards, activities on human resources education, and participation in international funding projects.
Currently, these scientific activities balance and strength the traditional scientific productivity (journal publications, books, chapters in books), related to the management of modern science .
We take into consideration the definition, with precision, of certain scientific criterions that enable us to consider the significance of various types of publications, products and activities and to assign a determined score to them. These single scores will be summed at the end of the SRIF calculation if desired (Table 1).
Evaluation System of Scientific Publications and Products
|Articles in journals included in the SCI||9||4.5|
|Chapters in a book||8||4|
|Edition of a book||7||3.5|
|Articles in journals not included in the SCI||1||0.5|
|Oral presentations in Congresses/Workshops/Conferences||0.5||0.25|
|Poster presentation in Congresses/Workshops/Conferences||0.2||0.1|
|Electronic items: Databases, CD-ROM, Software, video||1||0.5|
Note: SCI, Science Citation Index
System for Scientific Activities Evaluation
|Editor in Chief||5||2.5|
|Editorial Board member||3||1.5|
|Scientific committee member||3||1.5|
|Societies, Foundations and Agencies|
Note: international teaching will be considered if the professors are included in international courses. International scientific categories (PhD, MSc) will be considered when they were obtained in international institutions.
In the classification of publications, some aspects are important for differentiating in a better way the several types of publications (books, articles, presentations in congresses):
- Reviewing rigorousness of submitted works: for instance, an article in a journal has usually a more rigorous selection than an oral presentation in a congress.
- Degree of idea originality in the work: it is supposed that a book authorship, in comparison with book editing work, generally allow the expression of more original thinking.
- Quantity of required work: a book editing usually requires a more accurate organization and presentation of several contributions than a poster in a meeting.
However, it is essential the flexibility of simple categories in the classification. The maximum and minimum values, fixed for every type of publication or product that tolerate variability in the score regarding its impact, for instance: a book authorship can receive a score from a minimum of 5 (national) to a maximum of 10 (international).
This solution assures enough variability to the scores, and avoids the risks of reductionism and rigidity because of simple classifications based on the importance of publication types, for instance, “article in journal” or “book” category. Accordingly, there is still the possibility that a certain book chapter receives, taking into account its specific value (international/national), a similar or superior score than an article in a journal.
Regarding the scientific publications and products scores, in order to stress the significance of the first author work, we propose to multiply the score by two when the SRIF of the first author is calculated.
Here, we describe in detail every category already described in Table 1.
- Books and journals articles: traditionally, they are the most wanted scientific products; therefore they have the maximum scores. It is highly acknowledged that international books may have slightly superior maximum scores than articles in journals included in a Citation Index if they are published by very prestigious publishing editorial companies. In the evaluation of one specific journal, the IF already takes into account the comparison of journals from the same discipline, the rejection rate of the journal, understood as the index of difficulty to overcome the publishing sieve that for some journals is above 90%, and the prestige of the journal editorial board. Remarkably, the calculation of the individual publications scores will also take into consideration the number of actual citations trough the Web of Science from the ISI because this number will be multiplied by the publication score (national/international) and later by the journal IF that the journal sustain at the time of the publication in the numerator. The denominator will be always 5 considering this calculation will be performed with articles published in the last five years. The calculation would be the following:
This calculation considers besides the international/national journal IF, the actual citation index of individual articles in a 5 years period. In this way, it is presented an alternative to the IF taking into account previous works postulating IF bias regarding the citation analysis  and the time (two years) included in the traditional IF calculation . The publication score calculation of the SRIF also differentiate from the very recent proposal of Mark R. Graczynski named Personal IF  that do not take into account the authorship order significance and the limited time period of the previous two years.
- Contributions to a book: it is accepted a book chapter requires less work and analysis than a whole book. In addition, the scientific visibility will be much greater to book authors.
- Presentations in congresses: the authorship of a poster presentation in a congress is considered inferior than an oral presentation. Thus, it is reasonable that a meeting poster, usually requiring a smaller commitment, assume lower scores in the category.
We will also take into consideration some activities that fluctuate from the scientific to the technical field, from the theory to the pragmatic field. In these activities, there are works developed for journals, congresses, panels, scientific management, awards, teaching, and projects coordination. It is also highlighted in a more detailed manner between a member of the scientific meeting committee and the chairperson; and in the case of journals among the editors and journals referees.
A summary of the scores attributed by the authors in these activities are described in Table 2.
Regarding the journals work, we propose to take advantages of the IF again. This time we will multiply the editors/referees score by the IF media (a mean between the journal IF at the beginning of the journal work and at the end of the work period), afterwards this number will be multiplied by the years in charge. The calculation of the work in individual journals would be in the following way:
Editors/referees IF = score × IF media × years of work
In this way, the extremely needed peer reviewing work for the success of modern science  would be rewarded. Regarding the importance of considering the peer reviewer responsibilities, it is important to take into account that writing and finalizing an article is a very complex process where reviewers can usually offer a valid and crucial scientific contribution that can make an article ready to be published and appreciated in the scientific community. In order to make peer-reviewers more compliant in their fundamental role for the improvement of science, one possible solution is to create a new index that can take into account other activities usually associated with being a researcher, such as reviewing manuscripts. By replacing the journal-centered IF with a single researcher-centered IF, that can include reviewing activity too, the evaluation of individual scientific impact in the community will be more accurate and could motivate researchers, overall young ones, in reviewing without frustration [45-47].
In order to check the validity of this SRIF proposal, our goal is to carry out a pilot study in a university or research institute to compare the usefulness of our SRIF with the traditional IF. It is important that our method could be challenged in a statistical study with the traditional IF.
In this work we have discussed that IF has merits and defects. We have also stressed on its limitations and the alternatives of such an index to continue the significant debate already emerged in the global scientific community. What are we going to do with the IF? Some authors think it is good to preserve such an index while is found another parameter or a better parameter . In the meantime the scientific community should finally elaborate not a substitutive index but an integrative one, or a battery of indexes. In this regard, we believe should be considered carefully the SRIF as an assessment system of scientific productivity. This system, with appropriate changes and integrations, would be able to adapt well to different scientific disciplines.
GC and DL conceived and designed the study. GC and DL contributed equally to this work. LS participated in the acquisition and analysis of data. All authors helped to draft, read and approved the final manuscript.
The authors declare that they have no competing interests.
This article was supported by the TECNOB Project (Technology for Obesity Project) -“Compagnia di San Paolo” Italian private foundation.