Research Management Review, Volume 22, Number 1 (2017) A Measure of Excellence of Young European Research Council Grantees Javier Arevalo Research Services, University of Helsinki, Finland ABSTRACT Bibliometric benchmarking can be an aid to researchers pondering whether to apply for competitive grants. In this paper, the highly prestigious grants offered by the European Research Council to young scientists of any nationality were scrutinized. The analysis of the 2014–2015 data indicates that over 75% of life science grantees in the starting category (2–7 years after completing a Ph.D. degree program) had at least 14 papers and an H-index of 10 (28 and 16, respectively ,in the case of the consolidator category—i.e., 7–12 years after obtaining the Ph.D.). Yet other signs of excellence, expert advice, and the limitations of metric approaches need to be considered. INTRODUCTION in ERC language), which will be put to the Think of a scientist pondering whether test during the evaluation process. to submit an application to the European However, a second criterion will have to be Research Council (ERC). The carrot is a met: the CV and track record also must be tasty one: joining the prestigious club of excellent. So what sort of measure of those—over 5,500 members of 66 excellence would that be? After all, even for nationalities, including over 180 U.S. the best scientist at a given department or nationals—who have received, since 2007, institution, a track record that is not one of the 5-year 1.5–2.5 million euro grant perceived as outstanding at the European (European Research Council, 2015). The level will have no chance, meaning months scientist may have an excellent project idea of preparation spent in vain. The opposite (ground-breaking and high-gain high-risk, may also occur, with scientists who could 1 Research Management Review, Volume 21, Number 1 (2016) potentially succeed not submitting an rounds were retrieved from Scopus. Figure application due to excessive modesty. 1 displays quartile values, which may offer Benchmarking merits against previous a useful reference to the pondering young grantees may be the next thought—in most scientist while controlling for outliers. The cases these are young scientists either highlighted first quartile values, for within the starting (2–7 years after Ph.D.) or example for the LS1 panel (Molecular and consolidator (7–12 years after Ph.D.) Structural Biology and Biochemistry), reveal categories (European Research Council, that 75% of grantees in the starting category 2016). And here is where, at least within had at least 13 papers (counted as Scopus- some disciplines (including the life type articles and reviews), 523 citations (not sciences), bibliometric analyses could prove shown) and H-index of 10. The equivalent handy. So with a focus on the ERC Life values for consolidators in the same panel Science (LS) domain and its 9 panels, key are 24 papers, 949 citations, and an H-index publication metrics for all 488 Starting and of 16. Consolidator grantees in the 2014 and 2015 Figure 1. Bibliometrics for the 2014–2015 ERC Starting and Consolidator Grantees in the life Sciences. (A) and (B) display number of papers and (C) and (D) H-indices. Value ranges are indicated by vertical lines. Boxes delimit the first (Q1) and third (Q3) quartiles, with median values also marked. Q1 values are shown and connected by a line. 2 Research Management Review, Volume 22, Number 1 (2017) The graphs also capture differences support such claims. With regard to the between panels, likely reflecting variation in latter, the data showed tremendous panel publication and citation practices by field. variations, with, for example, 80% of Values were somewhat lower in panels such starting grantees in LS1 having such a paper as LS3 (Cellular and Developmental versus just 18% in LS7. Considering the Biology) and LS5 (Neurosciences and tough international competition for these Neural Disorders). The LS7 panel grants, these and any other merits are (Diagnostic Tools, Therapies and Public simply likely to help—for example, it has Health) showed some of the highest values, been observed that ERC panels tend to possibly consistent with the fact that this life select applicants who have published high- science panel received the most impact articles (Robitaille et al., 2015). applications. The highest interquartile range Without splitting hairs about the values values were found in some of the shown in the figure, we could derive the consolidator panels (e.g., LS7 and LS4— general conclusion that young scientists Physiology, Pathophysiology and whose metrics are above first quartile Endocrinology), indicating a higher values should probably stop pondering and variability in grantees’ metrics. start applying. ERC MYTHS AND FACTS All things considered, this look into While little has been shared in the metrics confirms something that comes as literature, there is a great deal of expertise no surprise: the ERC is highly competitive on the ERC among European research and only for the best, with the 12–15% institutions. In the words of the late success rates for 2015 as a reminder. Yet, Professor Ilkka Hanski (ERC Advanced metrics could be of use in planning and grantee and former panel chair), an H-index rationalizing efforts when targeting approaching 10 could be considered a good research funding. The fairness of the ERC indication when applying for ERC Starting, process [aside from political considerations which seems in line with the values shown (The Guardian, 2014) or exasperation about above. With respect to the often claimed ever-changing deadlines] is widely “hidden” additional criteria, such as the acknowledged, especially with regard to need for prior international mobility, recognizing excellence. Not surprisingly, preliminary evidence reassuring feasibility, rankings of excellence nowadays include or a paper in Science, Nature or Cell (The counts of ERC grants, and an increasing Guardian, 2014), no evidence was found to number of organizations are re-modelling 3 Research Management Review, Volume 21, Number 1 (2016) their schemes to imitate the ERC, with even evaluation. Furthermore, the metrics some national programs sponsoring the best analyses only capture a part of the picture non-funded ERC applications (Nature Cell (e.g., overlooking merits such as awards or Biology, 2010). As institutional recruitment the scientist’s contributions to multi- practices follow suit, the issue at hand will authored papers). Whereas substantial continue to attract attention. differences in the material and methods A recent report commissioned by the used impeded a comparison of results with ERC indicated that scores attributed by the two related studies [those of Pecha on a evaluation committees match well with 2012 Starting grantees’ cohort (Pecha, 2014), applicant performance as measured by and of the MERCI project with 2007 and bibliometric indicators (Robitaille et al., 2009 Starting applicants (MERCI project, 2015, p. 69). Still, obvious caution should be n.d.)], the described analyses can be easily exercised in the interpretation of our values replicated with future ERC cohorts, and given the limitations of the metrics may be likewise relevant for other approach [see Science (2016) for a discussion competitive research funding schemes. on metrics for young scientists]. In the case When interpreting values, the golden rules presented here, only grantees’ profiles were of metrics also need to be remembered: use considered, with the excellence of the more than one metric to give insights into project idea criterion not accounted for. an issue, and support conclusions with While effort was put into cross-checking the expert analysis (Colledge & Verlinde, 2014). accuracy of the data (e.g., checking Research advisors, ERC National Contact researchers’ ORCID and their own websites Points, and particularly ERC-experienced when necessary), the publication numbers scientists will be great supports for the in Scopus may not exactly match the pondering scientist who, if still doubting, scientists’ record at the time of the should just be encouraged to apply. LITERATURE CITED Colledge, L., & Verlinde, R. (2014). SciVal metrics guidebook, Version 1.01. New York: Elsevier. https://www.elsevier.com/__data/assets/pdf_file/0020/53327/scival-metrics-guidebook- v1_01-february2014.pdf European Research Council. (2015). Ideas newsletter (December). https://erc.europa.eu/sites/default/files/publication/files/Newsletter_December_2015.pdf European Research Council. (2016). 2016 work programme (July). https://erc.europa.eu/sites/default/files/document/file/ERC_Work_Programme_2016.pdf 4 Research Management Review, Volume 21, Number 1 (2016) The Guardian. (2014). European research funding: It's like Robin Hood in reverse (7 November). http://www.theguardian.com/higher-education-network/2014/nov/07/european-research- funding-horizon-2020 MERCI project. (n.d.). Monitoring European Research Council’s implementation of excellence. Final Report. http://cordis.europa.eu/result/rcn/177023_en.html Nature Cell Biology. (2010. A maturing European Research Council (Editorial), 12, 307. http://www.nature.com/ncb/journal/v12/n4/full/ncb0410-307.html Pecha, O. (2014). Results of a bibliometric analysis of successful applicants for ERC starting grants in 2012. Prague: Technology Centre AS CR. Robitaille, J. P., Macaluso, B., Pollitt, A., Gunashekar, S., & Lariviè, V. (2015). Comparative scientometric assessment of the results of ERC-funded projects, bibliometric assessment report. Report prepared for the European Commission. https://erc.europa.eu/sites/default/files/document/file/ERC_Bibliometrics_report.pdf Science. (2016). Measure of success (Letters, NextGen Voices). 28, 352. http://science.sciencemag.org/content/suppl/2013/10/03/342.6154.36.DC1 ABOUT THE AUTHOR Javier Arevalo is a Senior Advisor in Research Funding at the University of Helsinki. After dedicating the last two years to advising life science researchers on international funding applications, he is now part of a newly formed team supporting strategic research projects. Prior to his work as research advisor, he worked for 7 years as a researcher and lecturer at the School of Forest Sciences, University of Eastern Finland. This included research and project work in countries such as Brazil, China, Kenya, Russia and Burkina Faso. He holds a Ph.D. in Forestry and Agriculture Sciences from the University of Eastern Finland, and a M.Sc. also in Forestry from the Technical University of Madrid (Spain). 5