The future h‐index is an excellent way to predict scientistsˈ future impact

Typically, when scientists are being considered for funding, appointment, or promotion, a committee reviews their publications and often estimates the importance of these using a metric such as the h-index.1 This just represents past accomplishments, however, and not potential future impact, which is probably more important. A new metric, the future h-index,2 has been introduced and is claimed to be an excellent way to predict scientistsˈ future impact. This is the premise debated in this monthˈs Point/Counterpoint. Hiring, promotion, and funding decisions are to a large extent driven by predictions about scientistsˈ future impact. A scientistˈs future impact can be defined with respect to many different objectives, such as research publications, funding, teaching ability, and outreach.3 Publications and their citations relate to a scientists ability to do research and are of great importance to decision makers in research focused institutions. This discussion focuses on a specific kind of impact estimation, the estimation of future publication impact. The excellence of a prediction is defined by the quality of the decision that it supports. Indices like the future h-index are noisy: some aspects of excellence are not visible in the publication record. They may also be biased: some aspects of excellence, e.g., interdisciplinary training, may not be observable. However, estimates of committees are also noisy. For example, members may have highly individual favoritisms. They may be discriminatory: gender, race, and other types of prejudices may cloud judgments.4 How useful each source of information is depends on the sizes of biases and noises, and these problems can be solved by predicting the future h-index.2 The h-index is one of the simplest and easiest metrics of publication impact,1 making it attractive to use. It is robust and hard to manipulate.5 However, the h-index does not account for the age of scientists and individual contributions of each co-author, and self-citation introduces biases in favor of authors with many co-authors.6 Also, it varies substantially from field-to-field, making it hard to compare researchers across disciplines.4 There are alternatives that fix these problems (e.g., Aziz and Rozing6) but they usually require access to more detailed information, making them impractical. However, both noise sources and biases can be addressed by statistics, and the future h-index is one such development. Committees of peers and experts are the current standard for predicting future scientific success. These committees increasingly use metrics,1 necessary because of the growing number of applicants per tenure-track job or funding opportunity.7 Why should we not try to find the statistically optimal combination of multiple features to achieve better predictions? For example, we could have an index that combines many indices such as funding, teaching, and other metrics from social media (e.g., “Altmetrics”8) and maybe try to predict them in an optimal manner, similar to Acuna et al.2 True excellence in future predictions comes from describing weaknesses of approaches, and then statistically finding ways of correcting them. The future h-index is just one small step in this direction. This proposition can be parsed into three questions. First, assuming 100% prediction accuracy, is the future h-index a good measure of a scientistˈs future impact? Second, how accurate is the future h-index model proposed by Acuna et al.?2 Third, how should its likely use as an evaluation tool shape the criteria we use to judge future h-index? The first question appears to challenge the h-index as a measure of research impact, but it does not necessarily do so. What it does challenge is the logic of associating an increase in the h-index between years t and t + Δt with research impact during those years. That increase is driven not only by citations to papers published between t and t + Δt but also citations to papers published before t. In fact there is a good evidence that the increase is largely driven by citations to previous work.9 Indeed, even though the mathematician Paul Erdős passed away in 1996, his h-index increased by 9 between 2001 and 2010, one of the largest increases of any mathematician over that period. Turning to the technical aspects of predicting the future h-index, the model of Acuna et al. is a good starting point but suffers a number of shortcomings. Collaborators and I discovered its predictive power depends heavily upon “career age.”10 In forthcoming work we further demonstrate much of the modelˈs “predictive power” arises from the cumulative and increasing character of the h-index itself.11 These challenges will likely be overcome by improving the model, but it is hard to overlook the fact that a sufficiently powerful model does not yet exist. The third question is the one most often overlooked. To most practicing scientists these models and measures will be curiosities, but to people making decisions on tenure track hires, fellowships, institutional and national tenure, etc. they will be tools.12,13 The true measure of the future h-index comes down to its suitability as a decision making tool. In that context a model must do more than simply fit the data, i.e., produce a high R2, it must also produce accurate rankings. Further, for late career awards (e.g., election to the National Academy of Sciences) it is desirable to measure a scientistˈs future impact based on papers published before and after year t. But in the case of a tenure track hire it is critical to identify the scientist whose work after t will have the greatest impact. Again, the future h-index cannot discriminate between the future impact of previously published papers and the future impact of work yet to be published. The future h-index is an excellent contribution to the scientific community and pushes the discourse in an important direction, but a great deal of technical and conceptual refinement is required before it is an excellent way to predict scientistsˈ future impact. It seems that there is a deep level of agreement about the central issues that Dr. Penner so eloquently put into the three central questions. The h-index is not perfect at measuring research productivity and it would be better to predict something more meaningful for decision makers—once such a measure has been properly defined. In fact, it would be useful to better understand the aims of the various players in the academic market. Methods related to utility elicitation14 could help by allowing a scientific approach for measuring what we should predict. The features that we used in the formulation of the future h-index were based only on the publication record, which is not overly indicative early on in the career.10 Early career predictions could be significantly improved by adding other features such as courses taken, grades obtained, and text analysis of letters of recommendation. Since we did not have access to such information, we focused on the publication record only—which seems to work relatively well a few years into the scientific career, even in other scientific domains.15 Our approach and feature set is just a first step on the path of providing useful predictions of impact. Predictions are ultimately used for decision-making, rendering rank predictions or other model assessments more appropriate than simply demonstrating a high R2. For example, granting tenure may be about future publication success, funding, and teaching abilities, whereas election to the National Academy of Sciences may be about career-length impact. Data-driven predictive tools offer an opportunity to impartially help during this process. The way we see our own approach is a starting point that can be improved by both better formulating what matters and by better predicting the variables that matter. Finally, quantitative approaches need to be compared to the alternatives. Human experts are known to be biased in many ways and, just like any algorithm, might not optimize the variables that truly matter for the decision. A deeper analysis about how to improve these approaches would be an exciting topic for a future discussion. Indeed I agree with Dr. Acuna that the future h-index stands as a good first step towards the use of quantitative approaches in hiring and advancement decisions in academia. However, there is still a great deal more careful and rigorous work to be done before a mature suite of tools is devised. In my opening I highlighted several key flaws that must be addressed, but there are other factors that will play a role in determining whether or not quantitative approaches will be widely accepted. As Dr. Acuna points out, a good framework must be able to integrate information on many other facets of the academic career, including funding, teaching, communication and leadership skills. Any successful approach must also be flexible in how it integrates these data, such as being able to handle input that plays a nonlinear role in determining future impact. Perhaps most importantly it is critical that any framework explicitly produces easily interpreted confidence bounds for its predictions, clearly indicating to the user when it is being stretched beyond its “comfort zone.” Satisfying these criteria in one modeling framework is a challenge to be certain, but it does not seem that any one piece is impossible. If one huge challenge does lie on the path to an appropriate modeling framework, I speculate it is not a matter of the model at all but rather the availability of highly accurate career data for a huge number of scientists. Indeed, few quantitative studies of full and complete career trajectories have surpassed 5000 careers. Development of a suitably flexible and accurate prediction framework will require data sets that stretch, at the very least, into the hundreds of thousands and be spread across all academia. With such diverse data it may be possible to develop a model of academic careers capable of being fully validated, but at the moment the lack of such data represents a massive road block. Dr. Acuna thanks Dr. Konrad Kording for helpful discussions.