Since joining the editorial board of Optical Engineering several years ago, I have often been drawn into various discussions about journal impact factor. In many of the conversations, impact factor was an assumed standard of performance for journals, and the content of the discussion focused on various decisions it impacts. These include deciding whether to submit to a journal based on its impact factor, assessing a scientist’s or engineer’s performance based on the impact factors of journals in which they have published, and considering what actions to take in order to increase a journal’s impact factor.

More recently, much of the discussion appears to have shifted to questioning the efficacy of the metric and appropriateness of its widespread use. For example, recent editorials in Nature¹ and other journals use adjectives such as crude, misleading, and invidious to describe the metric and its effects on science. The American Society of Microbiology recently decided to discontinue advertising the impact factors of its journals in response to the controversy.² And the co-creator of the metric, Eugene Garfield, is said to have drawn a parallel with the unintended consequences of his creation to that of nuclear technology. It is interesting how this deceptively simple number has stirred up such widespread controversy.

At face value, journal impact factor is a straightforward metric: the average number of citations per published paper over a two-year time period. The underlying assumption is that impact on a scientific field is appropriately measured by number of citations. Recent criticisms about impact factor raise a number of different issues. Use of the arithmetic mean is questioned as a statistically poor measure as it can be sensitive to outliers, which in this case means a few very highly cited papers. Proponents recommend adoption of a median or citation distribution as more statistically sound measures. Others assert that impact factor is a poor relative measure between journals in different scientific fields, particularly favoring ones with high current interest and undervaluing ones lacking the same level of popularity, even as contributions in the latter fields may ultimately yield higher and more lasting scientific impact. There have also been assertions of manipulation of impact factors by journals pressuring their authors to cite their own journals. And finally, the increasingly common use of impact factor in evaluating individual performance is called into question as misuse of the metric.

In my opinion these are all legitimate criticisms. As I analyzed recent citation data for Optical Engineering papers to understand what factors drive impactful papers, it was apparent to me that the data were not well behaved, making simple statistical measures potentially misleading. Especially for an engineering-oriented journal such as ours, I question the inherent assumption that citations appropriately measure the scientific, commercial, or intellectual impact of journal papers. Analytical and experimental methods are often adopted from published papers and put into practice by engineers whose output is not journal publications but ultimately a new design, test methodology, production process, or other engineering product. This type of impact is extremely important for Optical Engineering, addressing a core industry-oriented constituency of SPIE. Impacts such as these are not at all captured by the impact factor metric.

Impact factor falls well short in terms of serving as a singular measure of journal quality. I joined the editorial board for Optical Engineering for many reasons, including its long history in serving the optical engineering community, rigorous manuscript selection process, strong reputation of its editorial board members, tight connection to outstanding SPIE conferences, established readership, and excellent print quality. I also consider these to be important factors in the determination process for manuscript submission. Again, none are directly captured in impact factor.

Despite all these misgivings, I find citation rates to be an important—albeit not singularly important—metric for gauging journal quality. Unfortunately, there just are not many other options for quantitative assessment. As I stated in my initial editorial, one of my primary goals is to continue to raise the quality of the journal. Since then, I have been experimenting with several approaches to do just that, some of which I have discussed in intervening editorials. But how do I know whether anything is working? For that I need metrics to provide a feedback mechanism. From a quantitative perspective, I am largely left with paper downloads and citations as a gauge for interest and impact, respectively, as most of the other attributes are somewhat intangible or not readily measured.

Recognizing the practical constraints of any quantitative measure of journal impact, I am ultimately not as severe in my criticism of the metric. With all due respect to Garfield’s nuclear analogy, I would like to offer one that I consider more a propos. We all have experienced somewhere in our academic careers expending many ounces of blood, sweat, and tears in a very challenging class with the end result ultimately captured by a single letter on our academic transcripts. Did our professors really feel that the letter grade was a complete statistic fully representing our mastery of the class material with all its complexities? And do we feel comfortable with future admissions officers, employers, and others making decisions on our career opportunities based solely on grade point average? No, we hope that they give consideration to a broader set of factors reflecting the full extent of our knowledge and capabilities, even as we were compelled to agree that grades do provide a useful measure.

In my opinion, the shortcomings of journal impact factor are similar to those of grade point average. It is a useful measure, and unfortunately one of the few that we have at our disposal. If we over-generalize its efficacy for the sake of simplicity, however, we are selling ourselves short. As scientists and engineers, we should appreciate that, since appropriately dealing with complexities of the real world is what our profession is about.