The Importance of the “Approach” Criterion On NIH Grant Scores

Sally Rockey, Deputy Director of Extramural Research at NIH, posted data on her blog concerning reviewer behavior on proposals reviewed in 2010. Recall that with the new NIH grant format, NIH implemented scores of one to five (lower is better) in each of five categories: Significance, Innovation, Approach, Investigators, and Environment, as well as the overall impact score. Of 54,727 applications reviewed in FY10, 32,546 were discussed and received overall impact scores. The NIH generated some data on these scores. Among their findings:

*While reviewers used the full scoring range (one through five) for each of the five review categories, their scores were distributed more widely for the Approach category.

*Criterion scores are moderately correlated with each other and with the overall impact score.

*Of the five scoring categories, the one best able to predict the overall impact score was the Approach section (followed by significance, innovation, investigators, environment.)

The language surrounding the changes to the NIH grant format in Zerhouni’s efforts to enhance peer review stressed the importance of Impact, Significance, and Innovation. And in the new format, the length of the Research Strategies was halved, forcing grantees to compress “Approach” sections such as the exhaustive literature review and the detailed methodologies. However, from the data in Rockey’s blog post, we might surmise that reviewers still heavily weigh the Approach category. And based on my own experience with pink sheets in the new format, reviewers’ nearly-insatiable desire for preliminary data appears to continue unabated, despite the reduced page limits on proposals.

Multiple Regression To Predict Impact Scores Using Criterion Scores

Criterion Regression Weight
Approach

6.7

Significance

3.3

Innovation

1.4

Investigator

1.3

Environment

-0.1

NIH “Two Strikes You’re Out” Rule, Two Years Later

Once upon a time, one was allowed three submissions on an NIH grant application—your initial submission (A0), and two resubmissions after review (A1 and A2). In January 2009 NIH did away with the second resubmission, giving rise to what the scientific community not-so-fondly dubbed the “Two Strikes You’re Out” rule. This policy was instituted under the leadership of previous NIH Director Elias Zerhouni as part of the effort to Enhance Peer Review. The goal was to fund the best science in the shortest time period possible.

So two years out, how successful has this policy been? Today NIH released a letter it composed in response to one it received from members of the scientific community. Before the policy went into effect, funding for A0s had dropped dramatically, despite the fact that applications that initially scored in the top 20% were eventually funded (they went on to be funded as A1s and A2s after having to wait 1-2 years.) This trend appears to have reversed: The proportion of funded A1 applications has remained constant while the proportion of funded A0s has increased dramatically. The NIH letter goes on to supply data demonstrating that the policy has not had a negative effect on R01 funding for New and Early Stage Investigators. Read more, and if you haven’t already subscribed to NIH Deputy Director Sally Rockey’s blog “Rock Talk”, I highly recommend you do so.