Will NIH Eliminate The Unpopular “Two Strikes You’re Out” Grant Review Policy?

In January 2009, NIH instituted a policy to sunset A2 applications, i.e. to eliminate a second resubmission. Since, investigators submit a grant application (A0) and if the application is unfunded they may submit one revised application (A1), and no more (previously, one could resubmit twice). This policy has been detested among research scientists since its inception and NIH has heard an earful of complaints about it. NIH spoke out recently on this topic. Will they consider eliminating the wildly unpopular policy? The short answer: Not bloody likely.

In a recent blog post, Sally Rockey gives some data NIH has compiled on this policy. One of the goals of the policy was to reduce the time to award, and in fact the time from A0 submission to award has been reduced from 93 weeks to 56 weeks by this policy. It is conventional wisdom that NIH grant applications, if funded, tend to be successful on their final submission, with very few succeeding as an A0. Yet surprisingly, since the policy went into effect, the proportion of funded A0 applications as compared to A1 apps has increased.

One major complaint of the policy is that it favors established investigators over new investigators. The data do not appear to support this idea, because the time-to-award for new applicants is not appreciably longer than for the entire population.

In response to the suggestion that NIH allow A2 applications only from those investigators whose apps fall just outside the payline, NIH modeled the potential impact of such a policy using FY11 data. From their model, NIH concluded that the result would be to shift awards from A0 and A1 apps toward A2 apps, which was what the policy was designed to avoid to begin with (see original blog post for details of their method.)

Rockey concludes: “Overall, these data indicate that the policy to sunset A2 applications continues to achieve the stated goals of enabling NIH to fund as much meritorious science as possible in as short a time period as possible. Any revision to the policy to allow additional resubmissions of all or a subset of A2 applications will displace equally meritorious A0 and A1 applications, and increase the time to award for many applications. For these reasons, we have decided to continue the policy in its current form.”

Sorry folks, it’s here to stay.

How The Continuing Resolution Affects The NIH Budget– And Your Grant Award

The NIH issued this announcement yesterday:

“The Department of Health and Human Services (HHS), including NIH, operates under a Continuing Resolution (CR) (H. J. Resolution 117) that was signed by President Obama as Public Law 112-175 on September 28, 2012.  The CR continues government operations through March 27, 2013 at the FY 2012 level plus 0.6 percent.

“Until FY 2013 appropriations are enacted, NIH will issue non-competing research grant awards at a level below that indicated on the most recent Notice of Award (generally up to 90% of the previously committed level). This is consistent with our practice during the CRs of FY 2006 – 2012. Upward adjustments to awarded levels will be considered after our FY 2013 appropriations are enacted but NIH expects institutions to monitor their expenditures carefully during this period.  All legislative mandates that were in effect in FY 2012 remain in effect under the CR, including the salary limitation set at Executive Level II of the Federal Pay Scale ($179,700), which was effective with grant awards with an initial Issue Date on or after December 23, 2011 (see NOT-OD-12-034 and NOT-OD-12-035).”

For grant applications that have just been reviewed, look for a delay (possibly lengthy) in funding decision pending the FY13 Appropriation (unless you are lucky enough to have a priority score well within the funding range.) For those in the gray zone (perhaps 7-16%, depending on your funding mechanism and your ESI status), you can expect a lengthy delay in the funding decision. Discuss your specific circumstances with your program officer.

The Importance of the “Approach” Criterion On NIH Grant Scores

Sally Rockey, Deputy Director of Extramural Research at NIH, posted data on her blog concerning reviewer behavior on proposals reviewed in 2010. Recall that with the new NIH grant format, NIH implemented scores of one to five (lower is better) in each of five categories: Significance, Innovation, Approach, Investigators, and Environment, as well as the overall impact score. Of 54,727 applications reviewed in FY10, 32,546 were discussed and received overall impact scores. The NIH generated some data on these scores. Among their findings:

*While reviewers used the full scoring range (one through five) for each of the five review categories, their scores were distributed more widely for the Approach category.

*Criterion scores are moderately correlated with each other and with the overall impact score.

*Of the five scoring categories, the one best able to predict the overall impact score was the Approach section (followed by significance, innovation, investigators, environment.)

The language surrounding the changes to the NIH grant format in Zerhouni’s efforts to enhance peer review stressed the importance of Impact, Significance, and Innovation. And in the new format, the length of the Research Strategies was halved, forcing grantees to compress “Approach” sections such as the exhaustive literature review and the detailed methodologies. However, from the data in Rockey’s blog post, we might surmise that reviewers still heavily weigh the Approach category. And based on my own experience with pink sheets in the new format, reviewers’ nearly-insatiable desire for preliminary data appears to continue unabated, despite the reduced page limits on proposals.

Multiple Regression To Predict Impact Scores Using Criterion Scores

Criterion Regression Weight
Approach

6.7

Significance

3.3

Innovation

1.4

Investigator

1.3

Environment

-0.1