Understanding the Capacity of NIH’s Peer Review System

With the complete elimination of any real cap on the number of allowable resubmissions to NIH, many of us have wondered how much the workload has increased at CSR and on NIH reviewers in general. In the past two years, outgoing NIH Deputy Director has been evaluating “the level of service that most peer reviewers are willing and able to provide, and how peer review service fits within the scope of reviewers’ other professional responsibilities.”

Among the key results of this evaluation:

  • More than 80% of mid-career R01 recipients have served as reviewers at least once in the past five years.
  • 88% of respondents who reported having been invited to review in the prior year had served at least once.
  • ~51% of respondents reported that peer review of grants should comprise less than 5% of their professional effort, but another 46% reported that peer review of grants should make up 5-10% of their worktime.
  • Respondents reported that they considered an assignment load of 6 applications per meeting, and 1 – 2 meetings per year, to be reasonable expectations. The typical load at CSR is more than this, and NIH would be hard pressed to review all the applications the scientific community submits if this preference became the norm.
  • ~3,500 qualified reviewers/year have not yet served in the last five years.

An article in the newly released CSR Peer Review Notes describes the information in more detail: http://public.csr.nih.gov/aboutcsr/NewsAndPublications/PeerReviewNotes/Pages/Peer-Review-Notes-Sep-2015Part5.aspx

The Importance of the “Approach” Criterion On NIH Grant Scores

Sally Rockey, Deputy Director of Extramural Research at NIH, posted data on her blog concerning reviewer behavior on proposals reviewed in 2010. Recall that with the new NIH grant format, NIH implemented scores of one to five (lower is better) in each of five categories: Significance, Innovation, Approach, Investigators, and Environment, as well as the overall impact score. Of 54,727 applications reviewed in FY10, 32,546 were discussed and received overall impact scores. The NIH generated some data on these scores. Among their findings:

*While reviewers used the full scoring range (one through five) for each of the five review categories, their scores were distributed more widely for the Approach category.

*Criterion scores are moderately correlated with each other and with the overall impact score.

*Of the five scoring categories, the one best able to predict the overall impact score was the Approach section (followed by significance, innovation, investigators, environment.)

The language surrounding the changes to the NIH grant format in Zerhouni’s efforts to enhance peer review stressed the importance of Impact, Significance, and Innovation. And in the new format, the length of the Research Strategies was halved, forcing grantees to compress “Approach” sections such as the exhaustive literature review and the detailed methodologies. However, from the data in Rockey’s blog post, we might surmise that reviewers still heavily weigh the Approach category. And based on my own experience with pink sheets in the new format, reviewers’ nearly-insatiable desire for preliminary data appears to continue unabated, despite the reduced page limits on proposals.

Multiple Regression To Predict Impact Scores Using Criterion Scores

Criterion Regression Weight