Understanding the Capacity of NIH’s Peer Review System

With the complete elimination of any real cap on the number of allowable resubmissions to NIH, many of us have wondered how much the workload has increased at CSR and on NIH reviewers in general. In the past two years, outgoing NIH Deputy Director has been evaluating “the level of service that most peer reviewers are willing and able to provide, and how peer review service fits within the scope of reviewers’ other professional responsibilities.”

Among the key results of this evaluation:

  • More than 80% of mid-career R01 recipients have served as reviewers at least once in the past five years.
  • 88% of respondents who reported having been invited to review in the prior year had served at least once.
  • ~51% of respondents reported that peer review of grants should comprise less than 5% of their professional effort, but another 46% reported that peer review of grants should make up 5-10% of their worktime.
  • Respondents reported that they considered an assignment load of 6 applications per meeting, and 1 – 2 meetings per year, to be reasonable expectations. The typical load at CSR is more than this, and NIH would be hard pressed to review all the applications the scientific community submits if this preference became the norm.
  • ~3,500 qualified reviewers/year have not yet served in the last five years.

An article in the newly released CSR Peer Review Notes describes the information in more detail: http://public.csr.nih.gov/aboutcsr/NewsAndPublications/PeerReviewNotes/Pages/Peer-Review-Notes-Sep-2015Part5.aspx

The End of Two Strikes You’re Out: Good or Bad News for Grant Applicants?

Yesterday, NIH and AHRQ announced that they had changed its resubmission policy. While a given application is still allowed only one resubmission, if you are unsuccessful on the A1, you can then submit the application again as an A0 without having to substantially redesign the content and scope of the project. This “new submission” will be reviewed without any association to the previous attempts. There will be no Introduction to the Revised Application, no explanation of how you addressed the reviewers’ concerns. Even if reviewers have seen the application in prior review cycles, they will be instructed to review it as new. (However, reviewers are human, and one wonders if this is realistic. It’s like asking a juror to ignore evidence they just heard, and we know from social psychologists that that doesn’t actually work.) Of course, the idea is that an applicant will use previous reviewer comments to strengthen the application, thereby improving their odds of funding.

Click here to read the full notice

In 2009, as part of their Enhancing Peer Review project, NIH eliminated the A2 in what has been dubbed by unhappy researchers as the “Two Strikes You’re Out” policy. NIH eliminated the A2 because meritorious research was most likely funded on the A2, which meant quite a delay to funding. The resubmission policy did indeed result in an increase in the number of awards made on A0 applications (although most funded applications are successful on the A1). About the “Two Strikes You’re Out” policy, Deputy Director Dr. Sally Rockey states: “…we heard increasing concerns from the community about the impact of the policy on new investigators because finding new research directions can be quite difficult during this phase of their career. Also, established investigators voiced concern about the need to redirect the research focus of productive labs in order to submit future NIH applications.” For one of many discussions on Dr. Rockey’s blog about the decision to sunset the A2 submission, click here.

In theory, it would appear that one could submit the same idea endlessly until one finds the best way to sell it to reviewers. The policy will no doubt appeal to many researchers, because it is now up to a researcher to decide when it is time to abandon a given project. If one waits long enough, one could submit to the same study section after most of the members have turned over, thereby having a new set of reviewers to weigh in on the project. One risk I see is that sometimes a PI is so blindly enamored with their idea that they have difficulty hearing that the reviewers are trying to tell them that the idea is simply not fundable in any form; i.e., no amount of tinkering with the writing or the details will fix it. A PI could waste a lot of time if they cannot see when it is time to abandon an unfundable project.

Judging from the comments on some of the NIH blogs (for examples, click here), many reviewers are thrilled to have more submission attempts, while others are dismayed that this marks a return to a huge number of submissions and long delays to funding. What do you think?

2013 Year In Review at NIH

Here is a link to a useful blog post written by NH Deputy Director Sally Rockey. It summarizes some of the main activities at NIH during the course of 2013. While it begins with a depressing recap of the far-reaching effects of the budget situation, it goes on to highlight some of the main goals and programs of the year. Major themes continue to include data science and efforts to diversify the scientific workforce. The blog is chock-full of hyperlinks to more information on numerous topics. If you plan to develop a long-term relationship with a federal funding agency, it is important to know its mission and funding priorities, and to familiarize yourself each year with the goals identified in the director’s appropriation report, and the suggestions made by the advisory council to the director. Read the full blog post here.