Understanding the Capacity of NIH’s Peer Review System

With the complete elimination of any real cap on the number of allowable resubmissions to NIH, many of us have wondered how much the workload has increased at CSR and on NIH reviewers in general. In the past two years, outgoing NIH Deputy Director has been evaluating “the level of service that most peer reviewers are willing and able to provide, and how peer review service fits within the scope of reviewers’ other professional responsibilities.”

Among the key results of this evaluation:

  • More than 80% of mid-career R01 recipients have served as reviewers at least once in the past five years.
  • 88% of respondents who reported having been invited to review in the prior year had served at least once.
  • ~51% of respondents reported that peer review of grants should comprise less than 5% of their professional effort, but another 46% reported that peer review of grants should make up 5-10% of their worktime.
  • Respondents reported that they considered an assignment load of 6 applications per meeting, and 1 – 2 meetings per year, to be reasonable expectations. The typical load at CSR is more than this, and NIH would be hard pressed to review all the applications the scientific community submits if this preference became the norm.
  • ~3,500 qualified reviewers/year have not yet served in the last five years.

An article in the newly released CSR Peer Review Notes describes the information in more detail: http://public.csr.nih.gov/aboutcsr/NewsAndPublications/PeerReviewNotes/Pages/Peer-Review-Notes-Sep-2015Part5.aspx

Top Ten Things NIH Reviewers Should NOT Say In A Review

Credit: Ambro at FreeDigitalPhotos.net

The Center for Scientific Review publishes their Peer Review Notes three times a year, and the most recent issue came out yesterday. The news items are always interesting and it is worth subscribing, if you don’t already. This issue contained an item about things NIH reviewers should not say. I repeat the list in its entirety here—I thought it might be fun for my grantees to see reviewers critiqued for a change.

What do you think of this list? Have you seen one or two of these on your Summary Statements? Me personally? I have seen variations on # 2, 4, and 10 in Summary Statements, and have strongly suspected reviewers of #1 and 5. I almost fell out of my chair laughing when I read # 7, sometimes I think CSR is a little out of touch with what actually happens on Study Sections:

  1. “I didn’t read the application, but I scanned it and saw the applicant said XXX. He doesn’t know what he’s doing.” Damning statements like this can skew a review discussion over something that might be insignificant in the context of the overall application. It’s better for you to ask other reviewers who have read the application carefully what they think about XXX.
  2. “This New Investigator does not appear to be fully independent since he continues to co-publish with his fellowship mentor/department chair, or does not have designated lab space, or has not been promoted in the past several years.”  Academic research organizations have widely diverse policies for faculty advancements and lab space, and many PIs maintain productive and healthy collaborations with mentors for many years after establishing themselves as bona fide investigators. You should focus more on the investigator accomplishments, such as being the first or senior author on a significant publication or giving presentations at major scientific meetings.
  3. “This application is not in my area of expertise . . . “  If you’re assigned an application you feel uncomfortable reviewing, you should tell your Scientific Review Officer as soon as possible before the meeting.
  4. “I don’t see this basic science research affecting my clinical practice any time soon.” An application does not necessarily have to show the potential for clinical or timely impact—if the applicant doesn’t make such claims. Basic research often takes time to pay off, and you’re charged to assess the “likelihood for the project to exert a sustained, powerful influence on the research field(s) involved.” Absence of an effect on public health does not necessarily constitute a weakness in basic science.
  5. “I like this project but I’m giving it a poorer score because the applicant has too much money.” Other funding is not a scoreable matter. You should focus on the application’s scientific and technical merit. However, you can note an excessive budget request in the budget section for NIH to consider.
  6. “This application has 2 great aims and 1 bad one. I would recommend deleting Aim 3, and I can give it a 1 or 2.” You cannot trade aims with scores. The application needs to be evaluated as a whole.
  7. “This R21 application does not have pilot data, which should be provided to ensure the success of the project.” R21s are exploratory projects to collect pilot data. Preliminary data are not required, although they can be evaluated if provided.
  8. “The human subject protection section does not spell out the specifics, but they already got the IRB approval, and therefore, it is ok.” IRB approval is not required at this stage, and it should not be considered to replace evaluation of the protection plans.
  9. “This application was scored a 25 and 14th percentile last time it was reviewed . . . .” You should not mention the previous score an application got, because this could skew the review discussion. Focus on the strengths and weaknesses of the current application as well as the responses to previous critiques.
  10. “This is a fishing expedition.” It would be better if you said the research plan is exploratory in nature, which may be a great thing to do if there are compelling reasons to explore a specific area. Well-designed exploratory or discovery research can provide a wealth of knowledge.

Grantwriting Workshops Offered by Meg Bouvier Medical Writing

On May 8-9, The Arizona Biomedical Research Commission (ABRC) will be hosting a series of workshops on NIH grant submissions, at which I will be the featured presenter. For details and registration information, click here.

Workshops are often a cost-effective way to educate a larger group of faculty on the NIH grant process. In Phoenix next week, I will be kicking off my presentations with a popular 3 ½ hour R01 workshop, which includes a workbook that contains exercises and samples of funded grant applications. After, I will be conducting a series of one-hour breakout groups on topics including NIH submission strategies, resubmissions, mistakes commonly made by applicants, the review process, and how to choose an appropriate funding mechanism (R01, R21, or R03). Each time I present to a group, I work with the client to customize the presentations to address the needs of a particular group of attendees.

The workshops have proved quite popular with departments and institutions and can be taken for CME credit. I draw upon my experience working each year with dozens of NIH submissions and summary statements. My experience as both a bench scientist and staff writer at NIH also informs my approach to NIH grantsmanship and trainings.

Please contact us to discuss a workshop that will fit your needs and budget, and for a sampling of workshop formats and topics.