Top Ten Things NIH Reviewers Should NOT Say In A Review

Credit: Ambro at FreeDigitalPhotos.net

The Center for Scientific Review publishes their Peer Review Notes three times a year, and the most recent issue came out yesterday. The news items are always interesting and it is worth subscribing, if you don’t already. This issue contained an item about things NIH reviewers should not say. I repeat the list in its entirety here—I thought it might be fun for my grantees to see reviewers critiqued for a change.

What do you think of this list? Have you seen one or two of these on your Summary Statements? Me personally? I have seen variations on # 2, 4, and 10 in Summary Statements, and have strongly suspected reviewers of #1 and 5. I almost fell out of my chair laughing when I read # 7, sometimes I think CSR is a little out of touch with what actually happens on Study Sections:

  1. “I didn’t read the application, but I scanned it and saw the applicant said XXX. He doesn’t know what he’s doing.” Damning statements like this can skew a review discussion over something that might be insignificant in the context of the overall application. It’s better for you to ask other reviewers who have read the application carefully what they think about XXX.
  2. “This New Investigator does not appear to be fully independent since he continues to co-publish with his fellowship mentor/department chair, or does not have designated lab space, or has not been promoted in the past several years.”  Academic research organizations have widely diverse policies for faculty advancements and lab space, and many PIs maintain productive and healthy collaborations with mentors for many years after establishing themselves as bona fide investigators. You should focus more on the investigator accomplishments, such as being the first or senior author on a significant publication or giving presentations at major scientific meetings.
  3. “This application is not in my area of expertise . . . “  If you’re assigned an application you feel uncomfortable reviewing, you should tell your Scientific Review Officer as soon as possible before the meeting.
  4. “I don’t see this basic science research affecting my clinical practice any time soon.” An application does not necessarily have to show the potential for clinical or timely impact—if the applicant doesn’t make such claims. Basic research often takes time to pay off, and you’re charged to assess the “likelihood for the project to exert a sustained, powerful influence on the research field(s) involved.” Absence of an effect on public health does not necessarily constitute a weakness in basic science.
  5. “I like this project but I’m giving it a poorer score because the applicant has too much money.” Other funding is not a scoreable matter. You should focus on the application’s scientific and technical merit. However, you can note an excessive budget request in the budget section for NIH to consider.
  6. “This application has 2 great aims and 1 bad one. I would recommend deleting Aim 3, and I can give it a 1 or 2.” You cannot trade aims with scores. The application needs to be evaluated as a whole.
  7. “This R21 application does not have pilot data, which should be provided to ensure the success of the project.” R21s are exploratory projects to collect pilot data. Preliminary data are not required, although they can be evaluated if provided.
  8. “The human subject protection section does not spell out the specifics, but they already got the IRB approval, and therefore, it is ok.” IRB approval is not required at this stage, and it should not be considered to replace evaluation of the protection plans.
  9. “This application was scored a 25 and 14th percentile last time it was reviewed . . . .” You should not mention the previous score an application got, because this could skew the review discussion. Focus on the strengths and weaknesses of the current application as well as the responses to previous critiques.
  10. “This is a fishing expedition.” It would be better if you said the research plan is exploratory in nature, which may be a great thing to do if there are compelling reasons to explore a specific area. Well-designed exploratory or discovery research can provide a wealth of knowledge.

Implications for Grantees of no more “Two Strikes You’re Out”

Credit: adamr at FreeDigitalPhotos.net

Ding, dong the evil resubmission policy is dead. What are the implications in terms of your grantsmanship strategy and writing?

The original intent of the ill-fated, much-hated “Two Strikes You’re Out” policy was to reduce the time from first submission to award. According to the statistics and metrics put out by NIH, it was achieving that goal. With the policy change that went into effect last month, grantees still submit an A0 (new application) and an A1 (first resubmission), with no A2 (second resubmission) available. But if they fail at both the A0 and A1, they no longer need to shelve the application forever. They can now submit the application again as an A0 to any IC or study section they choose, and the reviewers will be given no information about earlier versions or summary statements. I have yet to find a grantee who is unhappy about the change of policy. ESI and New Investigator grantees in particular have expressed relief that they will have multiple attempts to “get it right” (the incorrect assumption being that study sections are static entities and that each review will be similar to previous reviews in terms of expectations and criticisms.)

Yet, it seems to me that this resubmission policy will lead to even longer lengths of time from first submission to award than were seen when people were allowed three submissions total. In essence, the very problem that “Two Strikes You’re Out” sought to fix will be made worse than it was prior to the “Two Strikes” policy. Here is my tongue-in-cheek mathematical representation of the new resubmission policy:

(A0, A1) ∞

To my mind, this policy equates not only to longer times to award than ever before, but also to an unprecedented number of NIH submissions going forward. In recent years, the total aggregate number of R01 submissions has been fairly steady at about 28 or 29 thousand per year. I expect we will see a dramatic increase in that number in FY14 and into FY15, as all those shelved, failed submissions go back into circulation. At some point that number will decrease a bit to a steady state that is still significantly higher than 29 thousand applications per year.

Love or hate the new policy (and while I recognize that “Two Strikes” was flawed, I am firmly in the latter camp), how should you fold this information into your approach to grantsmanship?

1. Consider deferring your June 5 A0 to the Oct deadline. I suspect that everyone and his brother will be dusting off an old, unfunded application and submitting an A0 for June 5. I suspect that that will equate to a gigantic number of submissions for this cycle, and my suspicions appear to be supported by the sheer volume of A0s on which I have been contracted to work. (I haven’t seen a crushing number of applications like this since the stimulus money.)

2. Give yourself more time to upload to the grants.gov portal with any R01 deadline going forward, but particularly for the June 5 deadline. As you know, applications must be uploaded and time stamped. When traffic at the portal is heavy, there are delays in your ability to upload. Many grantees have told me they missed a deadline because they waited too long and could not upload due to traffic at the portal, or not enough time to correct error messages received. This problem will be exacerbated by the increased application volume from now on.

3. Format your application with greater care than ever. A sharp increase in submissions will likely mean that reviewers will be even more overworked than ever, and the increased time/cost of reviewing all these extra applications may mean that CSR will step up its experiments with virtual peer review. How should this affect your writing? Use lots of unique identifiers throughout the application (numbering system with multiple subheadings) to help orient reviewers who are exhausted and/or not meeting face to face while discussing your application. Make judicious use of formatting to highlight key words and phrases so that reviewers can skim and quickly grasp your main points.

4. Understand that the time from first submission to funding may be very, very long. Plan your work accordingly. Your career strategy must be able to accommodate long review times at NIH. Your career cannot come to a standstill while you wait years to find out if you will receive NIH funding. Understand that you cannot have the same science in review at two NIH ICs simultaneously. However, you can and should submit the same science to different federal agencies simultaneously (ex- NIH and HRSA, AHRQ, DoD, PCORI, NSF, etc). And of course you can blanket the private foundation landscape with the same project. (Should you receive funding from multiple entities— may you have such a problem—this conundrum can generally be worked out in consultation with the program officers.)

5. The standard for quality of a submission is likely to be raised, so you will need to write stronger applications than ever before. Historically, it seemed that if reviewers knew you had a resubmission available to you, they expected you to use it to hone and polish the submission. Because reviewers know you have endless resubmissions available, I wonder if they might raise the bar for quality of the application. They may want you to resubmit until they feel it is just right (which is a moving target, given that your original reviewer(s) may have rotated off the study section before you resubmit.) In addition, a sharp increase in the number of submissions will likely mean that competition will be stiffer, so the quality of the submission will need to be even higher to stand out from the larger crowd. Grantees will need to write their applications more thoughtfully than ever, taking extra time and care.

6. Institutions also will need to step up their game if they want to stay ahead of the competition. If your institution does not do so already, now is the time to implement Chalk Talks and Red Team Reviews (i.e., mock study sections) if you want to help your grantees succeed with the policy change. And of course, hire your grantees a professional grant writer for a few thousand dollars to improve the quality of their R01. It will improve their grantsmanship not just on this submission but on all submissions going forward, and if it helps land even one award it will have paid for itself by many orders of magnitude in the indirects received.

7. Write every submission as if it were your only shot at funding. I am afraid that less savvy grantees may think that they should jump into the game and submit an application that may not represent their best effort. After all, what harm is there in doing so if they can just keep submitting? And wouldn’t it be helpful to have multiple summary statements to hone one’s grantsmanship? Keep in mind that if you are submitting to the same study section repeatedly, they may not formally be given previous submissions and summary statements. But if you have some of the same reviewers, they will remember your past submissions, and it may color their impressions of the current submission. I liken it to a jury who hears testimony and is then instructed by a judge to ignore it in their deliberations. We know from the social psychology literature that despite our best efforts, it is not feasible to act as if we do not know something. If your assigned reviewers still serve on the study section, they may recognize your A0 as something they read in the past, and may recall their reaction to the previous submissions. Reviewers, like jurors, are human. Therefore, I maintain that grantees should write every submission as if it were their only one.

Can you think of other ways you will need to alter your approach to NIH applications because of the new resubmission policy? Do you like or dislike the new policy?

Reviewers of NIH Grant Submissions May Pay A Heavy Price When Their Own Submissions Are Reviewed


Grant scores are being posted right now. I was at a policy meeting last week with one of my grant clients and we waited with baited breath to see her scores. Although she has had a great deal of success on past NIH submissions, she was worried about this one, and with good reason. She recently agreed to serve as a reviewer on an NIH study section, but quickly realized that it meant that the resubmission of her own R01 competing renewal could no longer be reviewed in the most appropriate study section. Instead, it was assigned to a Special Emphasis Panel, or SEP. My client studied the list of reviewers on the SEP and learned that there was no one on the panel with the expertise needed to review her submission. She informed her SRO of this problem, but was told to sit tight and wait and see how the review went.

My client is considered an eminence in her field. She chairs a nationally-ranked academic department in her area, has landed numerous R01s, a U01, ARRA funding, and more. On her R01 competing renewal she scored in the 20th percentile, but the problems identified by the reviewers were certainly fixable and she resubmitted with hope, if not confidence. Then she accepted the position on the study section and the resubmission went to the (underqualified) SEP. The results of the resubmission were posted: Unscored.

It is hardly the first time I have heard such a story, it is just the most recent. It is considered an honor to be invited to serve on a study section. Many, like my client, choose to do so despite the time and effort involved because they feel they should give back for the many years of funding they have received from NIH. But the inadvertent result may be that the reviewer’s own submissions must be reviewed elsewhere, often on a panel that lacks the necessary expertise. As a result, my client and others have decided to terminate their service on study sections, which has the result of denying other NIH grantees reviews from those deemed the most qualified to provide them.

The current policy creates a situation where reviewers of a study section may be unable to get a fair review of their own grant applications at the most appropriate study section, thereby essentially penalizing them for service to NIH.

When discussing this problem last week in Washington with an NIH program officer, I was told to contact my local congressman. The PO felt that if we wanted to effect change to this NIH policy, the only approach was to inform a congressman that his constituents were not receiving millions in federal funding due to this policy. I plan to do so, and urge you all to do the same.