Implications for Grantees of no more “Two Strikes You’re Out”

Credit: adamr at FreeDigitalPhotos.net

Ding, dong the evil resubmission policy is dead. What are the implications in terms of your grantsmanship strategy and writing?

The original intent of the ill-fated, much-hated “Two Strikes You’re Out” policy was to reduce the time from first submission to award. According to the statistics and metrics put out by NIH, it was achieving that goal. With the policy change that went into effect last month, grantees still submit an A0 (new application) and an A1 (first resubmission), with no A2 (second resubmission) available. But if they fail at both the A0 and A1, they no longer need to shelve the application forever. They can now submit the application again as an A0 to any IC or study section they choose, and the reviewers will be given no information about earlier versions or summary statements. I have yet to find a grantee who is unhappy about the change of policy. ESI and New Investigator grantees in particular have expressed relief that they will have multiple attempts to “get it right” (the incorrect assumption being that study sections are static entities and that each review will be similar to previous reviews in terms of expectations and criticisms.)

Yet, it seems to me that this resubmission policy will lead to even longer lengths of time from first submission to award than were seen when people were allowed three submissions total. In essence, the very problem that “Two Strikes You’re Out” sought to fix will be made worse than it was prior to the “Two Strikes” policy. Here is my tongue-in-cheek mathematical representation of the new resubmission policy:

(A0, A1) ∞

To my mind, this policy equates not only to longer times to award than ever before, but also to an unprecedented number of NIH submissions going forward. In recent years, the total aggregate number of R01 submissions has been fairly steady at about 28 or 29 thousand per year. I expect we will see a dramatic increase in that number in FY14 and into FY15, as all those shelved, failed submissions go back into circulation. At some point that number will decrease a bit to a steady state that is still significantly higher than 29 thousand applications per year.

Love or hate the new policy (and while I recognize that “Two Strikes” was flawed, I am firmly in the latter camp), how should you fold this information into your approach to grantsmanship?

1. Consider deferring your June 5 A0 to the Oct deadline. I suspect that everyone and his brother will be dusting off an old, unfunded application and submitting an A0 for June 5. I suspect that that will equate to a gigantic number of submissions for this cycle, and my suspicions appear to be supported by the sheer volume of A0s on which I have been contracted to work. (I haven’t seen a crushing number of applications like this since the stimulus money.)

2. Give yourself more time to upload to the grants.gov portal with any R01 deadline going forward, but particularly for the June 5 deadline. As you know, applications must be uploaded and time stamped. When traffic at the portal is heavy, there are delays in your ability to upload. Many grantees have told me they missed a deadline because they waited too long and could not upload due to traffic at the portal, or not enough time to correct error messages received. This problem will be exacerbated by the increased application volume from now on.

3. Format your application with greater care than ever. A sharp increase in submissions will likely mean that reviewers will be even more overworked than ever, and the increased time/cost of reviewing all these extra applications may mean that CSR will step up its experiments with virtual peer review. How should this affect your writing? Use lots of unique identifiers throughout the application (numbering system with multiple subheadings) to help orient reviewers who are exhausted and/or not meeting face to face while discussing your application. Make judicious use of formatting to highlight key words and phrases so that reviewers can skim and quickly grasp your main points.

4. Understand that the time from first submission to funding may be very, very long. Plan your work accordingly. Your career strategy must be able to accommodate long review times at NIH. Your career cannot come to a standstill while you wait years to find out if you will receive NIH funding. Understand that you cannot have the same science in review at two NIH ICs simultaneously. However, you can and should submit the same science to different federal agencies simultaneously (ex- NIH and HRSA, AHRQ, DoD, PCORI, NSF, etc). And of course you can blanket the private foundation landscape with the same project. (Should you receive funding from multiple entities— may you have such a problem—this conundrum can generally be worked out in consultation with the program officers.)

5. The standard for quality of a submission is likely to be raised, so you will need to write stronger applications than ever before. Historically, it seemed that if reviewers knew you had a resubmission available to you, they expected you to use it to hone and polish the submission. Because reviewers know you have endless resubmissions available, I wonder if they might raise the bar for quality of the application. They may want you to resubmit until they feel it is just right (which is a moving target, given that your original reviewer(s) may have rotated off the study section before you resubmit.) In addition, a sharp increase in the number of submissions will likely mean that competition will be stiffer, so the quality of the submission will need to be even higher to stand out from the larger crowd. Grantees will need to write their applications more thoughtfully than ever, taking extra time and care.

6. Institutions also will need to step up their game if they want to stay ahead of the competition. If your institution does not do so already, now is the time to implement Chalk Talks and Red Team Reviews (i.e., mock study sections) if you want to help your grantees succeed with the policy change. And of course, hire your grantees a professional grant writer for a few thousand dollars to improve the quality of their R01. It will improve their grantsmanship not just on this submission but on all submissions going forward, and if it helps land even one award it will have paid for itself by many orders of magnitude in the indirects received.

7. Write every submission as if it were your only shot at funding. I am afraid that less savvy grantees may think that they should jump into the game and submit an application that may not represent their best effort. After all, what harm is there in doing so if they can just keep submitting? And wouldn’t it be helpful to have multiple summary statements to hone one’s grantsmanship? Keep in mind that if you are submitting to the same study section repeatedly, they may not formally be given previous submissions and summary statements. But if you have some of the same reviewers, they will remember your past submissions, and it may color their impressions of the current submission. I liken it to a jury who hears testimony and is then instructed by a judge to ignore it in their deliberations. We know from the social psychology literature that despite our best efforts, it is not feasible to act as if we do not know something. If your assigned reviewers still serve on the study section, they may recognize your A0 as something they read in the past, and may recall their reaction to the previous submissions. Reviewers, like jurors, are human. Therefore, I maintain that grantees should write every submission as if it were their only one.

Can you think of other ways you will need to alter your approach to NIH applications because of the new resubmission policy? Do you like or dislike the new policy?

The End of Two Strikes You’re Out: Good or Bad News for Grant Applicants?

Yesterday, NIH and AHRQ announced that they had changed its resubmission policy. While a given application is still allowed only one resubmission, if you are unsuccessful on the A1, you can then submit the application again as an A0 without having to substantially redesign the content and scope of the project. This “new submission” will be reviewed without any association to the previous attempts. There will be no Introduction to the Revised Application, no explanation of how you addressed the reviewers’ concerns. Even if reviewers have seen the application in prior review cycles, they will be instructed to review it as new. (However, reviewers are human, and one wonders if this is realistic. It’s like asking a juror to ignore evidence they just heard, and we know from social psychologists that that doesn’t actually work.) Of course, the idea is that an applicant will use previous reviewer comments to strengthen the application, thereby improving their odds of funding.

Click here to read the full notice

In 2009, as part of their Enhancing Peer Review project, NIH eliminated the A2 in what has been dubbed by unhappy researchers as the “Two Strikes You’re Out” policy. NIH eliminated the A2 because meritorious research was most likely funded on the A2, which meant quite a delay to funding. The resubmission policy did indeed result in an increase in the number of awards made on A0 applications (although most funded applications are successful on the A1). About the “Two Strikes You’re Out” policy, Deputy Director Dr. Sally Rockey states: “…we heard increasing concerns from the community about the impact of the policy on new investigators because finding new research directions can be quite difficult during this phase of their career. Also, established investigators voiced concern about the need to redirect the research focus of productive labs in order to submit future NIH applications.” For one of many discussions on Dr. Rockey’s blog about the decision to sunset the A2 submission, click here.

In theory, it would appear that one could submit the same idea endlessly until one finds the best way to sell it to reviewers. The policy will no doubt appeal to many researchers, because it is now up to a researcher to decide when it is time to abandon a given project. If one waits long enough, one could submit to the same study section after most of the members have turned over, thereby having a new set of reviewers to weigh in on the project. One risk I see is that sometimes a PI is so blindly enamored with their idea that they have difficulty hearing that the reviewers are trying to tell them that the idea is simply not fundable in any form; i.e., no amount of tinkering with the writing or the details will fix it. A PI could waste a lot of time if they cannot see when it is time to abandon an unfundable project.

Judging from the comments on some of the NIH blogs (for examples, click here), many reviewers are thrilled to have more submission attempts, while others are dismayed that this marks a return to a huge number of submissions and long delays to funding. What do you think?

The Center for Scientific Review Board Makes Grant Review Suggestions to NIH

The NIH Center for Scientific Review (CSR) publishes Peer Review Notes to inform reviewers, NIH staff, and others interested in news related to grant application review policies, procedures, and plans. The latest issue of Peer Review Notes was sent out last night (they publish three times per year). Here are items that caught my attention:

  • Become more scientific in assessing approaches to improve the efficiency and particularly the quality of NIH peer review.
  • Work hard to understand and address possible disparities in NIH awards.
  • Collaborate with the NIH and scientific communities to identify critical problems, such as the definition of a “new” application, and to develop solutions.
  • Help the public understand the role of NIH peer review in advancing science and health in the United States.

I certainly wanted more clarity on those first three bullets, some of which I found in another article in this issue of Notes, which I have copied below (my comments appear after each numbered suggestion):
CSR’s Council Suggests Five Ways NIH Can Help Applicants CSR’s Advisory Council recently asked NIH to consider five ideas for helping applicants with promising research ideas to stay in the game despite historically low funding rates. Because these ideas deal with trans-NIH policies beyond CSR, Council members asked CSR’s Director to share them with the appropriate NIH officials.

CSR Council Ideas

1. Treat all applications as new and let investigators instead of NIH decide when resubmission is futile. Council members suggested that the resulting reviews would be more independent and simplified since earlier reviews would not be considered. Reviewers might also be more focused on merit because they wouldn’t get sidetracked by considering how investigators responded to previous reviews. Our Council suggested doing a pilot where investigators who opt-in could resubmit any R01 application as many times as they wanted, but they could submit no more than two research project grant applications in any 12 month period. Reviewers would be encouraged to send strong messages about applications that need substantial revision.

          Meg’s comment: I suspect PIs would love to submit an R01 application as many times as they like, though some folks would balk at being limited to two RPGs per 12-month period (I assume they mean any RPG at any IC). Many PIs I know would appreciate a clear message from reviewers about whether they should resubmit. The grant score alone does not always help them decide, as I have seen applications go unscored because the reviewers wanted an entire aim added or taken away, but they were very favorable about the rest of the application (in this instance, clearly the team should resubmit even though the A0 is unscored.)
_________________________
2. Encourage more NIH Institutes and Centers (ICs) to allow investigators to respond to their reviews prior to Council consideration so very promising applications that might slip through the system could be identified. Principal investigators (PIs) with “gray zone” applications would be asked to provide a response to their reviews. IC Program staff would submit these comments and applications to their Councils, which provide the second level of peer review.
          Meg’s comment: Again, I can imagine that most PIs would welcome the opportunity to speak persuasively about their project if they score near the funding line, though this strategy adds to the workload for both the PI and PO, which is something NIH has been trying to avoid.
________________________
3. Enhance communications with PIs: Study sections and NIH program staff should do better at communicating to PIs about applications that are unlikely to be successful or, alternatively, are of potential interest.  [See our last PRN newsletter: Make the Best Use of the “Additional Comments to Applicant” Box]
          Meg’s comment: My clients and I spend plenty of time trying to read between the lines of pink sheets to figure out if the reviewers would welcome a resubmission. It can be exceedingly difficult for a PI to read his/her pink sheets, let alone accurately assess the subtleties of the comments. On the one hand, a clear message from the reviewers about resubmission would be welcome. On the other hand, if the PI feels he can address the problems, or feels the review was less-than-fair and s/he would like to wait it out and resubmit to the same study section after there has been some turnover, they should have the opportunity to do so regardless of what the study section said in the pink sheets. Note that this CSR Council recommendation seems to contradict their first recommendation above: “…let investigators instead of NIH decide when resubmission is futile.”
______________________
4. Encourage NIH ICs to take full advantage of the R56 funding mechanism to provide bridge funding to promising investigators. These “High Priority, Short-Term Project Awards” provide 1 year funding for high-priority new or competing renewal R01 applications that score just outside an ICs funding limits.
          Meg’s comment: The little-known R56 funding mechanism is not one for which a PI can apply directly, it is awarded to a PI with a promising application to another grant mechanism. The award is made at the discretion of the Program Officer, which is one of the very many reasons I am a huge advocate of the PI cultivating a relationship with the PO. Note also that I had a client receive an R56 even though his original application was nowhere near the funding line. He had a great relationship with his PO, and the PO believed in him and his work.
___________________________
5. Provide longer-term funding for some PIs: For investigators with large and successful programs, NIH should consider offering funding for a longer duration but at a lower overall amount. The savings would be used to fund more applications. Restrictions on participating PIs would be necessary to ensure that the result would be revenue-positive.
          Meg’s comment: This is such a mixed bag I don’t even know where to begin.
____________________________
What do you think of the CSR Advisory Board recommendations to NIH?