The Center for Scientific Review publishes their Peer Review Notes three times a year, and the most recent issue came out yesterday. The news items are always interesting and it is worth subscribing, if you don’t already. This issue contained an item about things NIH reviewers should not say. I repeat the list in its entirety here—I thought it might be fun for my grantees to see reviewers critiqued for a change.
What do you think of this list? Have you seen one or two of these on your Summary Statements? Me personally? I have seen variations on # 2, 4, and 10 in Summary Statements, and have strongly suspected reviewers of #1 and 5. I almost fell out of my chair laughing when I read # 7, sometimes I think CSR is a little out of touch with what actually happens on Study Sections:
“I didn’t read the application, but I scanned it and saw the applicant said XXX. He doesn’t know what he’s doing.” Damning statements like this can skew a review discussion over something that might be insignificant in the context of the overall application. It’s better for you to ask other reviewers who have read the application carefully what they think about XXX.
“This New Investigator does not appear to be fully independent since he continues to co-publish with his fellowship mentor/department chair, or does not have designated lab space, or has not been promoted in the past several years.” Academic research organizations have widely diverse policies for faculty advancements and lab space, and many PIs maintain productive and healthy collaborations with mentors for many years after establishing themselves as bona fide investigators. You should focus more on the investigator accomplishments, such as being the first or senior author on a significant publication or giving presentations at major scientific meetings.
“This application is not in my area of expertise . . . “ If you’re assigned an application you feel uncomfortable reviewing, you should tell your Scientific Review Officer as soon as possible before the meeting.
“I don’t see this basic science research affecting my clinical practice any time soon.” An application does not necessarily have to show the potential for clinical or timely impact—if the applicant doesn’t make such claims. Basic research often takes time to pay off, and you’re charged to assess the “likelihood for the project to exert a sustained, powerful influence on the research field(s) involved.” Absence of an effect on public health does not necessarily constitute a weakness in basic science.
“I like this project but I’m giving it a poorer score because the applicant has too much money.” Other funding is not a scoreable matter. You should focus on the application’s scientific and technical merit. However, you can note an excessive budget request in the budget section for NIH to consider.
“This application has 2 great aims and 1 bad one. I would recommend deleting Aim 3, and I can give it a 1 or 2.” You cannot trade aims with scores. The application needs to be evaluated as a whole.
“This R21 application does not have pilot data, which should be provided to ensure the success of the project.” R21s are exploratory projects to collect pilot data. Preliminary data are not required, although they can be evaluated if provided.
“The human subject protection section does not spell out the specifics, but they already got the IRB approval, and therefore, it is ok.” IRB approval is not required at this stage, and it should not be considered to replace evaluation of the protection plans.
“This application was scored a 25 and 14th percentile last time it was reviewed . . . .” You should not mention the previous score an application got, because this could skew the review discussion. Focus on the strengths and weaknesses of the current application as well as the responses to previous critiques.
“This is a fishing expedition.” It would be better if you said the research plan is exploratory in nature, which may be a great thing to do if there are compelling reasons to explore a specific area. Well-designed exploratory or discovery research can provide a wealth of knowledge.
Yesterday, NIH and AHRQ announced that they had changed its resubmission policy. While a given application is still allowed only one resubmission, if you are unsuccessful on the A1, you can then submit the application again as an A0 without having to substantially redesign the content and scope of the project. This “new submission” will be reviewed without any association to the previous attempts. There will be no Introduction to the Revised Application, no explanation of how you addressed the reviewers’ concerns. Even if reviewers have seen the application in prior review cycles, they will be instructed to review it as new. (However, reviewers are human, and one wonders if this is realistic. It’s like asking a juror to ignore evidence they just heard, and we know from social psychologists that that doesn’t actually work.) Of course, the idea is that an applicant will use previous reviewer comments to strengthen the application, thereby improving their odds of funding.
In 2009, as part of their Enhancing Peer Review project, NIH eliminated the A2 in what has been dubbed by unhappy researchers as the “Two Strikes You’re Out” policy. NIH eliminated the A2 because meritorious research was most likely funded on the A2, which meant quite a delay to funding. The resubmission policy did indeed result in an increase in the number of awards made on A0 applications (although most funded applications are successful on the A1). About the “Two Strikes You’re Out” policy, Deputy Director Dr. Sally Rockey states: “…we heard increasing concerns from the community about the impact of the policy on new investigators because finding new research directions can be quite difficult during this phase of their career. Also, established investigators voiced concern about the need to redirect the research focus of productive labs in order to submit future NIH applications.” For one of many discussions on Dr. Rockey’s blog about the decision to sunset the A2 submission, click here.
In theory, it would appear that one could submit the same idea endlessly until one finds the best way to sell it to reviewers. The policy will no doubt appeal to many researchers, because it is now up to a researcher to decide when it is time to abandon a given project. If one waits long enough, one could submit to the same study section after most of the members have turned over, thereby having a new set of reviewers to weigh in on the project. One risk I see is that sometimes a PI is so blindly enamored with their idea that they have difficulty hearing that the reviewers are trying to tell them that the idea is simply not fundable in any form; i.e., no amount of tinkering with the writing or the details will fix it. A PI could waste a lot of time if they cannot see when it is time to abandon an unfundable project.
Judging from the comments on some of the NIH blogs (for examples, click here), many reviewers are thrilled to have more submission attempts, while others are dismayed that this marks a return to a huge number of submissions and long delays to funding. What do you think?
In January 2009, NIH instituted a policy to sunset A2 applications, i.e. to eliminate a second resubmission. Since, investigators submit a grant application (A0) and if the application is unfunded they may submit one revised application (A1), and no more (previously, one could resubmit twice). This policy has been detested among research scientists since its inception and NIH has heard an earful of complaints about it. NIH spoke out recently on this topic. Will they consider eliminating the wildly unpopular policy? The short answer: Not bloody likely.
In a recent blog post, Sally Rockey gives some data NIH has compiled on this policy. One of the goals of the policy was to reduce the time to award, and in fact the time from A0 submission to award has been reduced from 93 weeks to 56 weeks by this policy. It is conventional wisdom that NIH grant applications, if funded, tend to be successful on their final submission, with very few succeeding as an A0. Yet surprisingly, since the policy went into effect, the proportion of funded A0 applications as compared to A1 apps has increased.
One major complaint of the policy is that it favors established investigators over new investigators. The data do not appear to support this idea, because the time-to-award for new applicants is not appreciably longer than for the entire population.
In response to the suggestion that NIH allow A2 applications only from those investigators whose apps fall just outside the payline, NIH modeled the potential impact of such a policy using FY11 data. From their model, NIH concluded that the result would be to shift awards from A0 and A1 apps toward A2 apps, which was what the policy was designed to avoid to begin with (see original blog post for details of their method.)
Rockey concludes: “Overall, these data indicate that the policy to sunset A2 applications continues to achieve the stated goals of enabling NIH to fund as much meritorious science as possible in as short a time period as possible. Any revision to the policy to allow additional resubmissions of all or a subset of A2 applications will displace equally meritorious A0 and A1 applications, and increase the time to award for many applications. For these reasons, we have decided to continue the policy in its current form.”