“Why wasn’t my grant funded?”
Given most research funding agencies have success rates of 20% or less, this is a question that is asked by the majority of applicants every year. Often the only answer members of assessment panels are allowed to give is unsatisfyingly circular: because the application wasn’t ranked highly enough.
But how do such panels make their decisions? Here’s how it’s supposed to work.
Most panels consist of ten or more experienced researchers with expertise related to the applications they will consider. They are usually given about 100 applications to read, each of 50 pages or more.
A primary spokesperson will lead the discussion at the meeting. Sometimes a secondary and tertiary spokesperson are also added to balance the comments of the primary spokesperson and explore issues that may have been missed. Consequently, each panel member will be paying special attention to between 20 and 30 grants.
Grants may also be sent out to two or more discipline-specific reviewers, who will send back reports on the merits of the projects and views on the quality of the researchers.
These reports are discussed together with the grants when the panel comes together at the week-long assessment meeting.
The primary spokesperson summarises the strong and weak points of the project, its importance and feasibility related to the achievements of the research team. After the secondary and perhaps tertiary spokesperson have also provided comments, there is a general discussion.
Then the Chair calls for scores. The spokespeople will often declare their scores but the rest of the panel may vote in secret. There is sometimes a proviso that anyone who wishes to score away from the consensus should declare their score. This serves to limit any extreme views that may emerge without discussion.
The scores are then tallied and grants are ranked.
At the end, all the scores are reviewed and the panel considers whether it has scored consistently throughout the week and makes adjustments if necessary. This is important, as sometimes the scores are tougher at the beginning when people tend to give moderate scores because no one wants to appear too radical. Sometimes a few grants from previous years are considered first to calibrate the panel.
There is always a concern that personal connections may have, or will be perceived to have, influenced the process, so most panels are scrupulous in excluding from the room anyone who has any connection with researchers whose work is being ranked. One can never totally eliminate “who-you-know” biases, but panels try very hard to do just that.
What actually happens
In my experience sitting on various panels, I read all the grants and I am amazed at the quality. I shouldn’t be. These are grants from people who have been successful at every stage of their careers, and the documents have usually been honed further by advice from senior mentors or even internal grant polishing teams made up of academic colleagues.
When I look at the ten grants for which I am primary spokesperson, my heart sinks. I realise that only two or perhaps three of these grants will get funded. I need to find reasons to “not fund” at least some of the grants, so I hunt for some that are hopeless.
Sometimes I can’t find any. Sometimes I find one or two teams who have not really demonstrated significant expertise yet. They may have fewer publications, or ones that are not directly related to the topic in hand, in which case I may regard that as an objective reason to place them near the bottom.
I also look for really stellar ideas. But given I don’t know the disciplines as well as the researchers themselves, and the grants are typically very well presented, all the ideas look great to me!
But at the panel meeting, I find that opinions vary about whether ideas are brilliant or not. Sometimes the most brilliant ideas are the most divisive. So inevitably track record and recent publications tend to count for more.
I work through and I nearly always find two applications where the researchers have recently struck gold. They now want to follow their gold seam and keep harvesting exciting results. None of us think that papers in Nature and Science, or other high impact factor journals, are everything – but we all recognise such papers have cleared a high bar based on interest and tough reviewing – so if I do see applications with top recent publications, I often do rank them near the top.
Now my top two spaces are filled.
And the rest
Suddenly I realise that with a 20% success rate, no more, or perhaps only one more grant in my pile will be funded. With two grants at the top, and perhaps two at the bottom, I now have six left. But they all look good.
Nevertheless my job is to rank them. I will be inclined to rank the grant I understand best or the one that will be most exciting if the project works at the top. Or the one that comes from a group whose record is really impressive. Panels tend to agree on track records, so past performance does count disproportionately.
Interestingly, while the reviewers’ reports can be very helpful – like most researchers I like to make my own decisions – generally these are only influential if everyone feels out of their depth on the topic.
Applicants should not be too worried by preliminary referee reports because the panels may well overrule both extreme negative and positive reports. When you see your referee reports each year, be careful not to over-react. All referees feel they have to say some good and some bad things.