The results of the Research Excellence Framework (REF) will be announced tomorrow. That makes today the perfect day for two blogs about the REF. Today, before we know the results, we can look at the REF through a sort of Rawlsian veil of ignorance http://en.wikipedia.org/wiki/Veil_of_ignorance . Today nobody knows (for sure) whether their department is a REF winner or a REF loser. Critiques of the REF are harder to dismiss as sour grapes (similarly all those in favour of the REF should get their tributes in now).

First, let’s make clear what I’m not saying:

1. Academics/universities shouldn’t be judged or accountable for the money they receive from the public purse
2. Money shouldn’t be distributed according to merit
3. Colleagues who have slaved hard to ensure their department does as well as possible in the REF were wrong to do so (responding to incentives is sane, even if the incentives aren’t)
4. Darling our research is so ground-breakingly original it is crude and bourgeois to think you can measure it[1]

Clearly it is completely unrealistic to think that universities can receive public cash without ‘the public’ wanting to determine that allocation[2]. Today I just want to ask: given that the government gives cash for research, and given that it will need some vaguely justifiable way to do so, is the REF really the best way? (And in the next post, if the answer is no – spoiler alert: I suspect the answer is no – why do we have the REF?)

The REF is not a useful means of distributing cash

Because it provides little new information. Scores in the RAE (the substantially similar precursor to the REF) correlated very closely with the distribution of income from research councils[3]. To a first approximation, if you just doled out REF cash (within subject areas) according to how much Research Council income university departments had received over the last x years, you’d end up with essentially the same outcome. And where there are differences, we have no objective basis for considering one distribution of money as superior to another.

Because it wastes resources

I’m not primarily concerned about the direct financial costs of the REF. What really worries me is the diversion of high calibre individuals from their research to the REF. Not just the many days spent by REF panel members and reviewers, but also the many days academics have spent preparing for the REF: carrying out mock REFs, advising other departments on their submissions, being advised by other academics on their submissions, writing submissions, hyphenating lots of pairs of words in submissions so they fit the word limit, etc, etc. All of that time could have been spent doing something else. This is rent-seeking pure and simple. The REF is a zero-sum game, and this activity serves no wider social purpose.

Of course, you could argue that if the REF money were doled out in proportion to research grants[4] (as suggested above), that would increase the load on grant reviewers, and grant applicants – because each grant would become more valuable, which might increase the number of applications per grant. This is true to an extent, but there’s three reasons to suspect that the ratio of rent-seeking to productive work would be lower in this case:

1. Grants are an important input to REF outputs (papers etc) so arguably much of this value should already be ‘priced in’ to people’s cost-benefit consideration of whether to apply.
2. Grants are for work not yet done, so a fair proportion of the work that goes into preparing a grant application (and reviewing it) is time spent productively thinking about how the work will be done[5] (even if the bid is unsuccessful, the same work is often recycled for other applications). The REF, on the other hand, is conducted ex post. No useful feedback is provided on why a particular output was rated 4* or 1*, and if it were to be, it’s mostly too late and many of the people involved in the work aren’t even still in the department[6]. But most importantly, the REF is a whole extra parallel system. A whole new set of rules to learn, committees to attend, etc, etc, just in order to review much of the same research again[7]. It must be more time consuming than, say, simply expanding the grants system by x%[8].
3. In the REF the links between the useful work (the research) and the money are weaker. If you get a grant, you get the cash, and you have to do the work. The money doesn’t go to people who used to be in the same department as you, or who might one day be in the same department. Sure, unlike prizes, grants are based on ex ante judgments, but if you hadn’t done good work before you wouldn’t have got it and if you disappoint you’re less likely to get another. The REF rewards departments for work that was partially done by people who were partially in the department for some proportion of the past. The same work can be submitted non-rivalrously by multiple different departments, and needs owe nothing to the department itself: hence the practice of paying a fee to a hot shot American who could barely find your university on a map, so you can claim their research as your own.

Next up – if the REF is a bad idea, why does it exist?

[1] There’s some truth in this, of course: measurement of the significance of research (often only a year or two after it was published) is notoriously hard. But let’s not go there now.

[2] Since in practice, it isn’t the public but the government that determines the allocation, we might wonder what implications a heavy reliance on government funding has for the role that academics should play in society as free-thinkers, ready to challenge authority and accepted wisdom, whether scientific or otherwise. But that’s a question for another day.

[3] See e.g. http://www.timeshighereducation.co.uk/news/ref-alternative-delivers-similar-outcome-for-qr-funding/2003104.article and http://www.rsc.org/chemistryworld/Issues/2007/March/RidingRAERollercoaster.asp this should surprise no-one, since the people judging merit are substantially the same in each case, and grants are supposed to lead to REF-able outputs!

[4] Note I’m not recommending this as the best possible system. But I do think it’s a useful comparator.

[5] We can all think of many ways the grant system could be improved. But just as I’m not looking at the myriad flaws in the detail of the REF, so I’ll gloss over the flaws in the grant system. Both could be improved, and if both were improved, we’d be back with the same meta issues.

[6] Some were never really there in the first place!

[7] Research councils are increasingly also reviewing work ex post, triplicating the review process.

[8] Not that the REF doesn’t function like prizes, which may be an efficient system of distributing some science funding, since most of the effort stimulated by, say, the Nobel prize is productive (even for those who don’t win) and the judging costs are much smaller: the Nobel committee doesn’t give all medical researchers or economists a score each year!