5% support Welsh independence, no wait 17%, no wait, 3%, no wait…
Neatly segueing from posts on Scottish independence to a subject closer to my professional heart – survey design…
Yesterday, the BBC released an ICM poll finding that support for Welsh independence was a mere 3%. This was lower than the 5% found earlier in the year (though the difference is within the margin of error, making the BBC headline claim tenuous at best).
Just as ITV’s poll was welcomed by those favouring independence, the BBC’s poll was greeted with enthusiasm by unionist parties, who interpreted the poll as a blow to Plaid Cymru, who support Welsh independence as a long term goal:
— Welsh Conservatives (@WelshConserv) September 24, 2014
(The Welsh Conservatives twitter feed was strangely silent about the ITV poll on the 15th September).
So how can two polls, conducted by reputable polling agencies, get such different answers a few weeks apart?
Well, the first answer is obviously that all polls have a margin of error. But that’s unlikely to explain the full size of the gap.
Post #indyref hangover?
Second, the BBC poll was conducted after the Scottish result, while the ITV/YouGov poll was conducted during the campaign when the “Yes” camp was doing well, and Scottish independence seemed quite possible. Previous polls have found that more Welsh people support Welsh independence in the event of Scottish independence – “who wants to be Montenegro to England’s Serbia” – but the difference is usually just a couple of % points. Plus the YouGov poll specifically looked at this issue, asking two separate questions and it made little difference to the results: 17% supported independence regardless of whether Scotland voted for it or not. So while the timing of the polls probably made some difference it’s unlikely to be the full story.
Different questions, different answers
The biggest reason for the gap between the two polls is almost certainly due to the way the questions were asked. ITV/YouGov asked a straight yes/no question:
if there was a referendum held tomorrow on Wales becoming an independent country and this was the question, how would you vote?
Should Wales be an independent country?
-Would not vote
The BBC/ICM poll asked a very different question:
Which of these statements comes closest to your view?
– Wales should become independent, separate from the UK
– The Welsh Assembly should have more powers than it currently has
– The powers it currently has are sufficient and should remain as it is now
– The Welsh Assembly should have fewer powers than it currently has
– The Assembly should be abolished and Wales governed directly from Westminster
– None of these
– Don’t Know
While only 3% chose independence, 49% chose more powers. It’s well understood that including more options in a survey question is likely to decrease the % opting for any given choice – the options compete against each other. For example, it seems likely that if the independence option were removed from the BBC survey, most of the 3% would transfer to the more powers option, and the headline result would have been “majority in favour of more powers”.
The poll probably also reflects the way people think about the issue as one which evolves over time. Wales is very far from being an independent country (it has considerably less autonomy than, say, American states or German länder) and it’s hard to imagine it becoming independent without several further steps of devolution/federalism happening first. I suspect many of the people answering “more powers” in the BBC poll would have chosen independence in the ITV poll, and the difference may indicate a reasonable preference for gradual change. Indeed, this suggests that the ITV poll, asking as it does about a referendum “tomorrow”, might actually underestimate support for independence.
Although it’s clear that the precise structure of the question matters, I’m not necessarily saying there’s a conscious bias here (unlike the Daily Wales website , which also makes another interesting point about question wording). There’s not necessarily a right or wrong way to do it, it depends what you are interested in. Clearly by offering an option including more powers, the BBC/ICM poll is more informative about that option and possibly more informative about independence: maybe some people supporting independence in the ITV poll would prefer greater devolution, but would take independence over the status quo.
But if you did want to rig a poll…
I do know that if I wanted a poll to find low support for independence, I’d include plenty of other options. Indeed adding a strong federal option to the BBC question could probably reduce support even further, attracting a few of the 3%. On the other hand, if I wanted to increase apparent support for independence I’d use a question like ITV’s, but ask something like “Do you think Wales should become an independent country within a generation”. I suspect that would find considerably more than 17% in favour (especially if one excluded the ‘don’t knows’/’wouldn’t votes’ from the headline figures, as is common). I could go one step further and do the first poll in the autumn (when risk aversion is high) and the second in the spring, and get an even bigger gap, or play around with the questions that precede the focal question:
Read the small print
So the answer to the question “What % support Welsh independence” could be less than 3% or a fair bit greater than 17%, depending on how we ask the question, when we ask the question, and how we analyse and present the results. The take home message is therefore that you should be very wary about headlines based on opinion polls: there’s just no substitute for looking at the small print yourself.
Update: Prof Roger Scully has a nice post on question wording and Welsh opinion polls here
 Fieldwork was conducted 8-11th September
 Field work was conducted 19-22nd September
 I’m grateful to Prof Roger Scully, of Cardiff University, for confirming the question wording.
 Daily Wales could well be right, but it’s hard to impute motives from survey questions alone, which is all I have – they might know more.
 However, given what we know about status quo bias, and given that more powers for the assembly are already on the cards, this seems unlikely