The importance of survey design: how many people really support Welsh independence?

5% support Welsh independence, no wait 17%, no wait, 3%, no wait…

Neatly segueing from posts on Scottish independence to a subject closer to my professional heart – survey design…

On the 15th September (during the Scottish independence campaign[1]), ITV Wales released a YouGov poll which found that 17% of Welsh people favoured independence for Wales, a higher than normal %.

Yesterday[2], the BBC released an ICM poll finding that support for Welsh independence was a mere 3%. This was lower than the 5% found earlier in the year (though the difference is within the margin of error, making the BBC headline claim tenuous at best).

Just as ITV’s poll was welcomed by those favouring independence, the BBC’s poll was greeted with enthusiasm by unionist parties, who interpreted the poll as a blow to Plaid Cymru, who support Welsh independence as a long term goal:

(The Welsh Conservatives twitter feed was strangely silent about the ITV poll on the 15th September).

So how can two polls, conducted by reputable polling agencies, get such different answers a few weeks apart?

Well, the first answer is obviously that all polls have a margin of error. But that’s unlikely to explain the full size of the gap.

Post #indyref hangover?

Second, the BBC poll was conducted after the Scottish result, while the ITV/YouGov poll was conducted during the campaign when the “Yes” camp was doing well, and Scottish independence seemed quite possible. Previous polls have found that more Welsh people support Welsh independence in the event of Scottish independence – “who wants to be Montenegro to England’s Serbia” – but the difference is usually just a couple of % points. Plus the YouGov poll specifically looked at this issue, asking two separate questions and it made little difference to the results: 17% supported independence regardless of whether Scotland voted for it or not. So while the timing of the polls probably made some difference it’s unlikely to be the full story.

Different questions, different answers

The biggest reason for the gap between the two polls is almost certainly due to the way the questions were asked. ITV/YouGov asked a straight yes/no question:

if there was a referendum held tomorrow on Wales becoming an independent country and this was the question, how would you vote?

Should Wales be an independent country?

-Yes

-No

-Would not vote

-Don’t know

The BBC/ICM poll asked a very different question[3]:

Which of these statements comes closest to your view?

– Wales should become independent, separate from the UK

– The Welsh Assembly should have more powers than it currently has

– The powers it currently has are sufficient and should remain as it is now

– The Welsh Assembly should have fewer powers than it currently has

– The Assembly should be abolished and Wales governed directly from Westminster

– None of these

– Don’t Know

While only 3% chose independence, 49% chose more powers. It’s well understood that including more options in a survey question is likely to decrease the % opting for any given choice – the options compete against each other. For example, it seems likely that if the independence option were removed from the BBC survey, most of the 3% would transfer to the more powers option, and the headline result would have been “majority in favour of more powers”.

The poll probably also reflects the way people think about the issue as one which evolves over time. Wales is very far from being an independent country (it has considerably less autonomy than, say, American states or German länder) and it’s hard to imagine it becoming independent without several further steps of devolution/federalism happening first. I suspect many of the people answering “more powers” in the BBC poll would have chosen independence in the ITV poll, and the difference may indicate a reasonable preference for gradual change. Indeed, this suggests that the ITV poll, asking as it does about a referendum “tomorrow”, might actually underestimate support for independence.

Although it’s clear that the precise structure of the question matters, I’m not necessarily saying there’s a conscious bias here (unlike the Daily Wales website [4], which also makes another interesting point about question wording). There’s not necessarily a right or wrong way to do it, it depends what you are interested in. Clearly by offering an option including more powers, the BBC/ICM poll is more informative about that option and possibly more informative about independence: maybe some people supporting independence in the ITV poll would prefer greater devolution, but would take independence over the status quo[5].

But if you did want to rig a poll…

I do know that if I wanted a poll to find low support for independence, I’d include plenty of other options. Indeed adding a strong federal option to the BBC question could probably reduce support even further, attracting a few of the 3%. On the other hand, if I wanted to increase apparent support for independence I’d use a question like ITV’s, but ask something like “Do you think Wales should become an independent country within a generation”. I suspect that would find considerably more than 17% in favour (especially if one excluded the ‘don’t knows’/’wouldn’t votes’ from the headline figures, as is common). I could go one step further and do the first poll in the autumn (when risk aversion is high) and the second in the spring, and get an even bigger gap, or play around with the questions that precede the focal question:

Read the small print

So the answer to the question “What % support Welsh independence” could be less than 3% or a fair bit greater than 17%, depending on how we ask the question, when we ask the question, and how we analyse and present the results. The take home message is therefore that you should be very wary about headlines based on opinion polls: there’s just no substitute for looking at the small print yourself.

Update: Prof Roger Scully has a nice post on question wording and Welsh opinion polls here

[1] Fieldwork was conducted 8-11th September

[2] Field work was conducted 19-22nd September

[3] I’m grateful to Prof Roger Scully, of Cardiff University, for confirming the question wording.

[4] Daily Wales could well be right, but it’s hard to impute motives from survey questions alone, which is all I have – they might know more.

[5] However, given what we know about status quo bias, and given that more powers for the assembly are already on the cards, this seems unlikely

Advertisements

6 thoughts on “The importance of survey design: how many people really support Welsh independence?”

    1. Thanks Dafydd. It’s not an error, but I agree it’s confusing. What I mean is the figure could be less than 3% (if we used all the tricks to minimise the number) or more than 17% (if we did the reverse). i.e. we don’t really know! I’ve editing the wording to make this clearer.

  1. Hola! I’ve been following your web site for a long time now and finally got
    the bravery to go ahead and give you a shout out from Huffman Texas!

    Just wanted to mention keep up the good work!

  2. Find a reputation for the domain that relates perfectly in your product which is easily remembered.

    To increase the prevalence from the website and also to ensure
    it is appear around the paramount search pages, SEO technique is chosen by all
    the online business people. These engines like google base their search on keywords and links.

    I spent the final ten years without ever disclosing much information about
    myself on the Internet.

  3. Most of the filming was in Morocco, and Mohammad VI, King of Morocco, provided 1,
    500 of his military personnel with accompanying equipment to help in the filming.
    Whenever possible I try to use Pay – Pal rather then credit cards to pay as it is an added source of protection if I have
    problems with the product or sale. Used cars can be bought directly from
    the owner, from dealers, from used car superstores, or from
    auctions.

  4. When you learn search engine optimization with keywords it is important
    to make sure your content is diluted with them
    enough. There are lots of process in doing the SEO practices
    for a website but sometime SEO gives lots of pain due to use of un-ethical
    and spam SEO techniques. Google was not the first name they came up with, but it is now the first one on most people’s lips when asked to
    name a search engine.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s