A story published in Scotland on Sunday (SoS) on 7 March reported that a poll showed the Salmond Inquiry had “drive voters away from Scottish independence”.
The article noted that the poll found a “general preference for a No vote in any second referendum”, with support for No at 46 per cent, support for Yes at 43 per cent and don’t knows at 10 per cent.
That was a change from February, when the paper reported that Savanta ComRes put support for No at 42 per cent and for Yes at 47 per cent.
The SoS poll was carried out on 4 and 5 March, the two days immediately after First Minister Nicola Sturgeon appeared at the Salmond Inquiry.
The inquiry is being held to determine how the Scottish Government handled complaints of sexual misconduct against former First Minister Alex Salmond. It will also assess how Sturgeon acted in relation to the code of conduct for Scottish ministers.
Ferret Fact Service looked at how the poll was conducted and how its results can be interpreted.
What did the poll say?
Savanta ComRes is a market research company that carries out a range of voting intention polls on behalf of clients.
Last year SoS sister title The Scotsman commissioned it to do monthly polls looking specifically at how people intend to vote in any upcoming independence referendum. The first of these was completed in December.
As reported in SoS, the 7 March poll, which surveyed 1,015 people, put support for No at 46 per cent and support for Yes at 43 per cent, with don’t knows at 10 per cent. When undecided voters were stripped out, the figures were 52 per cent in favour of No and 48 per cent for Yes.
Why has it proved controversial?
Pollsters speak to a sample of people and use the data collected to make assumptions about a much larger group. In the case of election polling that larger group is the voting public.
In order to make sure that the small sample is representative of the larger group, pollsters must control for any biases that might arise if, for example, the sample had a lower proportion of women or people from ethnic minorities than in the population at large. That process is called weighting.
In its methodology, Savanta ComRes said the data it collected for the SoS poll was weighted to be representative of all Scottish adults by age, sex, region and how they voted in both the 2019 general election and the 2014 independence referendum.
However, it did not weight the data to take account of voter turnout, something that is often used to predict what the outcome of an election might be.
What is voter turnout weighting and why does it matter so much?
Voter turnout weighting controls data samples to take account of how likely people are to actually go out and vote.
Voter turnout varies widely, with 67.3 per cent of the electorate casting a vote at the last UK general election. Turnout at the 2014 independence referendum was much higher at 84.6 per cent, although that varied from a low of 75 per cent in Glasgow City to a high of 91 per cent in East Dunbartonshire.
Anthony Wells, director of political and social research at data analytics organisation YouGov, says that different assumptions, such as whether supporters of different parties will have different levels of turnout, can make a big difference to poll results.
By not using voter turnout weighting, the results of the SoS poll were based on the assumption that every single person eligible to vote would turn out to vote.
Is the Scotland on Sunday poll reliable?
Wells at YouGov says that because the data was weighted to take account of “all the usual” demographics it is a “legit poll”.
However, because the data was not weighted to take account of turnout the poll only gives an indication of how the Scottish population feels about independence, and may be less accurate than a turnout-weighted poll in indicating how the public would vote in a referendum. Savanta ComRes itself said that because it did not weight by turnout its survey “should not be treated as a headline voting intention”.
Wells also stressed that because this was a deviation from Savanta ComRes’s previous polls for The Scotsman, which did use turnout weighting, the results cannot be compared to the organisation’s earlier surveys.
University of Strathclyde polling expert Professor John Curtice, who maintains the What Scotland Thinks Twitter feed, noted that turnout weighting had “a particularly marked impact” on the February poll conducted by Savanta ComRes.
The February poll had support for Yes dropping from 51 per cent a month earlier to 47 per cent. Although the 7 March survey did show a drop to 43 per cent, that figure was measured on a different basis to the earlier ones. Curtice said it is therefore not possible to use the latest poll to conclude that the Salmond Inquiry has reduced support for a Yes vote.
Subscribe to the FFS show
The FFS Show is a podcast about truth and misinformation on the internet. Find it on Spotify, iTunes and elsewhere.
What do other Savanta ComRes polls say?
It is not clear why the SoS poll did not consider voter turnout, but The Scotsman published a new Savanta ComRes poll on 11 March that used the same weightings as the December, January and February surveys.
That poll, which interviewed 1,009 adults between 5 and 10 March, put support for a No vote at 47 per cent and support for Yes at 45 per cent with 8 per cent saying they didn’t know.
As that poll was conducted on the same basis as the initial three it should be considered a replacement for the one published on 7 March for the purposes of comparison.
The most recent poll does indicate that support for independence may have dropped following the appearance of both Salmond and Sturgeon at the Salmond Inquiry. However, the reduction in support for Yes continues a trend seen in each of the polls conducted for The Scotsman.
Photo thanks to iStock/Scott O’Neill
