The Poll Bludger

From my paywalled article in Crikey yesterday:

In the wake of its most unambiguous failure at a federal election since at least 1980, Australia’s polling industry is licking its wounds.

The Nine/Fairfax papers have announced the Ipsos poll series will be put on ice, and those pollsters who do return to the field shortly will face catcalls whether they persist in recording a Labor lead we now know doesn’t exist, or only now start detecting a Coalition lead that eluded them through the entirety of the past parliamentary term.

Despite it all though, the pollsters’ performance hasn’t been without its defenders.

Spoiler alert: the latter refers to David Briggs and Nate Silver. But Peter Brent can now be added to the list, up to a point, following a review of the issues raised by the polling failure in Inside Story. Specifically, Brent observes that the primary vote miss was less severe than the two-party preferred; that the difference arose from a stronger-than-anticipated flow of minor party and independent preferences to the Coalition; that herding was less apparent on the primary vote (most markedly in the case of Ipsos’s reading of the balance of support between Labor and the Greens); and that the result was, if nothing else, no worse than the Victorian state election.

Another point noted is the strange consistency with which polls have pointed to extravagant gains for Labor in Queensland before and during election campaigns, only for them to fall away at the end. On this occasion, the falling away as recorded by pollsters wasn’t remotely on the scale needed to predict the result, with statewide polling published towards the end of the campaign landing at least 7% shy of what looks like being the Coalition two-party vote in the state.

The question of geographic variability in the pollster failure seemed worth exploring, so I have put together a table of state and electorate level polling published in the last fortnight or so of the campaign, available below the fold at the bottom of the post. Almost all of this polling was conducted by YouGov Galaxy, whether under its own name or as Newspoll. The only exception was a set of state-level two-party preferred totals from Ipsos, published at the tail end of the campaign by the Age-Herald (which performed rather poorly).

Below all this is a list of “average bias” figures, consisting of straight averages of the observed errors, be they positive or negative, rather than the absolute errors. This means combinations of positive and negative results will have the fact of cancelling out — although there were actually very few of those, as the errors tended to be consistently in the one direction. The national and state-level two-party results are estimates provided to me by Nine’s election systems consultant David Quin. With no Coalition-versus-Labor figures available from 15 electorates, this inevitably involves a fair bit of guess work.

A few points should be observed. Given that poll trends pointed to a clear long-term trend to the Coalition, pollsters may be excused a certain amount of Labor bias when evaluating polling that was in many cases conducted over a week before the election. This is particularly true of the Newspoll state aggregates, which cover the full length of the campaign.

Another issue with the Newspoll state aggregates is that One Nation was a response option for all respondents in the early part of the campaign, despite their contesting only 59 out of 151 seats. Their vote here accordingly comes in too high, and as Peter Brent notes, at least part of their failure could be explained by stranded One Nation supporters breaking in unexpectedly large quantities to the Coalition, rather than other minor party targets of opportunity like Clive Palmer.

In seat polling though, where the issue did not arise, the polls were remarkable in having understated support for One Nation, and overstated it for the United Australia Party. This was one face of a two-sided polling failure in Queensland, of which the other was a serious imbalance towards Labor in support recorded for the major parties. While Queensland has caught most of the attention on this score, the polls were just as far out in measuring the primary votes of the major parties in Western Australia. Things were less bad in Victoria, but Coalition support was still significantly underestimated.

The only bright spots in the picture are New South Wales and South Australia, where Newspoll just about nailed the Coalition, Labor and Greens primary votes, and got the big things right in four seat polls. While Labor’s strength was overstated in Macquarie, it does now appear Labor will pull through there – for more on that front, stay tuned to the late counting thread.

Continue reading “Of swings and misses: episode three”

At last, some commentary on the Senate count. Only one of the results is in doubt, with New South Wales, Victoria, Western Australia and South Australia all turning in good old-fashioned results of Coalition three, Labor two and Greens one, Jacqui Lambie snaring a seat in Tasmania at the expense of a third Liberal, and the territories behaving as they always do.

The exception is Queensland, where the third Liberal National Party candidate, the second Labor candidate, and the first candidates of the Greens and One Nation are in a game of musical chairs in which one will miss out when the music stops. In actuality, LNP and One Nation are pretty much home and hosed at this point, so the issue is whether the last seat will go Labor or Greens.

Based on the primary vote, it looks like Labor will miss out, reducing them to a single Senate seat, for which the only precedents are the Western Australia and South Australian results in 2013. Labor has only 0.60 quotas spare after the election of its first candidate, adrift of the LNP on 0.76, One Nation on 0.72 quotas and the Greens on 0.68. Probably the leading authority on the count is Ross Leedham on Twitter, who it appears expects Labor to narrow the primary vote gap a little on late counting, and then to take it up to the Greens with preferences.

To get a sense of how preferences are likely to behave, I have wrangled with the data file from the 2016 election, to produce “four-party preferred” measures for the various minors and micros who will be excluded from the count. This only uses above-the-line votes, and uses Australian Liberty Alliance preferences for Fraser Anning’s party, Family First’s for Australian Conservatives, the Renewable Energy Party’s for two micros with “climate” in their names, and the Health Australia Party for an anti-vaxxer party. Some very small parties that couldn’t be matched are ignored.

With these results used to project the preferences of the small parties, the Greens’ lead over Labor extends slightly, from 1.1% to 1.3%. This is somewhat contrary to the assessment of Kevin Bonham, who used preference guesstimates to conclude Labor would close the gap but not by enough. These estimates look like they might be on the high side for Labor, and I have further credited the Greens with some fairly heavy duty preference flows from micro-party minnows Bonham hasn’t bothered with.

Two complications arise from the United Australia Party, which will be the last party excluded before the conclusion. One is their how-to-vote card, which recommended a second preference to the LNP and a lower order preference to Labor. This renders unreliable the projection I have extrapolated from the tiny Palmer United Party vote in 2016, which gives One Nation too many Palmer preferences and the LNP too few. However, I’m projecting both to do well enough to win seats in any case.

The other is the possibility raised by Kevin Bonham that either the LNP or One Nation will make a quota before the UAP is excluded. If the former, UAP votes following the how-to-vote card will end up with Labor instead of the LNP, potentially making them competitive in the race against the Greens. However, I’m projecting the LNP to be fairly well short of a 14.3% quota with 13.0% at the point where the UAP are excluded, with One Nation also just shy at 14.0%.

With all that in mind, I’m going to work on the basis of a result of Coalition three and one seat each for Labor, One Nation and the Greens in offering the following summary of the state of the Senate post-election.

New South Wales. With the Coalition on 2.72 quotas, Labor on 2.11 and the Greens on 0.60, the result here looks sure to be three, two and one seats respectively, unless One Nation on 0.35 quotas can do something astounding on preferences. Jim Molan has clearly failed in his bid to have below-the-line votes overturn his demotion on the Senate ticket – he appears to be getting about 80% of the below-the-line votes, which could be generously estimated to account for 10% of the total. That would leave him with about 3% of the vote, or 0.2 quotas, to take on Nationals candidate Perin Davey, who will have the 0.7 or so quota surplus left after the election of the second Liberal.

Victoria. Very similarly to New South Wales, the Coalition are on 2.54 quotas, Labor on 2.23 and the Greens on 0.70, guaranteeing a result of three, two and one. Way behind the Greens on 0.20 quotas apiece are Derryn Hinch’s Justice Party and One Nation. Hinch’s failure is something of a surprise, his 2.8% vote share being well below the 3.7% recorded by his party at the state election last November. Presumably the 2.5% United Australia Party vote, modest as it was, came largely at his expense.

Western Australia. A clean result of Liberal three (2.93 quotas), Labor two (1.96 quotas) and Greens one (0.82 quotas). One Nation polled a reasonably solid 5.5%, but not nearly as well as the Greens, who recorded an insurmountable 11.7%.

South Australia. Three Liberal (2.63 quotas), two Labor (2.16) and one Greens (0.78), with One Nation too far behind the pace on 0.32.

Tasmania. Two Liberal (2.21 quotas), two Labor (2.17 quotas), one Greens (0.88) and one Jacqui Lambie (0.61).

The Sydney Morning Herald and The Age – or the Herald/Age, to adopt what is evidently Nine Newspapers’ own preferred shorthand for its Sydney and Melbourne papers – have revealed their opinion polling will be put on ice for an indefinite period. They usually do that post-election at the best of times, but evidently things are more serious now, such that we shouldn’t anticipate a resumption of its Ipsos series (which the organisation was no doubt struggling to fund in any case).

This is a shame, because Ipsos pollster Jessica Elgood has been admirably forthright in addressing what went wrong – and, importantly, in identifying the need for pollsters to observe greater transparency, a quality that has been notably lacking from the polling scene in Australia. In particular, Elgood has called for the establishment of a national polling standards body along the lines of the British Polling Council, members of which are required to publish details of their survey and weighting methods. This was echoed in a column in the Financial Review by Labor pollster John Utting, who suggests such a body might be chaired by Professor Ian McAllister of the Australian National University, who oversees the in-depth post-election Australian Election Study survey.

On that point, I may note that I had the following to say in Crikey early last year:

The very reason the British polling industry has felt compelled to observe higher standards of transparency is that it would invite ridicule if it sought to claim, as Galaxy did yesterday, that its “track record speaks for itself”. If ever the sorts of failures seen in Britain at the 2015 general election and 2016 Brexit referendum are replicated here, a day of reckoning may arrive that will shine light on the dark corners of Australian opinion polling.

Strange as it may seem though, not everyone is convinced that Australian polling really put on all that bad a show last weekend. Indeed, no less an authority than Nate Silver of FiveThirtyEight has just weighed in with the following:

Polls showed the conservative-led coalition trailing the Australian Labor Party approximately 51-49 in the two-party preferred vote. Instead, the conservatives won 51-49. That’s a relatively small miss: The conservatives trailed by 2 points in the polls, and instead they won by 2, making for a 4-point error. The miss was right in line with the average error from past Australian elections, which has averaged about 5 points. Given that track record, the conservatives had somewhere around a 1 in 3 chance of winning.

So the Australian media took this in stride, right? Of course not. Instead, the election was characterized as a “massive polling failure” and a “shock result”.

When journalists say stuff like that in an election after polls were so close, they’re telling on themselves. They’re revealing, like their American counterparts after 2016, that they aren’t particularly numerate and didn’t really understand what the polls said in the first place.

I’m not quite sure whether to take greater umbrage at Silver’s implication that Antony Green and Kevin Bonham “aren’t particularly numerate”, or that the are – huck, spit – “journalists”. The always prescient Dr Bonham managed a pre-emptive response:

While overseas observers like Nate Silver pour scorn on our little polling failure as a modest example of the genre and blast our media for failing to anticipate it, they do so apparently unfamiliar with just how good our national polling has been compared to polling overseas.

And therein lies the rub – we in Australia have been rather spoiled by the consistently strong performance of Newspoll’s pre-election polls especially, which have encouraged unrealistic expectations. On Saturday though, we saw the polls behaving no better, yet also no worse, than polling does generally.

Indeed, this would appear to be true even in the specifically Australian context, so long as we take a long view. Another stateside observer, Harry Enten, has somehow managed to compare Saturday’s performance with Australian polling going all the way back to 1943 (“I don’t know much about Australian politics”, Enten notes, “but I do know something about downloading spreadsheets of past poll data and calculating error rates”). Enten’s conclusion is that “the average error in the final week of polling between the top two parties in the first round” – which I take to mean the primary vote, applying the terminology of run-off voting of the non-instant variety – “has been about five points”.