Wide of the mark

A consideration of whether the poor reputation of seat polling is really deserved (short answer: yes).

Still no new polls, so let’s take a look at some old ones instead. After the 2016 election, I wrote an article for Crikey on the performance of the pollsters, particularly in regard to seat polls, and published here a chart showing the distribution of their errors. After being asked if the findings bore up over the seat polling conducted since, I have now conducted a similar exercise on seat polls conducted since the 2016 federal election, of which I identified 25 conducted in the final fortnight of various state elections and federal by-elections. However, rather than use the two-party results, which have separate issues of their own, I have produced separate results from Labor and Coalition primary votes. These can be found at the bottom of the post.

In the 2016 analysis, I concluded that the polls behaved more like they had a 7% margin of error than the 4% margin theoretically associated with polls sampling 500 to 600 respondents, as is typically the case with seat polls. It turns out that this chimes quite well with the polls conducted since. The mean error for the Coalition was +1.9%, which is to say the average poll had the Coalition that much too high high, while for Labor it was -0.5%. The difference is just inside statistical significance (the p-value on a two-sample t-test coming in at 0.047).

However, this does not mean you can confidently treat any given seat poll as biased to the Coalition, because their record is so erratic that any given poll could fall either way. The charts below record the spread of pollster errors (i.e. their result for a given party’s primary vote minus the actual result) as histograms, with two distribution curves laid over them – a thinner one in black, showing what the curve should theoretically look like with a 4% margin of error, and a thicker one in blue, showing their actual distribution. The lower and flatter the blue curve, the more erratic and unreliable were the results. As such, the charts show seat polls have been particularly wayward in predicting the Coalition primary vote. They have been somewhat nearer the mark with Labor, but still below theoretical expectations. The distributions suggest an effective margin of error for Labor of 6.5%, and for the Coalition of fully 9.5%.

It should be acknowledged, however, that a lot can happen over the last fortnight of an election campaign, and pollsters can always defend an apparent misfire by asserting that the situation changed after the poll was conducted. Perhaps significantly, the two worst performing polls in this analysis only barely fit within the two-week time frame. These were YouGov Galaxy polls from the Victorian “sandbelt” seats of Mordialloc and Frankston at the state election in November last year, crediting Labor with two-party votes of 52% and 51% in seats where the final results were 62.9% and 59.7%. If these cases are removed, the mean Coalition error comes down to +1.1% and the effective margin of error to 8.4%; while for Labor, the mean becames +0.1% and the margin of error 5.3%.

Author: William Bowe

William Bowe is a Perth-based election analyst and occasional teacher of political science. His blog, The Poll Bludger, has existed in one form or another since 2004, and is one of the most heavily trafficked websites on Australian politics.

831 comments on “Wide of the mark”

  1. WB
    “It should be acknowledged, however, that a lot can happen over the last fortnight of an election campaign, and pollsters can always defend an apparent misfire by asserting that the situation changed after the poll was conducted”

    Forgive my ignorance but couldn’t the pollsters test this by conducting a seat poll at the same time they conduct a full poll ( using different samples) or would this just prove how ineffective they are?

  2. WeWantPaul, I’ve repeatedly gone over and over the whole CPRS issue here on PB. Not really interested in going through all of it again hey haha.

    Same.

  3. Firefox;

    Yes, i acknowledge Greens have championed socially progressive issues, and part of the reason i will mix preferences between ALP and Green…

    My main point was that there are areas where the Greens are the best party to push for policy change and it benefits both society and themselves politically, but there are other areas where they conflict with Labor, and its destructive for both.

    Greens need to be more strategic about picking their battles if they want a good relationship with Labor, but some dont even want a good relationship…

  4. This article pretty much sums it up anyway and deals with some Labor myths

    Wait just a line above you said you didn’t want to discuss, so leave out your ‘guys who f*cked it up write webpages of self justification’ idiocy. Do you want to discuss the self justification of the guys who f*cked it up or do you not want to?

  5. Greensborough Growler says:
    Friday, April 26, 2019 at 9:48 am
    Di Natalie is unlikely to be the Greens Leader after the Election.

    Losing a couple of Senators will destroy his credibility.

    He’s uninspiring and has made a total hash of the job.

    I’m voting greens in the senate but by christ I feel bad about it with that turd in charge.

  6. Ana Navarro: Joe Biden is normal — and that sounds really good to me

    Why does this interest me as a registered Republican (yes, I know, I need therapy) who will not be voting in the Democratic primary?

    I care passionately about what happens in the Democratic primary because, before being a Republican or a Democrat, I am an American. I have one goal and only one goal: ousting Donald Trump from the White House.

    In the event you have been living on a desert island without WiFi for the last three years, let me fill you in: Donald Trump is intellectually, ethically and morally unfit to be president of the United States. He threatens the fiber and core of American values, embodying exactly the opposite of what makes America great.

    Joe Biden brings a lot to the table. He is everything Trump is not. He knows policy. He is a uniter. He calls for our better angels. He is empathetic and draws on his own grief to console and encourage others through theirs. He laughs easily. He is decent. He is devoted to his wife, and she is to him. Gold-leaf and wealth do not define him. He is not perfect and he has the humility to accept that.

    Joe Biden is not a shiny new toy. He is not flashy. He is not going to set hearts racing. He is comfortable, knowledgeable, experienced, reliable and dependable. We know what we’re getting with him. He inspires a feeling of trust and confidence.

    He is normal.

    After the chaos and turmoil of the last three years under Trump, “normal” sounds really good to me.

    https://edition.cnn.com/2019/04/24/opinions/joe-biden-is-normal—-and-that-sounds-really-good-to-me-navarro/index.html

  7. WWP

    The denial that it was Abbott that F’ed it up is what is so tiresome.

    All these fantasies of alternative realities will never change that.

  8. Onto more interesting further away topics, there are a lot of republicans running with the line that essentially the democrats MUST run a candidate whose sole purpose and sole policy is to defeat Trump. No other policy or purpose is acceptable. An explicit ‘well if they are going to run someone who has any progressive policy at all then I’ll be supporting Trump.’

    The Republic may linger but the good days are very over for our ANZUS ally.

  9. The denial that it was Abbott that F’ed it up is what is so tiresome.

    Yeah but Abbott was always an opponent, he WANTED it messed up. That the greens worked with him is what is difficult to understand. Lots of criticism for Labor working with Turnbull and lots of it is justified, but at least he didn’t team up with Abbott.

  10. pH_Red

    Ana Navarro: Joe Biden is normal — and that sounds really good to me

    His launch video is powerful. And clever. Deadly serious yet hopeful and uniting.

    He can beat Trump, no doubt. Can he win the primaries?

  11. PhoenixRed

    Hannity is too funny. What is actually happening is that fox want to cut him off for his communications with a Julian Assange etc. but of course, Hannity is throwing a red herring out there. Laugh out loud stuff.

    Also Anna Navarro is spot on. Exactly why I think Biden appeals across the board

  12. Barney in Da Lat @ #164 Friday, April 26th, 2019 – 10:12 am

    Believe it or not Peg, to get resources our Society needs to function sometimes requires mining. 😆

    Whose society needs uranium? Australia’s society generally doesn’t, not having any sort of nuclear industry. 🙂

    Let the societies who benefit from having uranium dig up their own land to get it. Australia doesn’t need to play that game. Or it should at least build some nuclear power plants and enrichment facilities (and waste disposal/storage facilities) if it’s going to play that game.

  13. Fess

    Yep. Trump was the change candidate and look what a shit show it has become.

    In order for stability to ensue, normal is needed.

  14. Vogon Poet says:
    Friday, April 26, 2019 at 10:16 am
    a r @ #166 Friday, April 26th, 2019 – 10:12 am

    slackboy72 @ #157 Friday, April 26th, 2019 – 10:03 am

    I’m voting greens in the senate but by christ I feel bad about it with that turd in charge.

    Second.

    Feel good
    don’t reward bad behaviour
    vote labor

    Yeah I’m faced with a choice. Support a greens led by Di Natale or possibly see an a-grade shit-c**t like Zed Seselja get back in as the second senator in ACT.

  15. Abbott

    When there is a raging lunatic on the loose, sensible people will adjust their bahviour in order to mange the risk. Pretending that one can simply carry on as if Abbott and his behaviour did not exist is childish and not the behaviour of people who should have anything to do with the executive government

  16. ‘How can you direct preferences to the man who let you down’ great question to Morrison and Liberal candidate re Palmer…..

    Morrison blustered with interference : we will not take policy advice from minor parties

    Not – we will not enter into preference deals with parties who let so many down

  17. Saw an interesting excerpt of James Campbell talking on Sky News about Clive Palmer. He mentioned that for him to deliver on his side of the preference deal, Palmer has to staff his polling booths. If you don’t have the volunteers to do this, this is extraordinarily expensive – say $20 per hour for 10 hours multiplied by the number of booths (someone will know how many). With Palmer, you’d be wanting the money up front.

    At my local booth – Merri Creek Primary in North Fitzroy – at the recent State election, the Liberals didn’t even have someone to hand out for themselves, let alone for a preference gatherer.

    I doubt that dealing with a character like Palmer will be going down well with the sort of voters who tossed out John Pesutto in Hawthorn, wiped the Liberals from the face of the earth in Albert Park and nearly turned Caulfield and Brighton into Labor seats.

  18. Just saw Scummos Presser from Townsville. From what I saw and heard, particularly from questions what I gather to be from the local reporters then the seat of Herbert will be retained by the Sitting Labor Member.

    The Fibs hopping into bed with Clive will see to that.

    Hostile questions indeed.

  19. A very recent (tho pre campaign launch) Politico/Morning Consult poll (a large sample) had Biden ahead of Trump by 8 pts (excluding undecideds).

    A Biden-Klobuchar ticket would be Trumps worst nightmare. But I doubt that would work.

    Bidens running mate choice will be more important than usual.

  20. “Yeah but Abbott was always an opponent, he WANTED it messed up. That the greens worked with him is what is difficult to understand. Lots of criticism for Labor working with Turnbull and lots of it is justified, but at least he didn’t team up with Abbott.”

    As a bystander in the eternal arguments over who stuffed up emissions negotiations in the Rudd I era, I’d say both sides neglect the degree to which both the Greens and the ALP underestimated the ferocity and sheer petty bloody mindedness that Abbott would bring to destroying whatever was eventually agreed upon.

    By which I mean that I don’t think it mattered a jot whether the original CPRS got up or not. What we know from the perspective of 2018 is that Abbott’s only political skill is tearing stuff down. He was making trouble for Turnbull as opposition leader before he rolled him over emission reductions, and I reckon it’s pure fantasy to think that if the CPRS had been passed, he would have quietly accepted it; he was going to roll Turnbull at some point anyway, or at least try, and if he’d succeeded, he would have cheerfully ditched Liberal support for the CPRS, campaigned against it as a carbon tax, and quite possibly we’d have ended up at a similar place to where we are now. I don’t think that either the ALP or the Greens appreciated that at the time – Abbott was known to be a bit crazy, but thought to be ineffective. We all know how that turned out.

    We can argue all we like about whether the Greens or the ALP misstepped, but Abbott has been a uniquely destructive influence in Oz politics and most of the counterfactuals presented don’t really take this into account. No matter what was passed or not passed, he was never going to sit idly by, and with the sort of the Murdoch press may have succeeded in wrecking whatever was achieved by anyone.

  21. SK

    Normal is white old tied to the establishment.

    Biden is at his peak polling now.

    The most impressive mover so far is Mayor Pete.
    I will repeat however.

    Normal it is not. Different rules for nomination. 20 candidates so far

  22. Steve – Hard to believe that Clive will spend a penny on manning polling booths. Why would he bother? He doesn’t give a rats if the libs don’t get his preferences. All he cares about are the prefs going the other way. If the Libs think they can trust his word, they are totally absolutely bonkers.
    Itep – Very true. And won’t that piss of poll workers AND liberal voters. Here’s one from our party and here’s one from the fat crook.

  23. In any case, things are not going to get better for Trump over next few months. Lots still to play out. The GOP are going to be faced with even more crapola going forward.

  24. I believe that the Greens could lose as many as three senators at the federal election. Larissa Waters in Queensland, Sarah Hanson Young in South Australia and Mehreen Faruqi in New South Wales.

  25. Itep,

    My point is that if the Libs can’t properly staff booths for themselves then they will struggle to provide help to anyone else. Ours is not a huge booth, but 1350 people voted there without a Liberal to be seen.

  26. Red13 says:
    Friday, April 26, 2019 at 9:21 am
    Guytaur and others, the endless repetitive flame wars achieve what exactly?
    Why don’t you all wait and see what happens on 18 May?
    Then, if you wish, have at it. Until one knows the political landscape after 18 May all of this to and fro is useless.
    That being said, I bet not one Party negotiator from any side takes any notice of what is said on PB in their deliberations.
    Give it a rest, there are bigger fish to fry along the road to May 18.
    Think about and discuss that more often. I’d like to hear more on what is happening on the ground in your various seats. Discuss what is happening and your take on it.
    That would be far more interesting to digest as we await further polling.

    Good advice Red13, but totally wasted on those who seem to think that boring the rest of us shitless day after day is a good way to spend their waking hours. Best to just scroll past and search for the more informative comments.

  27. SK
    Normal is white old tied to the establishment.
    Biden is at his peak polling now.
    The most impressive mover so far is Mayor Pete.
    I will repeat however. Normal it is not. Different rules for nomination. 20 candidates so far

    Are you Trumps twitter writer?
    I like Buttigieg. I like Sanders. I like Warren. (Undecided about Harris and Booker). All these people are important to the Democrat Party and they may yet out shine Biden in the debates and primaries.
    I didnt mention ‘normal’ in my posts and I dont see the relevance. What is relevant is who can win swing states to win the Presidency. Progressives can; if they get enough peeps enrolled and out to vote. But the flip side; progressives may also help to get the Trump base out to vote.

    Ideally you have a candidate that has broad appeal who also listens to the progressive arm of the party. If that is ‘normal’ then…. super.

Leave a Reply

Your email address will not be published. Required fields are marked *