Wide of the mark

A consideration of whether the poor reputation of seat polling is really deserved (short answer: yes).

Still no new polls, so let’s take a look at some old ones instead. After the 2016 election, I wrote an article for Crikey on the performance of the pollsters, particularly in regard to seat polls, and published here a chart showing the distribution of their errors. After being asked if the findings bore up over the seat polling conducted since, I have now conducted a similar exercise on seat polls conducted since the 2016 federal election, of which I identified 25 conducted in the final fortnight of various state elections and federal by-elections. However, rather than use the two-party results, which have separate issues of their own, I have produced separate results from Labor and Coalition primary votes. These can be found at the bottom of the post.

In the 2016 analysis, I concluded that the polls behaved more like they had a 7% margin of error than the 4% margin theoretically associated with polls sampling 500 to 600 respondents, as is typically the case with seat polls. It turns out that this chimes quite well with the polls conducted since. The mean error for the Coalition was +1.9%, which is to say the average poll had the Coalition that much too high high, while for Labor it was -0.5%. The difference is just inside statistical significance (the p-value on a two-sample t-test coming in at 0.047).

However, this does not mean you can confidently treat any given seat poll as biased to the Coalition, because their record is so erratic that any given poll could fall either way. The charts below record the spread of pollster errors (i.e. their result for a given party’s primary vote minus the actual result) as histograms, with two distribution curves laid over them – a thinner one in black, showing what the curve should theoretically look like with a 4% margin of error, and a thicker one in blue, showing their actual distribution. The lower and flatter the blue curve, the more erratic and unreliable were the results. As such, the charts show seat polls have been particularly wayward in predicting the Coalition primary vote. They have been somewhat nearer the mark with Labor, but still below theoretical expectations. The distributions suggest an effective margin of error for Labor of 6.5%, and for the Coalition of fully 9.5%.

It should be acknowledged, however, that a lot can happen over the last fortnight of an election campaign, and pollsters can always defend an apparent misfire by asserting that the situation changed after the poll was conducted. Perhaps significantly, the two worst performing polls in this analysis only barely fit within the two-week time frame. These were YouGov Galaxy polls from the Victorian “sandbelt” seats of Mordialloc and Frankston at the state election in November last year, crediting Labor with two-party votes of 52% and 51% in seats where the final results were 62.9% and 59.7%. If these cases are removed, the mean Coalition error comes down to +1.1% and the effective margin of error to 8.4%; while for Labor, the mean becames +0.1% and the margin of error 5.3%.

Author: William Bowe

William Bowe is a Perth-based election analyst and occasional teacher of political science. His blog, The Poll Bludger, has existed in one form or another since 2004, and is one of the most heavily trafficked websites on Australian politics.

831 comments on “Wide of the mark”

  1. this is such a dangerous time for australia. a clown leads an almost empty front bench with no ideas and at mercy of special interests – and is still fighting.

    i can’t remember such a time – esp with challenges so great for country

    this country boasts economic credentials – he is illiterate in that regards

  2. Success – so anyone who wants to show examples of html tags, you can use ampersand then “lt;” (that’s a lower-case L not a one) and it will be rendered as a visible “less than” symbol or angled bracket. Nerd-son says you can close the tag with the usual greater-than because html interpreter won’t have seen a start-of-tag symbol and will just read text as it is.

  3. Every time a Labor shadow minister appears he or she should say that it would be good if the public could see a contest of ideas in debate with their government counterpart minister/spokesperson.

  4. hungry jack

    Well, 5.

    1) Ask your neighbour.

    2) We don’t know the result yet.

    3) Think I just heard one. It’s not late, we have a long time to go.

    4) We’ve had a few leaders with no union background – however, all Laobr MPs must affiliate with a union. The party arose out of the union movement, after all.

    5) Someone has to put their hands up for the job, persuade people they’re capable of doing it, and get enough votes in the party room.

  5. Does anyone here think for one second that if things were looking up for the Coalition that Morrison and the Coalition’s media backers would’ve been running around demanding more debates? Since when do incumbent Prime Ministers want multiple debates with an opposition leader? This all tells me they’re even more toast than I thought.

  6. hungry jack @ #500 Friday, April 26th, 2019 – 6:03 pm

    four questions/mysteries
    1) why is the opposition leader so unpopular and for so long (dont blame murdoch – ask your neighbour) ?
    2) why is result not 55-45 despite murdoch and because of dearth of ideas and people in govt
    3) when do labor attack ads appear (it’s getting late)
    4) why is union affiliation such an advantage in leading labor – why can’t party be lead by someone with business or even education background, likewise liberals with someone from union?
    5) why can’t we have a dynamic younger even leader?

    My answers – others may have different perceptions.
    1. Shorten has been subject to an unprecedented campaign of vilification including a Royal Commission
    2. Baffles me too. Can only put it down to persistent anti-labor propaganda and a substantial body of rusted on anti-labor voters.
    3. Patience grasshopper. 3rd parties are already running some.
    4. The party has had leaders who have not been union leaders e.g. Rudd, Gillard, Latham, Beazley, Keating, Whitlam etc. Union leaders are the exception rather than the rule.
    5. How young do you want to go? Recent leaders have hardly been dotards.

  7. Jack Aranda @ #504 Friday, April 26th, 2019 – 6:08 pm

    Success – so anyone who wants to show examples of html tags, you can use ampersand then “lt;” (that’s a lower-case L not a one) and it will be rendered as a visible “less than” symbol or angled bracket. Nerd-son says you can close the tag with the usual greater-than because html interpreter won’t have seen a start-of-tag symbol and will just read text as it is.

    There is a problem with it though if someone quotes the text that contains the example, as it then renders and the example is lost. The only way I’ve found to show HTML examples reliably is using an image, typically screen captured out of notepad. 🙂

  8. Matt31 @ #507 Friday, April 26th, 2019 – 6:11 pm

    Does anyone here think for one second that if things were looking up for the Coalition that Morrison and the Coalition’s media backers would’ve been running around demanding more debates? Since when do incumbent Prime Ministers want multiple debates with an opposition leader? This all tells me they’re even more toast than I thought.

    +1

  9. Is the Liberal party commissioning private polling to prevent the pollsters releasing the bad news on national vote trend? 😉

  10. Zoomster Bemused was the first PB’er to treat me like a misguided Liberal requiring training rather than an outsider or troll. Now I just feel at home 🙂

  11. OH and I just watched a Liberal Party attack ad against Shorten. (Tax, Tax, Tax)

    Impressions
    * Me: That ad doesn’t do what they think it does. It made them look silly.
    * OH: That was a very cheap ad. They must be broke.

    Thought
    * They don’t have Turnbull’s money this time, maybe that’s where Palmer comes in.

  12. Dubbo? Interesting.

    The Shooters, Fishers and Farmers arent contesting Dubbo (Division of Parkes) but are contesting Calare next door (Orange).

  13. Late Riser @ #521 Friday, April 26th, 2019 – 6:22 pm

    OH and I just watched a Liberal Party attack ad against Shorten. (Tax, Tax, Tax)

    Impressions
    * Me: That add doesn’t do what they think it does. It made them look silly.
    * OH: That was a very cheap ad. They must be broke.

    Thought
    * They don’t have Turnbull’s money this time, maybe that’s where Palmer comes in.

    If I was preparing the ALP ads, and they are probably lucky I am not, I would subject such nonsense to good humoured ridicule.

  14. C@

    You can recognise posters by their obsessions and their shortcomings.

    bemused thought he knew everything there was to know about mental health services, hence the gratuitous dig at me earlier, because (years ago) I pointed out that there were mental health services operating he didn’t believe existed (I have visited them, and have family working in them).

    No one else is obsessive enough – or petty enough – to make that remark about me.

  15. EGW @ #496 Friday, April 26th, 2019 – 5:59 pm

    yabba @ #489 Friday, April 26th, 2019 – 5:53 pm

    ……..
    Supposedly sane men and women are responsible for more than 50 times more deaths each year than are mental patients (suicides excluded).

    Any breakdown of treated vs untreated?
    I understand the danger mainly come from psychotic patients who are untreated or not taking their medication.
    Unfortunately a lot of civil libertarians seem to be OK if patients die with their rights on. Sometimes a degree of compulsion is necessary when a patient lacks insight.

    EGW, who are you proposing should do the research into the bizarre questions that you ask? How in hell is anybody supposed to know how many untreated sufferers of psychotic mental illness there are?

    Should we get Morgan to do a face to face poll?

    Some problems are just plain HARD. Not all problems have solutions. Of the order of 5 to 10 people a year are killed in Australia by sufferers of psychosis. Around 1150 a year die in road accidents. Around 1400 a year die from accidental drug overdoses, mostly from prescription drugs. Around 4500 people suicide each year, of whom a massively disproportionate number are our first nations’ young people.

    Please, just stop pontificating and start doing something about a problem, any problem, that you think is important, instead of just bullshitting.

  16. Well if nothing else mate, it demonstrates just how obsessed some on here have been about the good ole bemused. Talk about jumping at shadows.

  17. bemused thought he knew everything there was to know about mental health services, hence the gratuitous dig at me earlier, because I pointed out that there were mental health services operating he didn’t believe existed (I have visited them, and have family working in them).

    bemused/EGW also had a dig at you a week or so ago, having only just started commenting here (again), that time about you as a teacher and a comment you made earlier that morning about schools. And then accused me of trolling the next day even though we’d had no interaction under his new screen name of EGW. The same MO and the same patterned targeting of women.

    It’s doubtful that a supposed long time lurker would burst onto the blog gratuitously sledging other commenters unless they were setting themselves up as a nath/wayne troll.

  18. briefly

    I was already contemplating a “Joan Kirner was the greatest education reformer ever’ type remark to test my suspicions, but now I don’t need to.

  19. It’s doubtful that a supposed long time lurker would burst onto the blog gratuitously sledging other commenters unless they were setting themselves up as a nath/wayne troll.
    _________
    I resent that, wayne never sledges other commenters. Actually, I rarely do either. Less than others anyway.

  20. I wonder if there is a correlation between the direness of the political news for the Coalition and Wayne popping up with his light relief.

  21. Tom

    Well, he wouldn’t have done it just to post here, but plenty of people are changing IPs because they’re getting connected to the NBN.

  22. yabba @ #530 Friday, April 26th, 2019 – 6:33 pm

    EGW @ #496 Friday, April 26th, 2019 – 5:59 pm

    yabba @ #489 Friday, April 26th, 2019 – 5:53 pm

    ……..
    Supposedly sane men and women are responsible for more than 50 times more deaths each year than are mental patients (suicides excluded).

    Any breakdown of treated vs untreated?
    I understand the danger mainly come from psychotic patients who are untreated or not taking their medication.
    Unfortunately a lot of civil libertarians seem to be OK if patients die with their rights on. Sometimes a degree of compulsion is necessary when a patient lacks insight.

    EGW, who are you proposing should do the research into the bizarre questions that you ask? How in hell is anybody supposed to know how many untreated sufferers of psychotic mental illness there are?

    Should we get Morgan to do a face to face poll?

    Some problems are just plain HARD. Not all problems have solutions. Of the order of 5 to 10 people a year are killed in Australia by sufferers of psychosis. Around 1150 a year die in road accidents. Around 1400 a year die from accidental drug overdoses, mostly from prescription drugs. Around 4500 people suicide each year, of whom a massively disproportionate number are our first nations’ young people.

    Please, just stop pontificating and start doing something about a problem, any problem, that you think is important, instead of just bullshitting.

    Not pontificating or bullshitting or saying it is not hard. All that you described was good and commendable.
    I have closely followed press reports of homicides involving mentally ill people and also read up on Coronial inquests and attended some.
    Check out the work of the ‘Treatment Advocacy Center’ in the US and the writings of E. Torrey Fuller.
    https://www.treatmentadvocacycenter.org/about-us/dr-e-fuller-torrey

    The late Dr Jean Lennane used to write similar things in NSW and was probably influenced by him.

    I suggest a lot more than 5 – 10 people a year are killed in Australia by people with a psychosis. I think there would be more than that in Victoria alone.

    Also, there are a lot of road fatalities that look suspiciously like suicides when a car hits a tree with no signs of braking and other odd circumstances.

    I am not looking for an argument and I take the opportunity to learn from your experience.

  23. I remember Poss once saying that as a moderator at his then place, he could even determine the type of computer screen someone was using to comment.

  24. And I was enjoying not having a resident misogynist around all day, every day.

    Interesting that only male contributors have welcomed him back. However, understandable.

  25. zoomster @ #642 Friday, April 26th, 2019 – 4:40 pm

    briefly

    I was already contemplating a “Joan Kirner was the greatest education reformer ever’ type remark to test my suspicions, but now I don’t need to.

    LOL I was considering a ‘Christine Nixon was the greatest police chief any state or territory has ever had’ for precisely the same reason!

  26. clem attlee @ #531 Friday, April 26th, 2019 – 6:36 pm

    Well if nothing else mate, it demonstrates just how obsessed some on here have been about the good ole bemused. Talk about jumping at shadows.

    I think you’re on to something there Clem.
    I enjoy your comments.
    It is really surprising how those who bleat loudest and longest about others are usually much worse than those they complain about. Good how they self identify.

  27. What is even more ‘understandable’ is our shared deeply held conviction that the current bunch of arseholes are chucked out in 3 short weeks.

    Why just leave it at that hey ?

    Surely that enough to go on with ?

Leave a Reply

Your email address will not be published. Required fields are marked *