Commenters at the business end of the country inform me, via Lateline, that tomorrow’s Newspoll will be as you see in the headline (after a 56-44 result a fortnight ago). It is against my religion to read anything into one poll in isolation. Nonetheless, I am tempted to interpret this as the interest rate hike being cancelled out by what Matt Price describes as the government’s “potentially quite good bad news”, namely last week’s stock market dive.
UPDATE: Kevin Rudd’s lead on preferred prime minister has widened from 44-39 to 46-39.
UPDATE 2: The Australian reports Labor’s primary vote is down from 48 per cent to 46 per cent (equal lowest since February), with the Coalition steady on 39 per cent.
UPDATE 3: Graphic here, Shanahan here.
Rudd has yet to issue a response to this vital issue:
http://www.theage.com.au/articles/2007/08/21/1187462218754.html
Possum: Thanks for your reply. I will have to discuss further with you. I think I understand most of it, although statistics was always the weakest part of my (if I may say so myself) otherwise strong maths education. I should certainly brush up on variance analysis 🙂
In any case, if I read you correctly (and you certainly said as much way back upwards) you agree that the “trend to the government since March” is almost entirely an artefact of the analysis itself.
Its amazing that for all the claims of the Liberals economic superiority from Liberal supporters. We have so many people who fail to understand economics 101 and the effect of the global economy on the Australian economy during times of recession and boom.
I try to be generous, but for the lifge of me I can never work out whether the kinder interpretation is that the Liberal cheersquad are ignorant of basic economic history, or that they are aware of these facts but lie to achieve political points.
Doug at 146
sorry about my indecipherable wordiness 😉
In the same way that if the ALP gets 46 then 48 in two consequetive polls, the next poll is more likely than not to be less than 48.
If the monthly average of newspolls were used instead where the ALP got, say, 46 in one month, 48 in the next month – the following month is also likely to be less than 48.That relationship is statistically significant (as is the poll to poll relationship) but its not as strong in the month by month comparisons as it is in the poll by poll comparisons.
Martin at 148 said:
“In any case, if I read you correctly (and you certainly said as much way back upwards) you agree that the “trend to the government since March†is almost entirely an artefact of the analysis itself.”
That’s what my Newspoll and Morgan data tell me – not only with the trend to the government SINCE March, but also the trend to the Opposition OVER Feb and March.The method of analysis you use will determine the results you get.
Marginally on topic, but Simon Jackmans piece is worth having a giggle at:
http://jackman.stanford.edu/blog/?p=298
So many of you think Labor are competent with the economy…well can any of you tell me what Labor’s tax policy is???
No Spin Zone….
Asanque – I agree. I’m not a qualified economist but I have run businesses over the past 20 years and I have certainly passed Economics 101. No Australian government can claim expertise for running the economy. We are a small fish in a very big sea full of unknowns such as sharks and economic tsunamis. Howard’s ‘promise’ in 2004 to keep interest rates at record lows was a silly thing to say as he knew then he could do no such thing. The RBA adjusts interest rates to try to control inflation but if the global economy causes interest rate rises (as is happening now) there is nothing the RBA or any government can do about it.
Economically, this might just the election to lose for Labor. The US economy relies on consumer spending for about 70% of its activity. If that goes down too far (it’s falling now) imports from China will slow dramatically, the Chinese economy will slow resulting in less need for Australia’s minerals and energy and our economy will contract. We could have a recession. It would be a pity (from the Labor side) to win this year’s election just to have the economy tank and then be thrown out after one term.
I heard Gary Gray say the same thing when he was National Secretary of the ALP. I assumed that he would know what he was talking about on that point.
Another thing I remember his saying (I think on the same occasion) was that a great achievement of the Coalition in the lead-up to the 1996 election was convincing people that their lead was ‘soft’. He said that it hadn’t been ‘soft’, it had been ‘rock-hard’. On this point, however, certain knowledge is unavailable and I don’t assume that Gary Gray must have been right.
It seems to me, however, that journalists and commentators pretty much have to write _as if_ things are changing all the time (whether or not they actually think so, and whether or not they have any reason to think so). If they started saying ‘it’s all over bar the shouting, nothing can change what happens from here on’ they would also be saying ‘nothing to see here, folks’ and obviously that won’t sell newspapers.
Unlike many regular posters, I find the work of Aristotle and Possum breathtakingly banal and I think the accompanying posts which praise their work and agree with their conclusions, are not particularly persuasive as to the election outcome. Yes, the Australian Labor Party is well ahead on the opinion polling data conducted since Mr. Rudd took over the leadership of its parliamentary party, and yes, the 2PP and primary voting intentions have been in a consistent range since the start of the year. Moreover, the portents (like the negative aspects of the Government’s workplace relations laws, ennui with JWH, the time for a change factor and the estimation that the PM will be outperformed in a television debate by the alternative PM) all augur well for an ALP win. However, I would not like to see any wishful thinking on either side of politics get in the way of an informed analysis. Indeed, I think it was Gary Bruce on a prior thread who identified what will be the deciding issue in interpreting, correctly, opinion polling trends: that is, whether the Coalition parties can take primary votes from the ALP so that a dip in the ALP primary vote is matched by a rise in the Coalition primary vote. I respectfully agree with that point of view. As matters presently stand, I cannot be as confident as others (say, STROP at one extreme, or Glen at the other extreme) about who is going to win the election. I do know we will have a much better idea of the likely winner when we see opinion polling data after the election date is announced and in the three or four weeks leading up to the poll itself. That is when there will need to be a serious tracking of primary voting intentions between the ALP and the Coalition, and that is also when I am likely to find any predictions of Aristotle, Possum and any others based on that tracking data, more informative.
The established standard (from both sides) seems to be to be able to blame ‘the other mob’ for far longer than a single term of government…
Don’t you think we should turn down the thermostat on this hothouse? Even William seems to be getting into the act of attributing causes when there is no effect.
Adam is “not a statiticianâ€, but I am, so I’ve done a few calculations. I did a linear least squares regression analysis on the fortnightly reported ALP TPP for Morgan, Newspoll, ACN and Galaxy for the last 6 months. This is an analysis of the SAMPLES (the people who were polled). We then attempt to extrapolate to the POPULATION (“all of usâ€, in JWH’s words). The underlying assumption (the “null hypothesisâ€) is that nothing is happening in the population…. the analysis sets out to see whether the null hypothesis is true or not. Without going into great detail of the whys and wherefores of the variance analysis which is involved, we may summarise the conclusions as follows:
• There IS a trend. The reported ALP TPP in the samples is going down by about 0.03 % points per day. This is a significant decline but in the population it might be anywhere from about 0.02% to 0 .04% points per day (95% confidence limits)
• The trend is linear (F ratio is 0.39), i.e. there is no evidence that the “line†is bent, e.g. that it has levelled out in recent weeks.
• If you could extrapolate the line to mid November (the approximate election date) the 95% confidence limits for the Australia-wide ALP TPP are 48.5% to 56.7%.
The caveats include:
You shouldn’t extrapolated beyond the sample period
The ALP TPPs calculated from the samples may not be normally distributed
The reported ALP TPP may not really represent the way that people would vote
There may be a “house effect†between the pollsters (there is, of course)
I might have made mistakes.
Everything would be different if we used a longer sample period (but not much if we used a shorter sample period).
The calculations were done with a Fortran program I wrote about 40 years ago.It goes beyond what Excel can do, not least because it allows variance analysis in and from any hypothetical regression. Where the program and Excel do the same things however, they agree.
Everybody who posts here presumably has a computer and it probably has Excel. It is rather illuminating to enter all the data (we have 48 polls to work with since March) and then have Excel make a scatter-gram of them (resisting at all costs showing Excel’s attempt to connect the dots). This gives a good idea of the inherent riskiness in analysing all these numbers and especially in divining whather there is a trend among them.
I didn’t use a Henderson moving average. Has anyone got an Excel macro for this?
Off topic but Haneef wins appeal.
David:
To be fair, Possum has just written a post in which prediction and description are explicitly distinguished, so I don’t think it is that fair to describe Possum’s work as prediction.
And if there weren’t so many people, including some in the MSM, arguing that black is white then A&P and others wouldn’t need to spend so much time demonstrating the banal point that in fact black is black and white is white.
I think all sensible posters here agree that there is plenty of time for things to change and whether things do change in the manner that you describe will be key to deciding the election, and that the closer we get to the election the more confident our predictions will be. Some might even consider that observation to be rather, well, banal. 🙂
Geoff, haven’t you gotten one of your sets of numbers (TPP) between ALP-LP backwards?
e.g.
ALP TPP in the samples is going down by about 0.03 %
e.g.
ALP TPP are 48.5% to 56.7%.
40 days * 0.03% would = 1.2%
I’m not sure where you are starting from, of course, but those numbers (in my feeble mind) don’t gel.
What ‘trend’ do you get using the past 3 months?
Geoff Lambert said
Now you’re showing your vintage Geoff. I worked with Fortran from the eighties into the nineties on a HP1000, HP3000 and DEC 750 running VAX/VMS. It was all spatial data processing, goto’s and conditional goto’s and every other nightmare you can imagine. Glad to be out of it 🙂
pfffftt… try LISP.
I’m not sure how useful it is to extrapolate trends. I doubt anyone expect the 60% 2pp for the ALP would persist for very long. It has now drifted back down to a very healthy 55%.
In the ordinary course of events you’d say that’s it, game over – the coalition is toast – but there are two significant differneces.
1. Howard is not a rat – there are some things a rat won’t do – JWH has no such limits.
2. Treasury has a fair bit of cash and Howard has no shame in looking on the public money as something to spend to defend his place in history.
Hence I think this election has the potentual to go either way. The punters have Rudd a 60% chance – and I think thats about right. Its quite a unique election to have a party so far in front but the outcome potentually so close.
[• There IS a trend. The reported ALP TPP in the samples is going down by about 0.03 % points per day.]
If this is true Rudd will win the election by a few seats, assuming the election is in late November.
To me any prediction of this sort is hard to make, because polls measure the present, they don’t predict the future. What the polls have shown is astonishing consistency since April or so. Before April they were even higher for the ALP.
So many of you think Labor are competent with the economy…well can any of you tell me what Labor’s tax policy is???
Glen what was Howard tax policy in 1996? Never ever?
“• If you could extrapolate the line to mid November (the approximate election date) the 95% confidence limits for the Australia-wide ALP TPP are 48.5% to 56.7%.”
That certainly nails it down, then.
Now I remember why I skipped Intro Stats. I preferred the rigid certainty of English Lit.
#112 – The descriptive trend over the weighted aggregated poll data from March to July 2007 is linear and highly correlated (just look at the graph on my web site). I see no point in casting the trend line prior to that date (as it does not describe the earlier data). I agree that casting the line forward is absolutely speculative. It is not predictive, and I am not making a predictive claim with that trend line. Indeed, i have noted for the trend to continue, the dynamcs that have driven the trend to date must change.
Glen – Labor do have policies, check their website. I think Swan said a while back that Labor would not be changing tax policy much from the Libs. If Labor has anything significant to propose about tax policy. it will be during the campaign. If they release now, Howard will either steal it of trash it.
This “trend” business is only good for looking backwards. What it shows is a decline in support for the ALP from “bloody ridiculous” to “bloody fantastic.”
My prediction is it will start to plateau.
In any case, this reading of the TPP is tosh. Labor has a 46-39 split on primaries–itself a bit of a furphy as it aggregates a “coalition” vote, when no such coalition exists in most electorates.
By and large, the Liberals are basically in the situation where they need to take primary votes from Labor in the big five, or lose. I think Brisbane in particular is going to be a nightmare for Tories.
But what is the justification for highlighting the linear descriptive trend from March to July 2007 rather than the linear descriptive trend from December 2006 to July 2007 or the linear descriptive trend from May to July 2007?
Is there some reason why the former is more significant than the latter two?
And fellas. I’m at #116, not #112 🙂
tax – I meant ‘steal it or trash it’
Glen, i’d still like to know, what was Howards tax policy in 1996? I can tell you what it wasn’t – that changed fairly quickly though.
Glen already knows where Labor stand on tax policy because I have corrected him on more than one occasion in the past. Basically, Swan said a couple of months ago that Labor would be issuing a policy on tax closer to the election, presumably so that the Libs don’t steal it, as they have done on many other policy fronts.
Glen is deliberately dishonest.
The problem with trends – as has already been pointed out is that the starting point will define a large part of the trend.
The main problem is that polling data is autocorrelated.Vote estimation series are autoregressive by their nature.The longer run trends of autoregressive data are not found by simple linear regression.The residuals of such regressions are not normally distributed, heteroskedasticity is rampant.Linear trends over short terms do not capture the nature of underlying data generation process adequately.”best fitting” a straight line through a volatile dataset starting from an arbitrary point is an exercise in GIGO.The arbitrary starting point defines the initial conditions which may not be an accurate representation of the actual data at that point because the variance of the data at that arbitrary starting point has not been accounted for.You need a good reason to choose a starting point – a noisy peak that could be nothing more than a product of autoregressive behaviour (or in the real world, short term newstrading voter momentum boosted by random chance from sampling) isn’t one.
Taking a small arbitrary chunk out of a time series and analysing it in isolation – well that will send you as mad as a bag of cats.
Dear Possum,
Thank you for that concise explanation of aspirational nationalism.
Possum, I agree. Regarding this year’s polling, the most obvious starting point for a trend should be when Rudd took over as leader because, at least qualitatively, this was a point at which Labor effectively renewed itself. It is one of the two most logical starting points.The second most logical starting point is the beginning of the government’s current term in office, immediately after the last election.
No other point makes much sense. This includes Bryan’s arbitrary starting point of March. If you start the trend at the highest possible point, then of course that trend will appear downward.
But if you start at the point when Rudd became leader and Labor in effect became a New Labor, then the trend is flat bar a bump in the middle during March/April. Alternatively, if you start the trend just after the last election, it shows a steady decline in the coalition vote, perhaps leveling a little over this past year.
You can play around with it all you like, but a starting point in March is no less arbitrary than one starting in February or May. The trend should begin from a significant change in political circumstances, such as a new leader or the beginning of a new term in office.
No problem Growler – next up, a monologue on cooperative federalism 😉
Could I say something about seat margins. An item on ABC ‘Insiders’ a couple of weeks ago got me thinking about this issue. There was, of course, a swing to the Libs at the 2004 election and current margins are discussed in this light. But as was mentioned on Insiders, perhaps it is more relevant to look at pre-2004 election margins to establish what they might be now. There was no doubt a Latham ‘scary’ factor which would have driven quite a few voters to the perceived safe hands of Howard in October 2004, giving him the swings. But it can be argued that those votes were very soft and not based on policy but on Latham’s perceived character and personality. This could mean the ‘real’ margins after the 2004 election are a lot less than published. If this were true (and it could be) then the swings to Labor in the coming election in Liberal-held seats could be bigger than the ‘official’ figures indicate. The current polling figures, if translated as a uniform swing to Labor at the election, would produce the mother of all routs if margins are really as soft as some might believe. Does this make sense?
162 Geoff Lambert
No offence Geoff but your post makes absolutely no sense to me or am I missing something.
1/ a loss of 0.03% per day is hardly anything for Labor to worry about when the election is only 3 months away. You are talking about a one percent loss in TPP every month. That’s still landslide terrority for a Labor victory.
2/ Trends cannot continue for ever. What proof is there that these trends won’t bottom out. It’s fantasy land stuff to assume trends will continue. They may but equally they may not.
So you are a pasta sauce chef as well Possum. There is no end to your talents.
I’m a Renaissance Possum!
BTW – there’s a new Morgan Senate poll out today that I dont think has been mentioned yet.
http://www.roymorgan.com/news/polls/2007/4199/
Alex,
It makes sense to me. As discussed in a previous thread, seats like Paterson, Dobell, McMillan, Canning, Solomon, for example, swung heavily to Howard in 2004, and my be ripe for a big swing back.
As also discussed on Insiders, it may be useful to look at the 2001 margins, taking into account redistributions of course, for a proper estimation of margins for some of these seats.
Pauline Hanson Queensland Senate Support Now 5%
Still a long way from 14.3%. Will anyone help her over the line with preferences? Personally I hope not.
I am utterly confident that if movements in the opinion polls were graphed over the period since opinion polling began, we would see with certainty that no matter which direction the polls are moving in now, they will at some point in the future turn round and start moving back the opposite way (and then later still turn round _again_ and head back in the original direction, and then _again_, and …).
*IF* we had a good idea about the _reasons_ for the most recent trend, we could say that continued movement in the same direction is likely _until_ those factors change.
(Likewise, I feel confident in concluding that if the level of support for the ALP stays where it is now, then the ALP will win the election, but otherwise it might not.)
I have noticed in the past that some people like to assume that the things they are analysing conform to a normal distribution, but I have never seen any general reason for this assumption except that it allows people to draw inferences which they couldn’t draw without it. That doesn’t seem like a good reason to me. As far as I can tell, some things are normally distributed and some aren’t. However, I don’t know enough to know what difference it would make to your analysis if the figures you are looking at are not drawn from a normal distribution.
Not to me, although that may be because I have not fully understood you.
Perhaps you mean that the swing at this election will be inflated by the return to Labor of people who voted Liberal at the 2004 election solely because they were repelled by Latham (that is, they would have voted for Labor if it had had a different leader). This may be true. However, if it is true, the inflated swing will only be apparent in terms of a comparison to the actual results in 2004. The swing figures inferred from current opinion polls are all calculated off that base.
Look at it this way. The actual TPP for Labor in 2004 was 47 per cent. However, you might want to suggest that there was a ‘Latham turn-off factor’ of (say) 2 per cent, and that Labor would have got a TPP of 49 per cent without that. If a poll now shows a Labor TPP of (say) 55 per cent, we can infer a swing of 8 per cent from the 2004 election result. If you compare it to a hypothetical ‘Latham-adjusted base’ of 49 per cent, then you can get a ‘Latham-adjusted swing’ of 6 per cent. In that case, you are saying that the 8 per cent swing shown on the poll figures consists of a 6 per cent ‘real swing’ and a 2 per cent ‘Latham correction’. However, the poll result is still 55 per cent, no matter how you choose to describe the swing. If you add an 8 per cent swing to a ‘Latham-adjusted base’ of 49 per cent to get a total of 57 per cent, you are erroneously double-counting the ‘Latham correction’, once in the base _and_ once in the swing. (What you have to remember here, I think, is that figures for ‘swing’ are _always_ _inferred_ from figures for results, and don’t–and can’t–have an independent existence.)
Maybe that’s not what you had in mind, though. In which case, I don’t know what you did have in mind.
When Howard closed the gap on Rudd as PPM, Dennis was ecstatic, and said that this foreshadowed an improvement in Coal primary. What I’ll say is that, if Howard had been able to sustain a close PPM, or moved ahead, then the Coal vote would have gone up, as there is strong correlation between PPM and govt vote.
But Howard was unable to sustain that close result, and is now behind by 7 pts on PPM. If Rudd can sustain that lead, then Labor’s vote gap should widen eventually, though it’ll be much slower than PPM changes.
Homscedasicity?? Yeah, right, a big problem for us all in time series analysis and it’s hard to calculate because you have to muck about with residuals. I should have added that as yet another caveat.
However, in this data-set we have replicates because in most fortnights there is more than one poll published- up to 5 sometimes. So, you can do a variance analysis for the Within and Between replicates. For this data, there is no evidence of heteroscedasicity as time progresses.
As for normal distributions, what is required is that the numbers going into the regression are normally distributed. The numbers (responses) in each case (the TPP) can probably be considered to be the means of a binomial distribution. If the numbers are thus distributed, it can be shown that the means of the numbers are normally distributed (Central Limit Theorem). There is not enough data in 13 fortnights of polls to show that conclusively from the data-set itself, but there probably is in 15 years of fortnightly data- when I have time.
As for the comment: “I am utterly confident that if movements in the opinion polls were graphed … they will at some point in the future turn round.”
Well, true again. I could post all 1,200 poll results that are in my database since 1992 and this would then be manifestly obvious. I think I said here a few months ago that “tip points” seem to occur on the average at (as I recall) about 12-sample intervals (approx 6-months). It’s like WW I trench warfare or the 3-way war in “1984”, the front line washes back and forth continuously over “no-man’s land” but, in the long run nothing ever changes.
Hanson’s senate support at 5%, plus or minus a hefty margin of error.
She had all the splash she’ll get, with news of her new ‘party’ and new ‘policy’ (ie ‘No Muslims’) prior to this poll. Not to mention shysters like me traducing her reasons for running: http://www.news.com.au/story/0,23599,22274073-5007146,00.html (which elicited a lot of supportive comments).
JD @191
Generally, things conform to a normal distribution, irregardless of the underlying distribution. This is known as the “Central Limit Theorem”.
http://en.wikipedia.org/wiki/Central_limit_theorem
Exactly how this applies to opinion polling, I do not know.
From my quick research into the hetereoskedasticity described by Possum above, I believe he is saying that each poll in each new period will have it’s own variance, and probably standard deviation and other measures. This makes trendlines, ummmm… “interesting” one presumes.
Looks interesting, any other Brisbaneites on the forum planning to attend,
http://www.news.com.au/couriermail/story/0,23739,22274617-27537,00.html
Alex McDonnel’s comment on being the election to lose for Labor is spot on. The markets are still as nervous as they were. The half per cent drop in rates was like giving a fractious child a lolly for a sugar hit. It will wear off. It is quite likely we have seen the inevitable turn and that the USA will head into a recession as well as others.
The market is an irrational emotional creature often ignoring fundamentals.
I suppose that’s why they refer to bulls and bears.
At least Kevin Rudd has been warning of the end of the boom and how to manage then.
Father Christmas has his big bag stuffed with notes and promises of a great Christmas. The Reserve Bank will be keeping a watchful eye on that bag.
There was a comment about Janelle Saffin not being told of internal polling but was allowed to know it was looking good.
The seat of Page is not amongst those expected to go to Labor but it is highly likely to do so. That might be the one that Kevin Rudd needs.
It is all very well looking at overall trends in polling which obviously have some validity, but really we are looking at a series of by-elections overlaid by that trend.
In 1998 voters returned to Labor in safe seats but not in marginals and they lost with a majority of the vote.
What we hear of internal polling indicates that voters are returning to Labor this time in marginal and Liberal not-so- marginals.
The result is still likely to be 52.5% ALP to 47.5% Coalition but not evenly distributed by any means.
167 & 168
Double pffft. Way, way back in the dark days of the pre-RISC era, around the time personal computers could almost count to ten unaided, I used to write processor programs in machine code (hexadecimal). More fun than a telephone box full of drunk monkeys. Not.
That doesn’t say ‘generally things conform to a normal distribution’. It says that the sum of a number of variables, if specified conditions are met, tends to be normally distributed. The fact remains that some things are not normally distributed. If you tabulate the ages of the Australian population, they don’t conform to a normal distribution. Likewise, if you tabulate the incomes of the Australian population, they don’t conform to a normal distribution.