Currently working on an inventory forecasting implementation that takes sales history then forecasts it into the future based on a best fit from a range of different statistical algorithms.
Having read Nassim Talebs "The Black Swan" and "Fooled by Randomness" http://www.fooledbyrandomness.com/ i was initially skeptical that that approach would work or be even vaguely accurate. Surely in these uncertain times trend is dead and forecasting is a waste of time? Isn't 'gut feel' more important than some numbers in a spreadsheet?
To my surprise when back tested it forecasted within 10% of actual sales.
This lead me to a significant conclusion and raised two questions.
Having read Nassim Talebs "The Black Swan" and "Fooled by Randomness" http://www.fooledbyrandomness.com/ i was initially skeptical that that approach would work or be even vaguely accurate. Surely in these uncertain times trend is dead and forecasting is a waste of time? Isn't 'gut feel' more important than some numbers in a spreadsheet?
To my surprise when back tested it forecasted within 10% of actual sales.
This lead me to a significant conclusion and raised two questions.
1. Conclusion - My maths isn't up to scratch, i have subsequently re-enrolled to university mathematics course to fix this.
2. Question - How can I benchmark forecasted results based on past performance to check if the results may be worth using?
3. Question - Can i relate this concept to something more interesting than ERP?
At that point I decided to apply the concept to AFL football tipping.
The NAB bank has a footballing tipping comp http://tipping.afl.com.au/
I signed up.
To calculate past performance i would use the teams position on the ladder at the end of the home and away season. i.e. for each round I would pick the winner of each game based on who finished higher on the ladder in 2008.
Based on this simple method as at Round 7 i am currently ranked 5,810 out of 168,180 tippers in the 2009 AFL home and away season!
Now for the benchmarking.
1. How does this stack against the other punters in the comp?
2. How does this stack against pure chance (i.e. flipping a coin to pick the winner)?
8 Games per Round (max 8 correct picks)
Average Wins per round predicated by simple method outlined above 5.4 wins per round
Average Wins picked by all contestants for entire competition 4.4 wins per round
The simple method significantly outperforms the average footy tipper with all their experience, judgment and freely available information available at their fingertips.
Faith in forecasting restored (and lack of faith in the 'wisdom of the crowds' http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds reinforced).
2. Question - How can I benchmark forecasted results based on past performance to check if the results may be worth using?
3. Question - Can i relate this concept to something more interesting than ERP?
At that point I decided to apply the concept to AFL football tipping.
The NAB bank has a footballing tipping comp http://tipping.afl.com.au/
I signed up.
To calculate past performance i would use the teams position on the ladder at the end of the home and away season. i.e. for each round I would pick the winner of each game based on who finished higher on the ladder in 2008.
Based on this simple method as at Round 7 i am currently ranked 5,810 out of 168,180 tippers in the 2009 AFL home and away season!
Now for the benchmarking.
1. How does this stack against the other punters in the comp?
2. How does this stack against pure chance (i.e. flipping a coin to pick the winner)?
8 Games per Round (max 8 correct picks)
Average Wins per round predicated by simple method outlined above 5.4 wins per round
Average Wins picked by all contestants for entire competition 4.4 wins per round
The simple method significantly outperforms the average footy tipper with all their experience, judgment and freely available information available at their fingertips.
Faith in forecasting restored (and lack of faith in the 'wisdom of the crowds' http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds reinforced).
Fantastic post, I think although the black swans theory is based primarily on environments that aren’t so encapsulated and controlled as tipping, so while the theory of the book might be used by some to disprove some forecasting methods, its probably a more holistic approach to forecasting rather than a this specific.
ReplyDeleteThe results are fantastic in that not only does it prove that the algorithm beats simple 50/50 chance, which if nothing were to be taken into account, would be the basic chances of a win for any given team, but also proves that when intelligent decision making is taken into consideration (the average tipper – if you want to be so kind as to call them intelligent), it also performs. The only criticism I might put forward of this statement however, is that tipping ability is typically not a linear curve, but rather a skewed bell curve (there are only a hand full of truly effective football forecasters) however there are also arguments to rebut that as well…(the great thing about opinion is that everyone has one :P)
I think simply by the averages, this proves that forecasting in its most simplistic form…for SOME applications is really advantageous and its application in areas such as inventory forecasting (being a somewhat enclosed environment such as football) has some real merit.
CRI
Dunc, the problem with average in relation to tipping is that most (my guess) tippers tip their team week in, week out. The lower teams have a lot of fanatical followers who, to be kind, might have found your post on google but then given up after they got to the first three syllable word in your heading.
ReplyDeleteAnyway, to more important matters. Would you imagine that we could benefit from using stat-based forecasting models for consulting utilisation or revenue in a large prof services business? You might give some thought to what factors may inform the forecast and we could run some historical modelling to validate the various algorithms and combinations of.
Or, if you want to really have some fun, use the final bookies odds at the track as a means of forecasting race results. Then analyse the various state tote dividends to assess whether there it is worth generating arbitrage bets just before the jump. Any fool can do that and so the dividend should already reflect that fact but the power might be in applying forecasting principles to other influencing factors such as the size of the prize pool, the field size, the distribution of odds across the spread (even or very uneven - 1 odds on favourite).
Go Cats!
Dear CRI (Colourful Racing Identity).
ReplyDeleteGood Point.
It was wrong of me to try and link talebs work on the disproportional effect of the unexpected in complex system with some thing with a fixed domain and easy to calculate outcomes like footy tipping. Are you so sure that footbal tipping exhibits a skewed bell curve? Even the so called experts who according to their advertising 'live and breath footy' http://www.realfooty.com.au/experts/ struggle to beat a coin seem to struggle to beat a coin....
Dear Anon Cats Fan.
ReplyDeleteThe afl tipping has really got to me, Even the so called experts who according to their advertising 'live and breath footy' http://www.realfooty.com.au/experts/ struggle to beat a coin let alone beat my simple algorithm.
With your arbitrage scenario in a gambling situation like horses or sport, the problem is, to make money, you need a lot of cash and a lot of time.
The best example is blackjack, by playing a break even game, counting cards (which unlike popular myth is actually quite easy and does not require idiot savant abilities) and correct bet position sizing you have around a 1% edge on the dealer. However you need a bankroll of at least 10 times the maximum bet allowed and play a lot of hands to safely account for variations.
With regards to forecasting in a professional services environment, x is time (month, week), y is consultants (or consulting teams) and the data is margin, revenue, hours, utilization whatever you want, exactly like 'sales' data for inventory forecasting.
I'll give it a go as I'd imagine it would strongly trend.