In my free time, I have been working on developing a quantitative strategy for betting on Major League Baseball games. This has been a very interesting side project, but at the moment it has not been very successful.
The primary reason that I chose to focus on baseball games instead of football or basketball games is that baseball teams play many more games than any other sport. Over time, this should provide a larger sample size, giving me more significant results and eliminating the variance caused by small sample sizes.
There was a post on Gestaltu on Monday that looked at the topic of sample size in a similar manner. The author used the natural mean reversion that occurs in the NFL every season as an example to explain the impact that a small sample size can have on performance. The article also addressed the danger that small sample sizes can introduce into our assessments of fund managers and backtesting results.
The Number of Games Played Matters
The regular season in the NFL consists of 16 games played over 17 weeks. In comparison, the NBA and NHL each play 82 games in a season, and MLB teams play 162 games. Multiply those totals by the number of teams in each league and you will see that there are dramatically less NFL games than any other sport each year.
The trading comparison here is obvious. It would be difficult for us to seriously consider a strategy based on 16 trades. The strategy that makes 162 trades each year has a much better chance of avoiding unlucky trades, or at least recovering from them.
Average Teams Making the Playoffs
The article also stresses the point that every year in the NFL there are teams with average talent levels that sneak into the playoffs. Because of the small sample size, these average teams can reap tremendous rewards from a few lucky breaks during the season.
In similar fashion, many fund managers profit from similar lucky breaks in their performance history. The article shows that even track records dating back 10 years can lack samples sizes large enough to be statistically significant.
Just as average football teams can sneak into the playoffs and marginal fund managers can post impressive returns, average trading strategies can produce outstanding returns when backtested in their ideal environments.
Rather than focusing exclusively on backtesting results, we should also be looking at the underlying process of a strategy. We want our strategies to limit the number of input parameters and be able to stand up to thorough statistical analysis. You don’t want to get caught with all of your capital riding on the 2012 Baltimore Ravens during the 2013 season.