While many people pay lip service to the significance of survivorship bias, most of them do not realize how dramatically it can affect returns. Adjusting backtesting results for a system that trades a particular stock index has been shown to cut the annual return by as much as half. This can mean the difference between consistent profits and blowing up your account.
I wrote a post a few months ago that listed and explained a number of different backtesting biases that can skew the returns of a given trading system. Many of these biases come into play as a result of misinterpreting data. One of the reasons I find survivorship bias interesting is that it represents an inherent flaw with the data itself.
The Understated Turnover of Survivorship Bias
Wikipedia defines survivorship bias as:
“the tendency for failed companies to be excluded from performance studies because they no longer exist. It often causes the results of studies to skew higher because only companies which were successful enough to survive until the end of the period are included.”
While this definition is correct, system traders have a tendency to under-appreciate the amount of turnover that can occur in an index over a given period of time.
Frank Hassler from Engineering Returns illustrated this point by listing facts about the S&P 500 turnover from 1990 through 2010. During that time, the index of 500 stock contained a total of 1006 stocks. Of the stocks that comprised the index over those 20 years, 402 have been delisted. There are actually only 189 stocks that remained in the S&P 500 for the entire 20 year period from 1990 to 2010.
Despite being aware of survivorship bias and understanding how it could affect backtesting results, these numbers were still shocking to me. We just don’t realize how much turnover happens in the major indexes from year to year, and that can really have a tremendous impact on the returns of equities based systems.
Comparing Survivorship Bias Across Different System Types
Hassler took his survivorship bias analysis a step further by building a bias-free data set and comparing it to a biased data set using three different trading systems. He compared the results of a biased data set and his unbiased data set backtesting a mean reversion strategy, a long only trend following strategy, and a short only trend following strategy.
Hassler tested each of the strategies when applied to the Nasdaq 100, S&P 100, and S&P 500. The results were very consistent across all three indexes, and very significant to anyone trading system on those indexes.
Mean Reversion Testing
The mean reversion system that Hassler used for his test was very similar to the RSI 27/75 Mean Reversion System that we looked at a few months ago. Testing the system on the S&P 500 resulted in a compound annual growth rate (CAGR) of 22.48% for the biased data set and a CAGR of 14.51% for the unbiased data set. The biased data set experienced a maximum drawdown of 44.05%, while the maximum drawdown for the unbiased data set was 47.11%. The win rate of the system dropped from 68.05% to 66.5% when exposed to unbiased data. Testing both data sets on the Nasdaq 100 and S&P 100 produced similar results.
Long Only Trend Following Testing
The trend following system that Hassler used ranked all of the stocks in an index by RSI(14) and held long positions in the top 25 stocks. The results applying this strategy to the different data sets of the S&P 500 resulted in a CAGR of -4.11% for the biased data and -11.25% for the unbiased data. The maximum drawdown was 78.47% for the biased data set and 93.14% for the unbiased data set.
It is important to keep in mind that the actual returns of the system are irrelevant here. All we are concerned with is their relation to each other. Clearly, this is not a profitable system, but it does support Hasslers argument that survivorship bias can have a tremendous impact on all stock trading systems.
Short Only Trend Following Testing
This system was the opposite of the long only trend following system. It held short positions in the worst 25 stocks of an index in terms of RSI(14). Testing this approach on the S&P 500 resulted in a CAGR of -33.86% for biased data set and -21.95% for the unbiased data set. The maximum drawdown registered was -99.92% for the biased data and -98.79% for the unbiased data.
This was the most interesting part of Hassler’s research. When it comes to short only strategies, survivorship bias appears to work in an inverse matter. Including the stocks that performed so poorly that they were either removed from the index or delisted altogether gave the system opportunities to short them on their way down. The system does not benefit from these opportunities when we use the biased data set.
Survivorship bias makes long only strategies look better than they actually perform and short only strategies look worse than they actually perform. This combination goes a long way towards explaining the extra risk many traders associate with trading the short side. All of their biased backtesting data leads them to believe that short side strategies are much worse than long side strategies when their performance would actually be much closer.
Preventing Survivorship Bias
The best way for us to avoid exposing ourselves to survivorship bias is to do our own backtesting on unbiased data sets. Many traders who are new to developing systems do not see the value in paying substantially more for bias-free data. However, if my systems are only going to perform half as well as I expect them to, I want to know that before I put real money on the line. Wouldn’t you?