It seems like the more we learn about developing and testing quantitative trading strategies, the more we realize how easily our backtesting results can be ruined. One of the easiest ways that we can produce completely useless testing results is by using data that has not been properly cleaned.
Survivorship bias can creep into the price data we use for our backtesting in many subtle ways. Three of the most obvious ways are through differences in as-traded prices, failing to include delisted stocks, or through testing indexes as they are composed today instead of using their historical components.
Each of these data set flaws can have a varied impact on our backtesting results, depending on the type of strategy we are testing and a number of other variables. In a recent post, Cesar Alvarez took the time to test a few strategies using data that contained different versions of survivorship bias. His results showed that there are times when the bias can have a negligible impact, but there are also times when the impact is significant.
At different times throughout the year, each of the major indexes will make adjustments to the stocks that comprise the index. This is done in order to maintain the index’s ability to track the general market in a specific fashion.
By using only the stocks that comprise today’s version of an index as a backtesting universe, we are automatically eliminating all of the stocks that have performed poorly enough to be removed from the index over the length of our backtest. This leaves us with a universe of stocks that is much stronger than the version we would have actually had if we were trading the system live over that period.
Cesar’s research shows us a strategy that was able to post an annual return of 36.25% with a maximum drawdown of 24.54%. However, when the exact same strategy was tested on a historically correct version of the index, the annual return dropped to 14.07% and the maximum drawdown rose to 30.42%.
Delisted stocks are the most commonly understood form of survivorship bias. These are the stocks that our backtesting will miss because they are no longer trading due to acquisitions or bankruptcy. Much like the stocks that are no longer listed in indexes, failing to account for delisted stocks in our universe gives us a stronger universe of stocks than we would have had during live trading.
The evidence that Cesar provides in this case is interesting because it is inconsistent. When testing a mean reversion strategy, including delisted stocks appears to actually improve performance. However, when testing a trend following strategy, including delisted stocks had a negative impact on annual return and maximum drawdown.
Another survivorship bias idea that Cesar wanted to test was the influence of split-adjusted pricing on backtesting performance. Because of stock splits, there are many instances where historic split-adjusted prices of individual stocks are much lower than they actually were at the time.
Cesar’s theory was that this could have a negative influence on backtesting a strategy that required a minimum price for stocks it would consider. While this sounded quite reasonable, the results show that the difference was almost negligible in most cases.