Backtesting Biases and Variations

Last week, I wrote a post discussing how altering the timeframe of a system can change its results. That got me thinking about other ways that backtesting results could be skewed in one way or another based on user defined data such as the date range and market used. These simple differences can have a tremendous influence on the overall returns of any system, so it is important to pay them their proper respect.

When running backtests, it can be very easy to gloss over the down periods and cherry-pick the big return years. The problem is that you won’t have that opportunity when actually trading a system live. You will need to prepare yourself for the possibility that you select the wrong time or the wrong market to trade a given system. Otherwise, you run the risk of letting these backtesting biases adjust your expected return to values the system cannot possibly deliver.

Adjusting the Date Range

Let’s use our 10/100 Moving Average Crossover System from last week as a base. We tested it from January 1, 2001 through December 31, 2010 on the Vanguard Total Stock Market ETF (VTI). All of our tests last week used a starting portfolio value of $10,000, a 10% trailing stop, and a $7 commission.

MA crosses on VTI returned almost 90% over the last decade.

Based on those settings, our 10/100 MA Crossover System returned 89.8% over ten years. This works out to be an annualized return of 12% with a maximum drawdown of 16.2%.

If we would have started trading this system on January 1, 2003, we would have registered a total return of 39% in the three years of trading until the end of 2005. This would have been good for a 16.4% annualized return with a maximum drawdown of only 6%. As you can see, if we based our strategy on these results, we would be expecting the system to continue to produce these extremely high returns.

On the other hand, if we would have started trading this system on January 1, 2006, we would have seen a total return of only 2.5% in the first three years. We also would have had to sit through a 14.2% drawdown.

It is also worth noting that while the ten year track record of this system from 2001 through 2010 is very respectable, we wouldn’t have known that when we started in 2001. If we actually started trading this system in 2001, we would show a total return of -6.2% at the end of 2002. After two full years trading this system, we would not have had a single thing to show for it. The system didn’t find its first big winner until April 15, 2003.

As you can see, the time you chose to begin trading the 10/100 Moving Average Crossover System could have made all the difference over the course of what was a net-profitable decade. It is very important to keep this in mind when you are struggling through drawdowns.

Adjusting the Markets Traded

The market you choose to trade can have the same affect on your trading as the date you start trading. Let’s look at how the exact same system would have performed over the exact same decade if we chose to trade it on different ETFs.

Trading the 10/100 Moving Average Crossover System on the XLF, which represents financials, would have provided a total return of -9.4% for the decade with a maximum drawdown of 30%. It is obvious to us at this point that financials had a rough time during this period, but we would have had no clue about that when we started in 2001.

The XLY, which represents consumer discretionary stocks, also would have underperformed the VTI. Trading the system on the XLY would have returned a total of 39.4, or 6.4% annually, with a maximum drawdown of 21.3%.

XLY shows a 39.4% return over the same decade

If we would have been fortunate enough to trade the XLK, which represents the technology sector, we would have seen a tremendous total return of 95.7%. This works out to be an annual return of 14.2% with a maximum drawdown of 22.3%.

Once again, we see that decisions like what markets to trade and when to start can have a tremendous influence on our results. This is why it is so important to thoroughly backtest any strategy across many different combinations of date ranges and markets.