“Why don’t my trading results ever seem to match my backtesting results? I followed the system perfectly!”

Anyone who has ever traded a thoroughly backtested systematic strategy has likely asked themselves this question. The answer is that despite all the effort that you put into coding and backtesting your system, there were likely a number of biases that crept in to skew your results.

Backtesting biases come in all shapes and sizes. Some are statistical, while others are psychological. Most stem from incredibly slight discrepancies. They can be very hard to notice, but can distort your expectations. Unchecked biases can lead you to believe that a losing system is relatively safe.

In order to keep these biases out of your backtesting results, it is important to know the most common forms and how they can creep into your testing. Here are the four most common biases that I catch myself struggling with.

Backtesting bias blinds traders to the reality of the markets.

Data Snooping Bias

Data Snooping Bias is also referred to as Optimization Bias or Curve Fitting. This bias is the result of refining too many parameters to improve a system’s performance on a single data set. Like most biases, Data Snooping is fairly easy to understand at face value, however it has a habit of subtly creeping its way into system development.

A common example of data snooping starts out as an honest effort to improve a system. You then test if adding another indicator into the mix will improve your results. If it does, then you incorporate that indicator into the system and test adding another indicator. The end result is a system that is perfectly optimized to trade the exact data set you tested it on. The problem is that the system is only optimized for that specific data set, which already happened.

The best way to avoid data snooping, or curve fitting, is to keep your systems simple, using as few parameters as possible. It is also important to backtest your system on many different data sets across different markets and time periods.

Look-Ahead Bias

Look-Ahead Bias happens when backtesting results are distorted because trading decisions are based on information that was not yet available. This generally happens because extra data is accidentally included and we simply don’t realize it. These errors are often very slight, which is why they are so easily overlooked. They can, しかし, have a dramatic impact on the overall results.

An example of Look-Ahead Bias would be using an entire data set to calculate optimal parameters for a strategy and then testing the strategy on that data set using those parameters. Because the parameters were calculated using the entire data set, they are technically using future data to make trading decisions.

サバイバーシップ ・ バイアス

Survivorship Bias addresses the concept that many data sets do not include assets that are no longer listed. The result is that these data sets only include the assets that have survived to this point. This makes them skewed to the positive side because they do not include assets that have not survived.

A classic example of Survivorship Bias would be backtesting a stock trading strategy over a time period that included the dot-com crash. If stocks that went bankrupt during that crash are not included in the data set and your strategy would have been trading them, then your results are likely to be skewed.

The best way to avoid this issue is to purchase data that is free of Survivorship Bias. Another option is to use data that is more recent or markets that are less prone to Survivorship Bias.

Tolerance Bias

I’ve seen it written that every trader thinks that they will be able to withstand twice the drawdown that they will actually be able to tolerate. This is the root of Tolerance Bias, which is a trader overestimating his ability to tolerate drawdowns. This bias is completely psychological.

I provided a good example of this in my post about the 3 日高/低平均回帰システム. その記事で, I pointed out that the system had an overall drawdown of just over 15%. The system’s largest losing position was down over 18%.

While it is easy to look at the overall return of the system and agree that it was profitable, it is naive to think that it would not be a challenge to stick to the system at its low point. Imagine sitting on a 15% loss for the life of your system and holding a position that was down 18%. You would certainly be questioning whether markets had fundamentally changed in a way that made your system ineffective.

The best way to avoid tolerance bias is to dramatically underestimate your drawdown tolerance.