It is important to eliminate as much bias as possible from your backtesting process. This will ensure that any positive results are more likely to continue moving forward. However, eliminating those biases can be more difficult that you might think.
Data snooping is one of the more wide-ranging biases that you might be exposing your backtesting to. It can range from obvious things like limiting your data sample to a bull market period. It can also show up more discretely in areas like optimizing parameters for different indicators your strategy might be using.
CXO Advisory group took a look at snooping bias in a recent post that is actually a preview of a chapter in their upcoming book. The name of the chapter, Avoiding or Mitigating Snooping Bias, suggests that there is likely to be some form of snooping bias in any backtest. Our goal is to reduce its affect on the performance numbers as much as possible.
The chapter starts with a list of the different forms that snooping bias can take on:
Snooping bias, also called mining bias and more loosely benefit of hindsight, is a notorious artificial booster of backtest performance. It takes multiple forms:
- Picking the best of many rules/indicators (strategies, models) for a given data sample
- Optimizing rule parameters for a given data sample
- Restricting a data sample to find favorable performance of a given rule
- Running an investment contest among many individuals
The chapter then continues with a legal analogy:
A sentiment shared among researchers in stochastic fields is: “If you torture the data long enough, it will confess to anything.”
Because returns are noisy (substantially random), trying many combinations of rules, parameter settings and data samples will generate strategies that outperform benchmarks by extreme good luck.
A prosecutor (an investor) satisfied with false confessions is likely to lose in court (the market).
This analogy does a good job of providing a broad overview of snooping bias. While it is almost impossible to completely remove biases from your backtesting, using some simple common sense in your attempt to optimize a strategy can do a great deal to reduce exposure to snooping bias.