One of the most commonly heard phrases in the quantitative trading world is that “past results do not guarantee future performance.” While this could be chalked up to a bunch of nerds covering their you-know-whats, there is actually more to consider here.
We all know that the fact that a strategy performs well over a given backtesting period by no means indicates that it will perform well in the future. We know that there are plenty of different biases that come into play when backtesting strategies on any market. We also know that markets can fundamentally change at any time, rendering our favorite strategies ineffective.
But let’s be honest, we don’t really think that will happen to OUR strategy. Those are the kinds of stories that happen to some other trader. In a recent post on his blog Mechanical Forex, Daniel Fernandez explored the topic of backtesting and future profitability. He began by explaining why we backtest in the first place:
When we want to trade the financial markets we need to design some sort of strategy that is able to give us an above random chance probability to succeed — a trading edge.
In order to solve this problem the first impulse is to look at trading mechanisms that have succeeded in the past and then attempt to trade this exact same techniques going forward in order to generate similar profits.
Then, he begins to point out some of the false assumptions that backtesting leads us to make:
The idea therefore seems pretty straightforward, generate some system that performed significantly well for as long as possible into the past and this system should work in the same way going forward.
We make several important assumptions such as: the market will behave in the future as it has in the past, the longer we back-test into the past the more market conditions we cover, etc.
Daniel also points out some other issues that our backtesting may encounter:
Does the market really behave as it has in the past?
Does having a strategy with long term profitable back-testing guarantee profitability to any degree (does it even increase our chances beyond random chance?)?
Does having a 13 year back-test give you a higher chance of profit than a 2 year back-test?
As you can see, Daniel addresses a number of different assumptions that we are prone to make without even realizing it. He continues by defining a conservative Forex system that could have easily been constructed in 2001. This system would have looked great through backtesting results, but would have fallen flat on its face and blown up had we started trading it live in 2001. He also profiles a statistically similar system that would have performed well starting in 2001.
Daniel explains how many quantitative traders would have handled this case study:
Many of you would be tempted to perform elaborate statistical analysis to attempt to find differences between them so that you can say: “system A failed because of this and this while system B didn’t fail because of this and this”, in reality you can find no definitive answer (as least I haven’t), there is always a probability for a system to fail bluntly, regardless of its past statistical characteristics.
The important takeaway here is that any system, regardless of how well it has performed in the past, any system can stop performing well at any time. As quantitative traders, we must always be on alert for changes in market conditions that could negatively impact our systems.