One of the most frustrating aspects of quantitative trading is that most of the strategies we develop will end up failing. Experiencing system failure can be very difficult for a trader to handle on many levels. There will be a tough emotional and psychological impact to deal with, and there will also be financial losses to address.
Because system failure can be such a devastating event, we need to be prepared to recognize it as early as possible and have a plan to deal with it. System failure could be defined by drawdowns that are too large, drawdowns that are too long, or a general failure to create profits. Whatever definition you prefer, it is important to consider failure in quantitative terms, leaving subjective opinions out of the decision.
Daniel Fernandez from Mechanical Forex wrote a post this week on how to define and quantify system failure. In that post, Daniel discusses having a specific definition for failure that accounts for sample size, relative performance, and performance relative to historic testing results. His point is that traders need to have a quantitative limit at which they will give up on a system.
Avoiding Subjective Assessments
Daniel makes a great point about traders who have an emotional attachment to their strategies ignoring statistical evidence that the system is failing:
When the attachment – due to economical, psychological reasons, etc – is too great, a trader will always have problems with saying that a system failed, because the burden of failure might be greater than the burden of financial loss if the system continues to trade.
When we spend a large amount of time developing our system, we can naturally become attached to them. Just like parents dealing with disciplining their young children, we will have to separate our desire for these systems to succeed from our ability to realistically interpret what is actually happening.
Failure is Relative
Whether you choose to compare your system to a benchmark, historical backtesting, or a monte carlo simulation, you should have a pre-defined limit for how far the system will be allowed to deviate from its expected results. This will help to eliminate any subjective opinions about how well a system is performing.
Sample Size Matters
It is also important to have a pre-defined limit for the sample size that you will consider statistically significant. Comparing a 5 trade sample to a 5000 trade backtest is obviously quite flawed, but you have to set a number of trades that you will consider to be a good representation of your strategy.
As the number of trades increases or decreases, so does the significance of the depth or length of a drawdown. It is your responsibility to define the point at which the number of trades crosses the threshold of significance.