Algorithmic and Mechanical Forex Strategies | OneStepRemoved

Three Different Ways That Survivorship Bias Can Damage Your Backtesting

by Leave a Comment

It seems like the more we learn about developing and testing quantitative trading strategies, the more we realize how easily our backtesting results can be ruined. One of the easiest ways that we can produce completely useless testing results is by using data that has not been properly cleaned.

Survivorship bias can creep into the price data we use for our backtesting in many subtle ways. Three of the most obvious ways are through differences in as-traded prices, failing to include delisted stocks, or through testing indexes as they are composed today instead of using their historical components.

survivorship bias

Including only the stocks that have been strong enough to survive can open your backtesting results up to many different versions of survivorship bias.

Each of these data set flaws can have a varied impact on our backtesting results, depending on the type of strategy we are testing and a number of other variables. In a recent post, Cesar Alvarez took the time to test a few strategies using data that contained different versions of survivorship bias. His results showed that there are times when the bias can have a negligible impact, but there are also times when the impact is significant.

Index Components

At different times throughout the year, each of the major indexes will make adjustments to the stocks that comprise the index. This is done in order to maintain the index’s ability to track the general market in a specific fashion.

By using only the stocks that comprise today’s version of an index as a backtesting universe, we are automatically eliminating all of the stocks that have performed poorly enough to be removed from the index over the length of our backtest. This leaves us with a universe of stocks that is much stronger than the version we would have actually had if we were trading the system live over that period.

Cesar’s research shows us a strategy that was able to post an annual return of 36.25% with a maximum drawdown of 24.54%. However, when the exact same strategy was tested on a historically correct version of the index, the annual return dropped to 14.07% and the maximum drawdown rose to 30.42%.

Delisted Stocks

Delisted stocks are the most commonly understood form of survivorship bias. These are the stocks that our backtesting will miss because they are no longer trading due to acquisitions or bankruptcy. Much like the stocks that are no longer listed in indexes, failing to account for delisted stocks in our universe gives us a stronger universe of stocks than we would have had during live trading.

The evidence that Cesar provides in this case is interesting because it is inconsistent. When testing a mean reversion strategy, including delisted stocks appears to actually improve performance. However, when testing a trend following strategy, including delisted stocks had a negative impact on annual return and maximum drawdown.

As-Traded Prices

Another survivorship bias idea that Cesar wanted to test was the influence of split-adjusted pricing on backtesting performance. Because of stock splits, there are many instances where historic split-adjusted prices of individual stocks are much lower than they actually were at the time.

Cesar’s theory was that this could have a negative influence on backtesting a strategy that required a minimum price for stocks it would consider. While this sounded quite reasonable, the results show that the difference was almost negligible in most cases.

Four Reasons Why Short Strategies Are More Difficult Than You Think

by Leave a Comment

Short strategies are endlessly appealing to quantitative traders. The ability to take profits out of the market at times when everyone else is panicking and losing money is the type of thing that almost every trader dreams about.

While many traders acknowledge that there is money to be made on the short side of just about any market, we all know that it isn’t an easy task. Short selling is generally considered to be much more difficult than trading long positions, and there is good reason for that.

short selling

Short strategies have tremendous appeal to quantitative traders, but there are more complicated risks than backtesting results are able to document.

Cesar Alvarez has been trading short strategies for years. His logic explains the appeal:

I love shorting stocks because it is very hard psychologically, because of that, I believe that there is a good edge there.

In a post on his blog last week, Cesar admitted that he is considering discontinuing his short strategy that had been profitable from 2006 through 2012. He explained that four different issues came up over the course of 2013 that his strategy really struggled with. These issues don’t appear to be major hurdles in backtesting results, but they are much more complicated in live trading situations.

Can You Take All of the Signals?

The first concern that Cesar discusses is when brokers are unable to locate shares for you to borrow, which is essential in a short sale. Cesar says that, at times, he has experienced difficulties borrowing shares of extremely liquid stocks for short sales.

As you know, failing to take one or more of the trades signaled by a strategy could be the difference between a profitable year and a losing year. There is no way to predict or backtest for this type of issue.

Interest on Borrowed Shares

Another drag on the performance numbers of Cesar’s short strategy has been the interest charged for borrowing shares to sell short. He explains that there is very little consistency in this area. For that reason, it is difficult to adjust your backtesting to account for this variable interest.

Increasing Losses

The third issue that Cesar mentions is that a a position moves further against you, it will become a larger portion of your account. This is exactly the opposite of what happens with losers in long positions.

As the price goes further and further against a short position, more and more capital is required to buy back the stock that was borrowed and sold. Cesar discloses that this took a huge toll on his short strategy in 2013.

Black Swan Gaps

The final complication that Cesar has encountered with his short selling strategy is the large gaps that can occur for any number of reason. These Black Swan movements are completely unpredictable, so there is no way to account for them during backtesting. However, when they do show up, they have the power to wipe out an entire account quickly.

 

 

Are You Letting Data Snooping Affect Your Backtesting?

by Leave a Comment

It is important to eliminate as much bias as possible from your backtesting process. This will ensure that any positive results are more likely to continue moving forward. However, eliminating those biases can be more difficult that you might think.

Data snooping is one of the more wide-ranging biases that you might be exposing your backtesting to. It can range from obvious things like limiting your data sample to a bull market period. It can also show up more discretely in areas like optimizing parameters for different indicators your strategy might be using.

data snooping

Being overly selective about the data or parameters you use for your backtest can expose your results to data snooping bias.

CXO Advisory group took a look at snooping bias in a recent post that is actually a preview of a chapter in their upcoming book. The name of the chapter, Avoiding or Mitigating Snooping Bias, suggests that there is likely to be some form of snooping bias in any backtest. Our goal is to reduce its affect on the performance numbers as much as possible.

The chapter starts with a list of the different forms that snooping bias can take on:

Snooping bias, also called mining bias and more loosely benefit of hindsight, is a notorious artificial booster of backtest performance. It takes multiple forms:

  • Picking the best of many rules/indicators (strategies, models) for a given data sample
  • Optimizing rule parameters for a given data sample
  • Restricting a data sample to find favorable performance of a given rule
  • Running an investment contest among many individuals

The chapter then continues with a legal analogy:

A sentiment shared among researchers in stochastic fields is: “If you torture the data long enough, it will confess to anything.”

Because returns are noisy (substantially random), trying many combinations of rules, parameter settings and data samples will generate strategies that outperform benchmarks by extreme good luck.

A prosecutor (an investor) satisfied with false confessions is likely to lose in court (the market).

This analogy does a good job of providing a broad overview of snooping bias. While it is almost impossible to completely remove biases from your backtesting, using some simple common sense in your attempt to optimize a strategy can do a great deal to reduce exposure to snooping bias.

Intentionally Using Data Snooping in Backtesting

by Leave a Comment

One of the easiest biases that can impact our results during backtesting is data snooping, which can also be called look-ahead bias. This occurs when we allow price data that wouldn’t have been available at the time to affect a trading decision.

One of the most complicated aspects of quantitative trading is learning how to manage these types of biases and many quant traders spend a lot of time working to eliminate them.

data snooping

If you went back in time with all of our current price data, how well would your system perform?

In an interesting post titled The Case For Data Snooping, Pete from Shifting Sands takes a different approach. He suggests that if data snooping is used in a certain way, it can actually be quite helpful in testing a system’s potential. He lists three advantages that can be gained from data snooping:

However, it can be a useful tool. If we give our system perfect forward knowledge:

1) We establish an upper bound for performance.

2) We can get a quick read if something is worth pursuing further, and

3) It can help highlight other coding errors.

Pete explains that the first two advantages can help determine if the system is worth further development:
The first two are pretty closely related. If our wonderful model is built using the values it is trying to predict, and still performs no better than random guessing, it’s probably not worth the effort trying to salvage it.
The flip side is when it performs well, that will be as good as it will ever get.

He goes on to explain how data snooping can uncover coding errors:

Firstly, if our subsequent testing on non-snooped data provides comparable performance, we probably have another look ahead bug lurking somewhere.
Secondly, things like having amazing accuracy yet still performing poorly is another sign of a bug lurking somewhere.

He goes on to demonstrate how he uses this approach to test his systems for coding errors.

Pete’s concept of intentionally introducing a bias into our backtesting is very interesting. This kind of outside-the-box thinking makes you wonder what other ways we could adjust our thinking and use known biases to our advantage.

Backtesting Biases and Variations

by 4 Comments

Last week, I wrote a post discussing how altering the timeframe of a system can change its results. That got me thinking about other ways that backtesting results could be skewed in one way or another based on user defined data such as the date range and market used. These simple differences can have a tremendous influence on the overall returns of any system, so it is important to pay them their proper respect.

When running backtests, it can be very easy to gloss over the down periods and cherry-pick the big return years. The problem is that you won’t have that opportunity when actually trading a system live. You will need to prepare yourself for the possibility that you select the wrong time or the wrong market to trade a given system. Otherwise, you run the risk of letting these backtesting biases adjust your expected return to values the system cannot possibly deliver.

Adjusting the Date Range

Let’s use our 10/100 Moving Average Crossover System from last week as a base. We tested it from January 1, 2001 through December 31, 2010 on the Vanguard Total Stock Market ETF (VTI). All of our tests last week used a starting portfolio value of $10,000, a 10% trailing stop, and a $7 commission.

Backtesting bias in VTI

MA crosses on VTI returned almost 90% over the last decade.

Based on those settings, our 10/100 MA Crossover System returned 89.8% over ten years. This works out to be an annualized return of 12% with a maximum drawdown of 16.2%.

If we would have started trading this system on January 1, 2003, we would have registered a total return of 39% in the three years of trading until the end of 2005. This would have been good for a 16.4% annualized return with a maximum drawdown of only 6%. As you can see, if we based our strategy on these results, we would be expecting the system to continue to produce these extremely high returns.

On the other hand, if we would have started trading this system on January 1, 2006, we would have seen a total return of only 2.5% in the first three years. We also would have had to sit through a 14.2% drawdown.

It is also worth noting that while the ten year track record of this system from 2001 through 2010 is very respectable, we wouldn’t have known that when we started in 2001. If we actually started trading this system in 2001, we would show a total return of -6.2% at the end of 2002. After two full years trading this system, we would not have had a single thing to show for it. The system didn’t find its first big winner until April 15, 2003.

As you can see, the time you chose to begin trading the 10/100 Moving Average Crossover System could have made all the difference over the course of what was a net-profitable decade. It is very important to keep this in mind when you are struggling through drawdowns.

Adjusting the Markets Traded

The market you choose to trade can have the same affect on your trading as the date you start trading. Let’s look at how the exact same system would have performed over the exact same decade if we chose to trade it on different ETFs.

Trading the 10/100 Moving Average Crossover System on the XLF, which represents financials, would have provided a total return of -9.4% for the decade with a maximum drawdown of 30%. It is obvious to us at this point that financials had a rough time during this period, but we would have had no clue about that when we started in 2001.

The XLY, which represents consumer discretionary stocks, also would have underperformed the VTI. Trading the system on the XLY would have returned a total of 39.4, or 6.4% annually, with a maximum drawdown of 21.3%.

Backtesting bias for xly

XLY shows a 39.4% return over the same decade

If we would have been fortunate enough to trade the XLK, which represents the technology sector, we would have seen a tremendous total return of 95.7%. This works out to be an annual return of 14.2% with a maximum drawdown of 22.3%.

Once again, we see that decisions like what markets to trade and when to start can have a tremendous influence on our results. This is why it is so important to thoroughly backtest any strategy across many different combinations of date ranges and markets.

How Backtesting Biases Distort Expectations

by Leave a Comment

“Why don’t my trading results ever seem to match my backtesting results? I followed the system perfectly!”

Anyone who has ever traded a thoroughly backtested systematic strategy has likely asked themselves this question. The answer is that despite all the effort that you put into coding and backtesting your system, there were likely a number of biases that crept in to skew your results.

Backtesting biases come in all shapes and sizes. Some are statistical, while others are psychological. Most stem from incredibly slight discrepancies. They can be very hard to notice, but can distort your expectations. Unchecked biases can lead you to believe that a losing system is relatively safe.

In order to keep these biases out of your backtesting results, it is important to know the most common forms and how they can creep into your testing. Here are the four most common biases that I catch myself struggling with.

Backtesting bias

Backtesting bias blinds traders to the reality of the markets.

Data Snooping Bias

Data Snooping Bias is also referred to as Optimization Bias or Curve Fitting. This bias is the result of refining too many parameters to improve a system’s performance on a single data set. Like most biases, Data Snooping is fairly easy to understand at face value, however it has a habit of subtly creeping its way into system development.

A common example of data snooping starts out as an honest effort to improve a system. You then test if adding another indicator into the mix will improve your results. If it does, then you incorporate that indicator into the system and test adding another indicator. The end result is a system that is perfectly optimized to trade the exact data set you tested it on. The problem is that the system is only optimized for that specific data set, which already happened.

The best way to avoid data snooping, or curve fitting, is to keep your systems simple, using as few parameters as possible. It is also important to backtest your system on many different data sets across different markets and time periods.

Look-Ahead Bias

Look-Ahead Bias happens when backtesting results are distorted because trading decisions are based on information that was not yet available. This generally happens because extra data is accidentally included and we simply don’t realize it. These errors are often very slight, which is why they are so easily overlooked. They can, however, have a dramatic impact on the overall results.

An example of Look-Ahead Bias would be using an entire data set to calculate optimal parameters for a strategy and then testing the strategy on that data set using those parameters. Because the parameters were calculated using the entire data set, they are technically using future data to make trading decisions.

Survivorship Bias

Survivorship Bias addresses the concept that many data sets do not include assets that are no longer listed. The result is that these data sets only include the assets that have survived to this point. This makes them skewed to the positive side because they do not include assets that have not survived.

A classic example of Survivorship Bias would be backtesting a stock trading strategy over a time period that included the dot-com crash. If stocks that went bankrupt during that crash are not included in the data set and your strategy would have been trading them, then your results are likely to be skewed.

The best way to avoid this issue is to purchase data that is free of Survivorship Bias. Another option is to use data that is more recent or markets that are less prone to Survivorship Bias.

Tolerance Bias

I’ve seen it written that every trader thinks that they will be able to withstand twice the drawdown that they will actually be able to tolerate. This is the root of Tolerance Bias, which is a trader overestimating his ability to tolerate drawdowns.  This bias is completely psychological.

I provided a good example of this in my post about the 3 Day High/Low Mean Reversion System. In that post, I pointed out that the system had an overall drawdown of just over 15%. The system’s largest losing position was down over 18%.

While it is easy to look at the overall return of the system and agree that it was profitable, it is naive to think that it would not be a challenge to stick to the system at its low point. Imagine sitting on a 15% loss for the life of your system and holding a position that was down 18%. You would certainly be questioning whether markets had fundamentally changed in a way that made your system ineffective.

The best way to avoid tolerance bias is to dramatically underestimate your drawdown tolerance.