curve fitting Archives - Algorithmic and Mechanical Forex Strategies

2 Painful hits

February 13, 2017 by Shaun Overton 14 Comments

December and January were extremely unkind to me. I took a huge loss on December 9 that coincided with the Fed meeting and another big punch in January. In total, I went from a 28% profit to a ~4% net loss.

Deservedly, my inbox quickly flooded with comments and suggestions on the drawdown. The most common of those was to stop trading during news events.

So… why am I still trading during news events? There are a few answers to that question.

Curve fitting

It’s not like the strategy loses money on every single news event. It’s 100% true that news events like the Fed meeting can and badly hurt. Say that I’m determined to exclude news events in the future. I’d have to

Collect historical news event data
Create a second algorithm, which selects the news events that forbid and allow trading to continue
Test how the news algorithm interacts with Dominari
Repeat this many times until I’m happy with the final result

Due to the tiny number of news events that impact the markets like the December 9th announcement, my data set is miniature. The risk of overfitting to historical news events is huge.

Working with tiny amounts of data provides little in the way of long run confidence. Focusing my efforts elsewhere is far more likely to improve performance and requires much less work.

Too many trades

Too many trades sounds a bit naive, so let’s dig into what that means. Dominari trades a portfolio of 7 different instruments. All instruments cross with USD.

EURUSD
GBPUSD
USDCHF
AUDUSD
NZDUSD
USDJPY
USDCAD

Many subscribers correctly observed that the major losses occurred with trades open on all 7 pairs in the portfolio at the same time. A good predictor of trade performance is the number of trades open simultaneously.

1-3 trades seems to be consistently profitable
4-5 trades leads to biting my nails
6-7 trades is neutral to disastrous

Testing and confirming the max open trades rule was quick and easy. 5+ trades is very dangerous.

Accordingly, Dominari now exits all open trades if there are 5 or more trades open at any given time.

The next feature of Dominari will be a reversal strategy. Dominari was clearly prone to sudden equity changes if 5+ trades were open at the same time.

Make the losses work for us

An obvious counter strategy is to open trades in the opposite direction whenever Dominari would otherwise open too many trades. Testing the idea is very easy.

Coding a Dominari reversal strategy, however, would require a major reprogramming of the expert advisor’s code.

The number of trades per year would be miniscule. I doubt that it would average even 1 trade per month.

The idea is that Dominari can be the normal trading strategy. Whenever Dominari opens too many trades, the strategy then switches into reversal mode and trend trades with a simple trailing stop.

Switching direction should mostly reverse the negative trade skewness back in the positive direction. Almost all of the offending trades open at exactly the same time.

If the biggest losing trades opened at different times, there would be the risk of being too late to the party. All blowout trades opening at the same time means that the strategy can realistically reverse 100% of would-be losses into profits.

Sitting at the top of the docket are changes to Pilum. You can expect to hear about those soon so that I can incorporate Pilum into the Dominari signals. Once that and 2 other internal projects are finished, I’ll be able to dedicate the time required to fully implement the Dominari Reversal System.

Equity stop loss

Dominari uses emergency stop losses on all tickets. That is appropriate 99% of the time for individual trades. Those emergency losses reset once per hour in line with the concept of the TODS.

A little of the problem was bad luck. My stops came within a handful of pips of being triggered. Then they reset even further away, which made a bad problem worse.

When all trades move at the same time, then clearly the strategy could suffer extreme losses.

The first attempted solution after the Fed announcement was to add a portfolio level stop loss. The way that I wrote it also updated once per hour. When a second negative movement came in January, I stopped trying to be clever. It’s a flat, simple, stupid stop loss. If I lose more than 4% on all open trades, the entire Dominari portfolio goes flat.

I’m still trading Dominari

I still have my money trading the Dominari system; my confidence in the long term performance hasn’t changed, but it obviously requires safeguards. The max number of trades and the portfolio level stop loss will go a long way to limiting the impact of big moves in the future. AND, I should get the counter-strategy developed relatively soon to turn potential frowns upside down.

Lastly, many of you questioned why I’ve been so quiet. The honest answer is that I needed some time to process what happened. It’s easy to feel overwhelmed and discouraged when you get knocked down. I needed some time to process what happened.

I also needed time to double check the changes that I made to the portfolio were actually beneficial. It’s very easy to appease traders when they’re upset by rushing out features before they’re thoughtfully considered.

My money is on the line (I lost 2,000 euros between the two moves). What hurt my subscribers hurt me, too.

One line of code makes all the difference

February 9, 2017 by Shaun Overton 4 Comments

I was really excited about my Pilum strategy two months ago. The research looked great and everything was ready to rock and roll. Demo testing began and then… not much happened.

The Quantilator is (mostly) finished, which finally gave me time to circle back and review what happened with Pilum.

Live demo trading of Pilum. Dec 9, 2016 to Feb 7, 2017

The expected outcome was that I would win 75% of the time. Trades were infrequent, so I thought maybe I’m just having bad luck. But then my win rate remained stuck around 50%. Simple statistical tests told me this was unlikely to be bad luck.

I used the research time to pour over my research code and to compare it with live trades. What I found was that a single line of code (AHHHHHHHHHHHHHHH!) was incorrectly calculating my entry price, dramatically overstating the profits.

The flawed code produced this equity curve from a single combination of settings:

When the actual, correct result looks like this with those same settings:

The accurate backtest of Pilum

I’ll be honest… I like the flawed backtest a lot more!

The new, single-setting backtest isn’t as good, but it’s still trade-worthy. There are some characteristics that I dislike and features that I love. Let’s dig into those.

What I dislike

The frequency of trades is very low. Out of 19 months there were a total of 43 trades. 43 trades to comprise a backtest on 40+ instruments is a very small number.

If it weren’t for the statistical pattern backing up the frequency, I would not consider the test. However, there are 20,000 bars each on the 44 instruments. There are 880,000 total bars used to analyze whether my Pilum pattern offers any predictive value.

The most valuable predictions, however, are also exceptionally rare. That’s why I’m not able to get the trading frequency higher, which would potentially smooth the returns.

What I love

My previous systems like QB Pro and Dominari traded actively for relatively small wins. Trading costs exercised a massive impact on the overall performance.

The accurate backtest of Pilum

Now look again at the correct equity curve (the image to the right). Do you see the final profit of roughly 0.14? That’s a 14% unleveraged return over a 19 month period.

Allocating 2:1 or 3:1 leverage on this strategy could average annual returns of 15-25%.

Detecting hidden risk

A key measure of risk is skewness. You may not use that term yourself, but it’s something most of you already understand. The biggest complaint about people trading Dominari was that the average winner relative to the average loser was heavily skewed towards the losers.

Dominari wins on most months, but when it lost in December it was devastating. I implemented what I thought was a portfolio stop after the December 9th aftermath. Then I had a smaller, but still very painful, loss in January. The portfolio level stop loss of 3% should prevent future blowouts now that I know what goes wrong.

I still believe in Dominari. But, I obviously lost the work of most of the year due to those events.

Knowing that skewness is a good measure of blowout risk (even if you’ve never seen it in a backtest, like happened with Dominari), Pilum looks extremely encouraging.

This is a histogram of profit and loss by days. You should notice a few things.

The tallest bar is to the right of 0. That means that the most frequent outcome is winning.

The biggest winning day is dramatically better than the worst losing day. The worst outcome was a loss of 2%. The best outcome is gains near 10% in a single day (unleveraged!).

This is the statistical profile of an idea that’s much more likely to grab an avalanche of profits than it is to get blown out.

It gets even better

Would you say that the blue and red equity curves are highly or loosely correlated? Look closely.

Writing this blog post made me think carefully about the Pilum strategy. I decided that maybe I should see if all of the profits are coming from different settings at the same time. There’s very little risk of overfitting the data as my strategy only has 1 degree of freedom.

The blue bars are the equity curve of Setting 1.

The red bars are for Setting 2.

Do you think these are tightly or loosely correlated?

If you said loosely correlated, then you are correct. Notice how each equity curve shows large jumps of profit. Did you notice how those profit jumps occur on different days?

The blue setting skyrockets on a single day in November 2016. It leaves the red equity curve choking in its dust.

But then, look what happens as I advance into December. The red curve dramatically catches up to the blue curve and even overtakes it.

The correlation between the 2 strategies is only 57%.

Combine multiple settings into 1 portfolio

This is a much nicer equity curve!

Loose correlations are a GIFT. Combining two bumpy equity curves into a single strategy makes the performance much, much smoother.

The percentages of days that are profitable also increases. Setting 1 is profitable on 58.0% of days. Setting 2 is profitable on 53.5% of days.

But… combining them makes Pilum profitable on 68.2% of days. Awesome!

That also provides more data, which puts me in a stronger position to analyze the strategy’s skewness. Look at the frequency histograms below. They’re the same type of histograms that I showed you in the first section of this blog post. As you’ll notice, they look a lot different.

Pilum most probable daily profit and loss

The most probable outcome for any given day is a small winner

The tall green bar is the most probable trading outcome for any given day with filled orders. The average day is a positive return of 0-1%.

The small red bar is the worst trading day of the combined strategy.

The small green bars are the best trading days of the combined strategy.

Look how far to the right the green bars go. The largest winner is more than 3x the biggest loss. And, there are so many more large winners compared to losers.

Giant winners are far more likely than comparable losses.

The Plan

I immediately pushed Pilum into live trading this combination of two strategies. I expect that adding a second degree of freedom and running about 30 different versions of the strategy – all with different settings – will add to the performance and smooth the returns even further.

Dominari hasn’t been working on my FXCM account, which is very difficult to accept because the lacking performance seems to be a buried execution issue. Pilum, however, trades very infrequently. It’s unlikely that execution quality will make a dramatic difference in the long term outcomes.

So, I’m going to convert the FXCM account to trading Pilum exclusively. That will be offered as a strategy on Collective2 within the next few weeks, a company with whom I’ve been working closely. Their users are more investor rather than trading oriented – they’re far more likely to view low trading frequency as a good thing. I suspect that most people here have a different opinion and want to see a lot of market action.

I’ll write an update on Dominari shortly.

The Difference Between Optimization and Curve-Fitting

February 3, 2014 by Andrew Selby Leave a Comment

Optimization and curve-fitting are two terms that are very common among quantitative traders. They are so common that many traders confuse the terms, or use them as synonyms when they actually have very different meanings.

Michael Harris recently published a guest post on System Trader Success that broke down the meaning of each of these terms and explained how they interact with each other. He also shared a process for determining how likely a strategy was to be exposed to a curve-fitting bias that is based on how its parameters are utilized.

Knowing the difference between optimization and curve-fitting can help you avoid exposing your strategy to backtesting biases.

Optimization vs. Curve-Fitting

Michael began by defining each of the two terms individually. What this shows us is that they have subtle differences with respect to each other. Here is how he explains it:

As already mentioned, curve fitting may involve optimization but the latter is a process with a much broader scope and includes many more possibilities than curve-fitting.

Michael looks at strategy optimization from the viewpoint of finding the best collection of entry and exit signals for a backtesting period. He explains that curve-fitting focuses more on the results than the signals that caused the result.

Is Curve-Fitting Really The Problem?

Another interesting point that Michael brings up is that there is no mathematical proof that optimized systems are more likely to fail because they are curve-fit. He suggests that it is possible for any optimized strategy to fail at any point, and that the strategy failure has nothing to do with what parameters the system uses.

He explains that a different form of bias is far more likely to cause a failure:

Nevertheless, optimization that causes selection of entry and exit collections is in general a problematic process because it introduces survivorship bias.

Michael argues that in almost every case where an optimized strategy fails, survivorship bias is more likely to blame than a curve-fitting bias.

How To Gauge Optimized Trading Strategies

While Michael does not believe that curve-fitting failures are nearly as prevalent as many traders believe, he does discuss how some strategies are more likely to be exposed to curve-fitting than others. In order to gauge how likely an optimized strategy is to be exposed to curve-fitting, Michael divides them into three different classes.

The first class contains strategies where the optimized parameters define both the entry and exit signals. These strategies are the most vulnerable to curve-fitting.

The second class contains strategies where just the entry signals are defined by optimized parameters. These strategies are less likely to be exposed to curve-fitting than those in the first class.

The third class contains strategies where the optimized parameters define only the exit signals. These strategies are the least likely to be exposed to curve-fitting.

Walk Forward Optimization

January 13, 2014 by Shaun Overton 6 Comments

If you were walking and randomly it started to rain, would you consider carrying an umbrella tomorrow? Of course you would.

The reason I ask a rhetorical question like that is when people observe a behavior, they respond accordingly. If they expect that something might happen again, they change their behavior to accommodate the change in outcomes.

When you think about forex robots, everybody has the dream of developing a strategy that works forever. It requires no changes. The initial settings always work. Turn it on and move to the beach.

Reality, of course, is more complicated than that.

Walk forward optimization continually optimizes throughout time instead of looking for one set of static settings

That leads to expectations of what you need to do when your strategy inevitably goes awry. It’s very possible that you come up with a strategy that works and does amazingly well on the current market. However, a past genius doesn’t mean future genius. There’s always the chance that your strategy will no longer work in the future.

Why is that? It’s the same reason that you might carry an umbrella tomorrow if it rains today. People observe the market performing in a consistent manner. As more and more people make the observation, people start trading on it. The market responds to those changes, and eventually the opportunity completely washes out as too many people have eared about it.

Walk forward testing is the process of determining whether or not your strategy has washed out. By testing on one set of data, and then testing it on a blind set, you can give yourself an indication of whether your strategy is bad or not. The goal of walk forward isn’t to prove that your strategy is good. It’s to prove that your strategy is not known to be bad.

The process of walk forward testing is very simple. You identify a set of information that you want to use for your testing and optimization. Using a real example, right now it’s the beginning of 2014. So maybe you want to look and test data from 2011 through 2012. That would be your in sample data, and then your out of sample data might be all of 2013.

In order to conduct a walk forward test, you would test and analyze your strategy 2011-2012. Then, to determine if it’s “not known to be bad”, you then walk forward to 2103 to see review the performance.

What you’ve done is a blind test. You didn’t know what how the strategy would perform in 2013 when you tested it in 2011-2012. By putting it on a blind sample, you give it the opportunity to fail.

The reason so many traders put their faith in walk forward testing is because it’s the absolute best tool to identify weaknesses in your optimization. When you’re testing a strategy, it is very likely that you’ve overfit to past opportunities.

Self feedback loops in the current market

Let me give you an example. In the current markets, a lot of traders have been banging gold on the market open where every day at market open., they sell as much gold as they possibly can. Sometimes it’s several multiples of the annual production in a span of a few minutes. What you see is an absolute freefall for five or ten minutes. That state persists for days at a time. But that doesn’t last forever. When enough traders start seeing that people bang gold on the open, they start doing the same thing.

Effectively, whoever wants gold to falloff on the market open has taught other traders to do that trade for them. As people expect gold to fall in the first five minutes of the open, they then change their behavior. Some try to jump on banging the open and go short.

Others start modifying their behavior. They notice that gold free falls for five minutes. Then, all of a sudden it stops, and more than like it reverts to the mean. They’ll start changing their tack and buying after so many minutes have elapsed from the open. They expect that the heavy volume that preceded the selling will eventually return to normal. As people change their behavior, other people respond in kind.

If enough people start selling on the open and then buying on the open five minutes later, you can see that a pattern is forming where one person responds to the actions of another. It’s a self feedback loop where the state that was working for the first couple of days no longer works in the future.

If you can identify a strategy that is able to survive those conditions, and is able to survive conditions where you didn’t do any testing and optimization, you give yourself better odds of succeeding in the future. It means that not very many traders have clued into this trading opportunity that you’ve discovered.

The approach to to walk forward testing is the antidote to the problem known as curve fitting. Curve fitting is the ultimate woulda coulda shoulda strategy. It’s akin to opening a chart from yesterday and saying I would’ve bought here and I would’ve sold here, already knowing what transpired.

Of course you’re going to “make money” in that situation. You know with perfect information what the market did. In the future, you don’t know the perfect information. The goal of a strategy is to deal with that ambiguity.

Curve fitting means that you’ve fit everything so perfectly to past market conditions that when new situations inevitably arise, sort of akin to the phrase, “history doesn’t repeat itself, but it rhymes,” your strategy does the same thing.

You want a strategy that does well on past performance, but you’re not coming up with a strategy to make money on historical markets. The purpose of developing a strategy is to make money in future markets. When you’re backtesting, you’re trying to strike the balance between solid historical performance and, most importantly, making sure that that historical knowledge extrapolates to future performance. Your goal is to make money.

Rolling Walk Forward Optimization

Rolling walk forward optimization takes the walk forward idea and continuously improves the strategy by exposing it to new data. So let’s say that you have a twenty four month sample period. One way to go about it would be to optimize your strategy for a period of two months, then to walk it forward to the third month. You observe the behavior and you reoptimize for the second and third month, then walk it forward to the fourth month.

By doing so continuously, you eliminate the decay time of the strategy and give it a chance to adapt to ongoing market conditions. It is sort of the redheaded stepchild to machine learning. Experience and losses give the strategy the opportunity to improve and adjust to the market changes through walk forward optimization.

…you eliminate the decay time of the strategy and give it a chance to adapt to ongoing market conditions

Another important consideration for walk forward analysis is the degrees of freedom within a system. For example, let’s say that you are analyzing a moving averaage cross. You’re using two moving averages and use a fixed stoploss and take profit. That would give you four degrees freedom. The fast moving average is the first degree. The slow moving average is the second degree. The third is the stoploss and the fourth is the take profit.

The more degrees of freedom that you allow in a system vastly increases the chances 0f curve fitting your systems to historical data. The absolute best systems maintain twelve degrees of freedom or less. You want to find trading opportunities that have large numbers of trades and that offer performance that you find satisfactory.

Another element to consider in your optimization is what are you optimizing for. Most people focus on the absolute return. Returns are great, but most traders care much more about how they make their money instead of how much. Let me give you an example. If I had a system that made $25,000 last year, would you want it? Almost everybody says yes.

If I have a system that made $25,000 last year, but you had to lose to $15,000 before you made any money. Most people don’t want that system. What this means is that you care a lot more about the performance on a day-to-day basis rather than end result. The problem with optimization and even walk forward optimization is that you’re not necessarily focused on what you care about in the real world: the way that you’re making your money.

Most charting packages focused on the net outcome and that can cause some weaknesses in your system. If you’re range trading, what you’ve really done is cherry pick the results that are the least affected by substantial news. In effect, you’ve chosen the settings that have not yet been affected by fat tails.

If you’re trend trading, you’ve done the exact opposite. You intentionally pick the settings that maximize the fat tailes that have happened in the past. With trend trading strategies, you probably aren’t going to find consistent performance. Instead, what you’ll find is that the optimization frequently causes long, ongoing droughts of incessant drawdown. Then suddenly, almost out of nowhere, it finds a mega monster winner that returns several multiples of the drawdown that you experienced. This is fine for a hypothetical backtests, but in the real world where you’re suffering losses on a near daily basis, most traders can’t take the pain. The weakness I find with most optimizations is that they don’t look at the consistency of performance. A potential substitute for optimizing a strategy would be looking at the linear regression of the equity curve over time. The best equity curve has the strongest linear regression slope.

Popular charting packages that implement rolling walk forward optimization are Amibroker, TradeStation, Multicharts and NinjaTrader.

Walk forward optimization in NinjaTrader

Open the Strategy Analyzer from the Control Center. Click File / New / Strategy Analyzer.

Open the strategy analyzer in NinjaTrader

Left mouse click on an instrument or instrument list and right mouse click to bring up the right mouse click menu. Select the menu item Walk Forward. You can also click on the “w” icon in the Strategy Analyzer toolbar. If you prefer hot keys, you can also use CTRL + W. Lastly, you can also push the “W” icon at the top left of the Strategy Analyzer.
Select a strategy from the Strategy slide out menu
Set the Walk Forward properties (See the “Understanding Walk Forward properties” section below for property definitions) and press the OK button.

There are many ways to select walk forward optimization in NinjaTrader

The Walk Forward progress will be shown in the Status Bar of the Control Center.

How Backtesting Biases Distort Expectations

June 10, 2013 by Andrew Selby Leave a Comment

“Why don’t my trading results ever seem to match my backtesting results? I followed the system perfectly!”

Anyone who has ever traded a thoroughly backtested systematic strategy has likely asked themselves this question. The answer is that despite all the effort that you put into coding and backtesting your system, there were likely a number of biases that crept in to skew your results.

Backtesting biases come in all shapes and sizes. Some are statistical, while others are psychological. Most stem from incredibly slight discrepancies. They can be very hard to notice, but can distort your expectations. Unchecked biases can lead you to believe that a losing system is relatively safe.

In order to keep these biases out of your backtesting results, it is important to know the most common forms and how they can creep into your testing. Here are the four most common biases that I catch myself struggling with.

Backtesting bias blinds traders to the reality of the markets.

Data Snooping Bias

Data Snooping Bias is also referred to as Optimization Bias or Curve Fitting. This bias is the result of refining too many parameters to improve a system’s performance on a single data set. Like most biases, Data Snooping is fairly easy to understand at face value, however it has a habit of subtly creeping its way into system development.

A common example of data snooping starts out as an honest effort to improve a system. You then test if adding another indicator into the mix will improve your results. If it does, then you incorporate that indicator into the system and test adding another indicator. The end result is a system that is perfectly optimized to trade the exact data set you tested it on. The problem is that the system is only optimized for that specific data set, which already happened.

The best way to avoid data snooping, or curve fitting, is to keep your systems simple, using as few parameters as possible. It is also important to backtest your system on many different data sets across different markets and time periods.

Look-Ahead Bias

Look-Ahead Bias happens when backtesting results are distorted because trading decisions are based on information that was not yet available. This generally happens because extra data is accidentally included and we simply don’t realize it. These errors are often very slight, which is why they are so easily overlooked. They can, however, have a dramatic impact on the overall results.

An example of Look-Ahead Bias would be using an entire data set to calculate optimal parameters for a strategy and then testing the strategy on that data set using those parameters. Because the parameters were calculated using the entire data set, they are technically using future data to make trading decisions.

Survivorship Bias

Survivorship Bias addresses the concept that many data sets do not include assets that are no longer listed. The result is that these data sets only include the assets that have survived to this point. This makes them skewed to the positive side because they do not include assets that have not survived.

A classic example of Survivorship Bias would be backtesting a stock trading strategy over a time period that included the dot-com crash. If stocks that went bankrupt during that crash are not included in the data set and your strategy would have been trading them, then your results are likely to be skewed.

The best way to avoid this issue is to purchase data that is free of Survivorship Bias. Another option is to use data that is more recent or markets that are less prone to Survivorship Bias.

Tolerance Bias

I’ve seen it written that every trader thinks that they will be able to withstand twice the drawdown that they will actually be able to tolerate. This is the root of Tolerance Bias, which is a trader overestimating his ability to tolerate drawdowns. This bias is completely psychological.

I provided a good example of this in my post about the 3 Day High/Low Mean Reversion System. In that post, I pointed out that the system had an overall drawdown of just over 15%. The system’s largest losing position was down over 18%.

While it is easy to look at the overall return of the system and agree that it was profitable, it is naive to think that it would not be a challenge to stick to the system at its low point. Imagine sitting on a 15% loss for the life of your system and holding a position that was down 18%. You would certainly be questioning whether markets had fundamentally changed in a way that made your system ineffective.

The best way to avoid tolerance bias is to dramatically underestimate your drawdown tolerance.

Analyzing Pairs with Correlation and Cointegration

May 10, 2013 by Rupert Hadlow Leave a Comment

Pairs Trading or Market Neutrality have long been seen as complex hedge fund style strategies with limited application for the retail trader. As part of our series on Correlation and Cointegration, we thought it would be beneficial to look at how both regression patterns can be used effectively to identify pair trading opportunities and scenarios, and how to reduce possible pitfalls.

The mystery that surrounds Cointegration and its complexity from a formulation point of view has somewhat put off many traders. Funnily enough, even as I write this post the spell check does not identify Cointegration as a word, which gives you an indication of how often the term is referenced.

Identifying Good Pairs Trades

Although there are a number of formulas or tests that can be used, one of the most widely adopted is that of the Augmented Dickey Fuller (ADF) Test. Formulating a p-value, the test allows the trader to identify how cointegrated two series are over a specified period in an efficient and simplified way.

To put this into context, if the prices for currency A and currency B are inputted into the model and the p-value comes out at 0.02, then this identifies that 2% of the time the two variables are not stationary or 98% of the time they are cointegrated. Make sure that you use a good number of values (eg 3 years on a daily chart), otherwise the calculation may not give you the most accurate indication.

There are a number of sites where you can download an excel version of the ADF test including www.quantcode.com

The next part in the calculation process is to work out the correlation. It is recommended that multiple time frames are used to also paint a picture of the cycle of the linear regression. To highlight how much of a discrepancy there can be, we ran scans on the EURUSD/GBPUSD and Gold/Silver Pairings. Results were very interesting.

Example 1: EURUSD and GBPUSD

30 Day Correlation:         73.46%
2 Year Correlation:         64.89%
13 Year Correlation:        89.20%
2 Year Cointegration:     13%

The long term correlation indicates that the pairs track a very similar path. However on a short term basis, they will gradually move apart. On a cointegration basis, only 13% of the time over the last two years have the pairs reverted back to the same mean.

Example 2: Gold and Silver

30 Day Correlation: 94.98%
2 Year Correlation: 26.99%
13 Year Correlation: 95.3%
2 Year Cointegration: 85%

The two year interval highlights how the prices are statistically out of their long term and short term ranges. According to the cointegration figure, the prices of both Gold and Silver reverted back to the same mean 85% of the time.

Conclusion: Gold and Silver is a better pairs trade than EURUSD and GBPUSD.

Cointegration and Correlation Technicals

Now that we determined that gold and silver show the highest correlation and cointegration, we need to analyse the technicals for specific entry and exit points. In the below example, there are three specific graphical representations (top chart is the gold price, middle chart is the regression correlation, and the bottom chart is the Silver price). The linear regression period is 360 days with no look back.

A chart showing gold, silver and their correlation

The diagram had two possible entry and exit points over the one year period. In August, the 360 regression channel indicated that the linear value would revert to its mean after a period above the 3^rd standard deviation.

According to the chart, a Short Gold and Long Silver pair signal would have been triggered. The second trade came in December, with the linear line breaking back through the bottom standard deviation channel.

Pitfall

With reference to the above example, we can see a number of issues that could greatly impact performance. Curve fitting as it is commonly known is best described as a specific series fitting a time variable without giving a true and clear picture of performance.

Strategies that require definitions from future pricing or large historical data are usually curve fitted. On paper it may look great, but in real trade scenario, the results may be completely different.

The 360 regression channel chart did not pass the out of sample (curve fitting) test. As can be seen below, when the look back period is amended to 162 Days, the signal would be completely different.

A moving window for the regression channel provides fewer opportunities for curve fitting

Solution

One of the solutions for curve fitting in pairs trading is to reduce the linear regression period to a shorter window or time frame. Although this can result in sensitivity to volatile movements, this reduces the potential risk to forward looking scenarios. Traders should also be aware of changes in the correlation and cointegration values of the pair, as these can shift quickly due to market mispricing or global economic and political events.

Curve Fitting

September 24, 2012 by Shaun Overton Leave a Comment

I generally avoid optimizing for philosophical reasons. Traders always care about the bottom line. The question people inevitably ask is “how much does the EA make?” Although understandable, I believe that such a black and white concern does not model real life.

In reality, people care very much not just about how much they make. They also care a great deal about how they make it. The difference is that they know the former and tend to ignore the latter. It’s the primary reason why traders cannot stick to successful systems. They tend to walk away whenever their personal pain threshold is exceeded.

Optimizing for balance is extremely risky because it ignores one critical factor: it’s entirely possible that the optimal strategy for a given time period is a random outcome. Short of nose diving to zero, it is very difficult to conclude a strategy’s merit solely on the basis of return.

More importantly, the process of parameter optimization implies that a magical combination of settings exists that’s just begging to get discovered. It’s plain fallacy to believe that using a 50 period moving average really makes a critical difference compared to a 51 period moving average. Yet I see traders all the time exhausting every possible outcome looking for historical sweet spots.

One test that I ran several years ago, and I wish that I had kept the screenshots, was to write a strategy that entered and exited the market purely at random. The only rule that I retained control over was the frequency.

The random process blew my mind. It was ultimately in a good way, but I realized that many of the strategies and expert advisors that I reviewed previously and deemed “not bad” were indeed truly worthless. 20 independent trials wound up with 20 different outcomes. Excluding commissions, most of the strategies neither made nor lost money. Most of the random trades exhibited wild round trips. Profits might run up 10%, decline to -10%, then settle near a 0% return. The profitable strategies stood out most in my mind. One or two of them resembled ATM machines that churned out consistent profits on most trades.

The easiest way to limit the risk of using random parameters is to look at the underlying indicators used in the decisions. Price crossing above or below a moving average, for example, exhibits a genuine entry and exit efficiency that NinjaTrader can evaluate (70% if used as a range bound exit strategy. Entries are random). Moving averages crossing one another, however, show an entry and exit efficiency of 50%. Using that metric, I confidently chuck that type of rule out the window.

If you’ve performed an analysis on the entry and exit efficiency and found something greater than 55%, then at that point optimizing is at least a less bad pursuit. I confess to using it in the past to look for hotspots on potential parameter settings. If you decide to go down that road, it’s important to consider them as suggestions to evaluate rather than a holy grail.

Signifigance Testing

May 16, 2012 by Shaun Overton Leave a Comment

I recently had someone from Portugal approach me with an Expert Advisor that he programmed. He felt that it was successful and wanted to get my opinion on its viability. I only know a small amount of information about the system, yet I was able to confidently reject it as unproven.

Statistics are the key. It provides a relatively straightforward toolbox to easily dismiss the simplest of systems. The measure in particular that I care about is called a student’s t-test.

Let’s first talk about when this test is appropriate. If you have fixed take profits and stop losses in place on 100% of all trades, then it is appropriate to use this test. The reason is that these values provide fixed limits on the outcome of trades. The distribution of your trades is likely to form a nice bell curve. In math terms, this is called a Gaussian distribution.

If your strategy uses dynamic exits based on market conditions, then this test is not appropriate. The distribution of forex, stock and futures prices do not follow a bell curve. The test depends on the assumption that the distribution under study follows a bell curve. If you make assumptions that don’t match what you’re measuring, then results are useless at best and dangerously misleading at worst.

If you’d like to go into the mathematics involved, then you can find numerous sources on the internet that explain t-tests. I also really enjoy the book Statistics by Freedman, Pisani and Purves.

Most trading systems do not follow bell curves, which makes the details of the t-test largely irrelevant. What’s useful about it is to show that you can generally feel better about the outcomes and predictions based on the number of results in the backtest.

Restrictions and Degrees of Freedoms

The goal of any test is to ensure that the results are accurate. The more frequently that you test a concept, the more confident that you feel about the outcome repeating itself consistently. The idea of predictability largely matches our intuitive expectations. If my co-worker shows up on time regularly, then I feel confident about him showing up on time tomorrow. If he shows up late regularly, then I know that he’s likely to show up late in the future.

The increase in experience increases the level of confidence. Eventually, the number gets so big that we feel very comfortable with the probabilities.

The Portuguese client presented a system based on moving averages with 3 filters, a stop loss and a take profit. Let’s assume that each filter only had one parameter. The number of restrictions for the buy trade is the moving average period (1), the one parameter for each of the three filters (3), the stop loss distance (1) and the take profit distance (1). This yields a total of 6 restrictions for buy trades. Assuming that sell trades use the same inputs, then we have a total of 12 restrictions.

We can’t begin counting our total number of trades (i.e., degrees of freedom) until the backtest shows at least 12 trades to account for our restrictions. It’s a good idea to not infer anything about a trading system until 30+ trades elapse. With the 12 restrictions in place, that sets the threshold for the minimum number of trades to reach a conclusion at 30 + 12 = 42.

Generally, it’s a good idea to see 300-400 trades before drawing conclusions about any system. The reason for this is that some events are very rare. They may only occur once every couple of hundred trials. Allowing the amount of information to approach this threshold allows the trader to more comfortably evaluate what hidden risks may be present.

The backtest that the Portuguese individual submitted only contained 27 trades. Knowing what we know about basic analysis, I comfortably decided that there is nowhere near enough information on the system to consider evaluating it.

A Word of Caution

An algorithm’s trading statistics are almost certain to change with time. Unless you have a mathematically sophisticated model for evaluating volatility, best practice demands that you evaluate the trading results in light of the type of volatility experienced. Making money in 2008 when the markets nose dived does not mean that you would have made money in 2010, which was much quieter in comparison. Any signal in 2008 that indicated a short would almost certainly show returns that include a handful of monster winners. If those monsters fail to show up because the volatility does not cooperate, then your expert advisor will more than likely flop.

3 Forex System Tips

April 23, 2012 by Shaun Overton Leave a Comment

Many systems nowadays promise profits without effort and easy pips right to your account. You probably know by now that real life doesn’t work that way. A genuinely profitable system is hard to come by. In this article I will describe three simple tips that help you enhance your existing trading systems. The goal is to make them more powerful and accurate.

Stop and Limit Entries to Catch Trends

Many trading systems use market orders to enter and exit the market. Market orders frequently exhibit low entry efficiencies. They get you in the market at a price that is not optimal. Placing a buy or sell order 10 pips from the price you wish to enter, in the direction of the trend for trending systems can make a big difference. For long trades put a buy order 10 pips higher than price, and for short trades put a sell order 10 pips lower than price. Range trading systems might consider limit entries, which would place the orders in the opposite direction of the example above.

Using ATR to Account for Volatility

Many novice system designers use constant pip distances in their forex trading system, i.e. 15 pips for stop loss, 10 pips for take profit, etc. This is a mistake as it doesn’t take into account changes in volatility. If the pair you trade exhibits changes in volatility, the system faces an increased likelihood of failure.

The Average True Range indicator (a.k.a. ATR) gives the average range of a forex pair or stock, and accounts for gaps as well. Instead of using constant numbers, use a percentage of ATR such as 50% ATR or 30% ATR. Once you do this change your system will automatically take into account volatility and will become much more flexible. Such systems will work better and will maintain profitability even in changing market environments.

Avoid Overoptimization

This is a tip especially for the programmers of you: over-optimization is the kiss of death of a trading system. Over-optimization, a.k.a. curve fitting, means that you add many Forex indicators and filters and use them all to confirm your signals, and optimize them all for maximum profits. In the backtest it will seem that your system is becoming better when in fact it will become good only for the past and will fail in any forward-test on real, live data. Therefore, it is crucial to only include the most important parts of your system and do not add indicators that don’t make sense at the price-action level. Remember the principle of Occam’s razor: “The simplest solution is usually the most efficient one”.

Michael Wells is an FX programmer and trader. His site contains his insights about Forex trading systems.

Optimize an Expert Advisor

February 20, 2012 by Shaun Overton 1 Comment

One of the lesser known features of the MetaTrader backtester is the optimization feature. It’s so small that you could be forgiven for overlooking it.

Optimization is the process to maximize a certain outcome. In this case, it’s profit. Any EA developer wants to maximize the amount of profit made over a given period of time. The MetaTrader optimizer allows the trader to search for the combination of inputs that yielded the maximum profit over a given period of time.

The process is identical to running a backtest, except that MT4 runs multiple backtests at the same time. It then organizes the results and offers up the best combination.

Telling the backtester to run in optimization mode is easy. Simply put a check next to the word Optimization. MetaTrader will then sort through the combinations that you tell it to consider.

Place a check in the box next to Optimization in the MT4 backtester

The next step is to click on the Expert properties button to the right. A new window appears that contains three tabs: Testing, Inputs and Optimization. These screens allow the trader to inform MetaTrader which variables to consider for testing and how to weight the results.

Testing

The top of the testing section applies to every type of backtest. Here you can select the starting balance. MetaTrader defaults the option to $10,000, although you can make this any amount of your choosing.

The second default option allows the trader to restrict the direction of trades. It’s a frequent expert advisor programming request. It’s also one that is unnecessary. Both the backtester and expert advisor options screen allow the trader the option of restricting trades to long only or short only without additional programming. If the EA is not well programmed, this setting may cause errors 4110 or 4100 to appear all over the trading journal. It’s harmless. The only effect should be that the backtester slows down. It’s the result of writing to the journal hundreds of times or more.

The testing tab of the MetaTrader backtester

A groupbox appears underneath these options that inexplicably relates to the optimization process. You’d think it would make more sense to place it in its namesake tab. That’s typical MetaQuotes logic at work.

The first line contains numerous parameters for choosing the best option. User overwhelmingly select for the largest account balance, but other options include the profit factor, expected payoff, maximum drawdown and drawdown percent.

The last line automatically uses a genetic algorithm. Optimization processes use either brute force methods or genetic algorithms. Brute force strikes most people as intuitive although obviously exhausting. The software tests every combination possible. Genetic algorithm’s attempt to make the process more intelligent. When the software sees that certain parameters almost inevitably lead to a losing performance, the algorithm skips similar tests where it expects to lose.

This is a great idea if you have a quality genetic algorithm. My opinion of the MetaTrader backtester is less than stellar. I don’t feel very confident about the algorithm at all. If you don’t mind spending extra time waiting for test results then I suggest unchecking this option. You don’t want to miss a potentially important combination.

Inputs

Most people find this screen confusing. The first column, called value, strictly controls inputs for simple backtests. The Value column is totally ignored during an optimization run.

The inputs tab of the MT4 backtester expert settings

The important columns for this task are Start, Step and Stop. Start is the lowest number that the MT4 backtester will consider. Step refers to the interval between the lowest value and the highest value. Tightly controlling this setting allows the user to gain quick insights into how changing the variable values affects performance without dragging the tests out for a full week. Stop is the highest number that the expert advisor will use.

The most obvious candidate for testing in this example is the Take Profit value. The default setting is listed at 50. If you trade the majors, you might want to consider settings ranging between 10 pips and 200 pips. That means that you set Take Profit row to 10 for the Start column and 200 for the Stop column. The real trick here is selecting the Step. If you choose Step = 1, then MetaTrader will run a separate test for every value between 10 and 200. That’s 190 tests, which is overkill. A step of 10 cuts the total number of tests down to 19.

Optimization

This section is the nit-picky part. If a trader feels it’s unacceptable to have 10 consecutive losses in a row, he can place a check next the the Consecutive wins box. MT4 automatically discards any tests which yield a result that contains anything checked off.

The optimization tab in the MT4 backtester allows users to discard tests with undesirable traits.

When you finish going through each of the tabs, push OK in the bottom right corner. It’s time to launch the tests.

Curve fitting in the MT4 Optimizer

A word of warning: my personal opinion is that optimizing an expert advisor is usually a very bad idea. The unique settings that yield the most profit in 2012 are unlikely to yield the most profit in 2013. If you don’t control for random chance, there’s a good probability that the 2012 best combination may result in catastrophic losses in 2013.

I recommend that traders pursue any strategy development work in NinjaTrader. I don’t like the idea of optimizing at all. Instead, I always focus on testing strategies for entry and exit efficiency. I know from years of experience that these values never fundamentally change on instruments of the charts traded. Entry and exit efficiencies make wonderful metrics for automated trading because they are so stable.