statistics Archives - Algorithmic and Mechanical Forex Strategies

One line of code makes all the difference

February 9, 2017 by Shaun Overton 4 Comments

I was really excited about my Pilum strategy two months ago. The research looked great and everything was ready to rock and roll. Demo testing began and then… not much happened.

The Quantilator is (mostly) finished, which finally gave me time to circle back and review what happened with Pilum.

Live demo trading of Pilum. Dec 9, 2016 to Feb 7, 2017

The expected outcome was that I would win 75% of the time. Trades were infrequent, so I thought maybe I’m just having bad luck. But then my win rate remained stuck around 50%. Simple statistical tests told me this was unlikely to be bad luck.

I used the research time to pour over my research code and to compare it with live trades. What I found was that a single line of code (AHHHHHHHHHHHHHHH!) was incorrectly calculating my entry price, dramatically overstating the profits.

The flawed code produced this equity curve from a single combination of settings:

When the actual, correct result looks like this with those same settings:

The accurate backtest of Pilum

I’ll be honest… I like the flawed backtest a lot more!

The new, single-setting backtest isn’t as good, but it’s still trade-worthy. There are some characteristics that I dislike and features that I love. Let’s dig into those.

What I dislike

The frequency of trades is very low. Out of 19 months there were a total of 43 trades. 43 trades to comprise a backtest on 40+ instruments is a very small number.

If it weren’t for the statistical pattern backing up the frequency, I would not consider the test. However, there are 20,000 bars each on the 44 instruments. There are 880,000 total bars used to analyze whether my Pilum pattern offers any predictive value.

The most valuable predictions, however, are also exceptionally rare. That’s why I’m not able to get the trading frequency higher, which would potentially smooth the returns.

What I love

My previous systems like QB Pro and Dominari traded actively for relatively small wins. Trading costs exercised a massive impact on the overall performance.

The accurate backtest of Pilum

Now look again at the correct equity curve (the image to the right). Do you see the final profit of roughly 0.14? That’s a 14% unleveraged return over a 19 month period.

Allocating 2:1 or 3:1 leverage on this strategy could average annual returns of 15-25%.

Detecting hidden risk

A key measure of risk is skewness. You may not use that term yourself, but it’s something most of you already understand. The biggest complaint about people trading Dominari was that the average winner relative to the average loser was heavily skewed towards the losers.

Dominari wins on most months, but when it lost in December it was devastating. I implemented what I thought was a portfolio stop after the December 9th aftermath. Then I had a smaller, but still very painful, loss in January. The portfolio level stop loss of 3% should prevent future blowouts now that I know what goes wrong.

I still believe in Dominari. But, I obviously lost the work of most of the year due to those events.

Knowing that skewness is a good measure of blowout risk (even if you’ve never seen it in a backtest, like happened with Dominari), Pilum looks extremely encouraging.

This is a histogram of profit and loss by days. You should notice a few things.

The tallest bar is to the right of 0. That means that the most frequent outcome is winning.

The biggest winning day is dramatically better than the worst losing day. The worst outcome was a loss of 2%. The best outcome is gains near 10% in a single day (unleveraged!).

This is the statistical profile of an idea that’s much more likely to grab an avalanche of profits than it is to get blown out.

It gets even better

Would you say that the blue and red equity curves are highly or loosely correlated? Look closely.

Writing this blog post made me think carefully about the Pilum strategy. I decided that maybe I should see if all of the profits are coming from different settings at the same time. There’s very little risk of overfitting the data as my strategy only has 1 degree of freedom.

The blue bars are the equity curve of Setting 1.

The red bars are for Setting 2.

Do you think these are tightly or loosely correlated?

If you said loosely correlated, then you are correct. Notice how each equity curve shows large jumps of profit. Did you notice how those profit jumps occur on different days?

The blue setting skyrockets on a single day in November 2016. It leaves the red equity curve choking in its dust.

But then, look what happens as I advance into December. The red curve dramatically catches up to the blue curve and even overtakes it.

The correlation between the 2 strategies is only 57%.

Combine multiple settings into 1 portfolio

This is a much nicer equity curve!

Loose correlations are a GIFT. Combining two bumpy equity curves into a single strategy makes the performance much, much smoother.

The percentages of days that are profitable also increases. Setting 1 is profitable on 58.0% of days. Setting 2 is profitable on 53.5% of days.

But… combining them makes Pilum profitable on 68.2% of days. Awesome!

That also provides more data, which puts me in a stronger position to analyze the strategy’s skewness. Look at the frequency histograms below. They’re the same type of histograms that I showed you in the first section of this blog post. As you’ll notice, they look a lot different.

Pilum most probable daily profit and loss

The most probable outcome for any given day is a small winner

The tall green bar is the most probable trading outcome for any given day with filled orders. The average day is a positive return of 0-1%.

The small red bar is the worst trading day of the combined strategy.

The small green bars are the best trading days of the combined strategy.

Look how far to the right the green bars go. The largest winner is more than 3x the biggest loss. And, there are so many more large winners compared to losers.

Giant winners are far more likely than comparable losses.

The Plan

I immediately pushed Pilum into live trading this combination of two strategies. I expect that adding a second degree of freedom and running about 30 different versions of the strategy – all with different settings – will add to the performance and smooth the returns even further.

Dominari hasn’t been working on my FXCM account, which is very difficult to accept because the lacking performance seems to be a buried execution issue. Pilum, however, trades very infrequently. It’s unlikely that execution quality will make a dramatic difference in the long term outcomes.

So, I’m going to convert the FXCM account to trading Pilum exclusively. That will be offered as a strategy on Collective2 within the next few weeks, a company with whom I’ve been working closely. Their users are more investor rather than trading oriented – they’re far more likely to view low trading frequency as a good thing. I suspect that most people here have a different opinion and want to see a lot of market action.

I’ll write an update on Dominari shortly.

How badly do I want in?

March 22, 2016 by Shaun Overton 10 Comments

You absolutely must check your trading system’s performance on a regular basis. You’re going to miss most of the problems from watching your equity curve alone.

That almost happened to me a few weeks ago. When I observed my account, I noticed that the real results had dramatically underperformed the hypothetical results. A quick review showed me that I only took 271 trades over the prior week, whereas my backtest expected to find 360.

I was only trading 75% of the setups! What could explain the missing trades?

Finding the flaw

One feature that I wrote into the MetaTrader version of the Dominari was a maximum spread feature. I’m paying commissions, so the idea of the rare but possible scenario of paying a 10 pip spread to enter a trade seemed intolerable. I added a maximum spread feature to prevent getting ripped off.

I also didn’t put much thought into what happens if the spread is too wide. My initial instinct was to put the EA into hibernation for a few seconds. It would then wake up and check the spread. If the spread narrowed enough, it would send a market order. But in my haste to start trading, I forgot to also require that the price be near my original requested price. That design would have allowed the market to drift up 10 pips and then, if the spread narrowed, dramatically overpay to get in the trade.

The new method for capping the spread paid uses limit orders if the spread is too wide. The advantage to this method is that it solves two simultaneous problems. The first one is easy to understand. A limit order has a limited price. It’s not possible for the price drift described in the above paragraph to occur. I either get the price I want or the market moves without me and I miss the trade.

Equity curve since I made the execution changes on March 16.

The second advantage to using limit orders on entry is the fact that a limit order rests on the broker’s server. The hibernating method could potentially miss fractions of a second where the spread temporarily narrows to an acceptable price. Limit orders catch all price quotes, improving my theoretical likelihood of a fill.

Reality proved the theory after a week of trading. Instead of taking 75% of all possible signals, I’m now taking 87.5% of signals. That’s a result of the new limit method and my willingness to pay a wider spread to enter a trade.

More improvement

The question at the top of my mind was, “Should I be willing to pay even more to enter these trades?” Like a good quant, I immediately decided to calculate the question instead of haphazardly guessing.

I wrote a script in MetaTrader to search for every limit order in my account which was cancelled. I then looked at what the hypothetical performance of those trades would have been if I had simply paid the exorbitant spread.

It turns out that I should be willing to pay a lot more money to enter these trades.

There have been 50 cancelled limit orders within the past week, 44 of which were theoretically profitable. The average theoretical profit per trade was €1.28 compared to €0.33 for all executed trades. That’s a massive 287% difference in profitability!

The other shocker was the percent accuracy. 44 out of 50 implies an accuracy of 88%, compared to 64% accuracy on executed trades. 50 signals isn’t a lot. Am I getting too excited about missed profits or is that bad luck?

Basic statistics gives an answer with a high degree of precision. If the real accuracy is 64%, then you would expect to see 50 * 0.64 = 32 winning trades in a random sampling. My observed, theoretical accuracy with these limit orders was 44 orders out of 50, which is 88% accurate.

It turns out that I should be willing to pay a lot more money to enter these trades.

The standard deviation for 64% accuracy on 50 orders is 0.48, which we can then use to calculate the standard error. The standard error on 50 orders is sqrt(50) * 0.48 = 3.42 orders.

And finally, the standard error gives us enough information to compute the z-score. The z-score is the observed values-expected values/standard error, which is (44-32) / 3.42 = 3.5. A z-score of 3.5 has a probability of 0.000233 occurring due to random chance, or about 1 in 4,299 tests.

Conclusion: The statistics say with high confidence that my non-executed orders are substantially more accurate than my executed orders.

With the orders being both more accurate and having a higher per trade value, I increased the maximum spread that I’m willing to pay by 53%. While that sounds oddly precise, the per trade value might be substantially overestimated. I ball parked a guess that paying 40% in trade costs for a high quality trade seems reasonable. That number may have to go higher in order for me to measure the details.

Ideas for exploration

The amazing extrapolation from the live order analysis is that the spread seems to predict my likelihood of success. Wider spreads make me more likely to succeed and with a better risk:reward ratio. My project over the next few days will be to start logging my spreads at signal generation time to evaluate whether the spread predicts the profitability of my signals.

Oddly enough, there might even be a paradoxical outcome where narrow spreads predict my failure. More on that when I have enough data to answer the question.

Random Trailing Limit

March 12, 2012 by Shaun Overton 14 Comments

I got the idea for a trailing limit from Van Tharpe’s market classic Trade Your Way to Financial Freedom. I made Jon Rackley read through the book as part of his training when I first hired him. He brought my attention to an unusual claim made on page 267. Van Tharpe says that it’s often possible to make money even with random numbers.

Random numbers are a pet theory of mine. I’ve long put effort into figuring out whether I could develop a strategy that trades totally at random and still makes money. As the owner of a programming company for traders, and as part of Jon’s initial training for NinjaTrader in December, I assigned him the task of programming the strategy into code.

The book states that trading with totally random entries and a 3 ATR trailing stop generally leads to making money. Flip a coin. You go long if it lands on heads. Go short if it lands on tails. One important note is that we elected to use pure random numbers in our programming instead of the pseudo-random numbers that computers generate. Doing so allows us to avoid time biases in the seeding process that generally pop up when the seeds used are close together in time.

The first working version that I reviewed displayed everything contrary to Van Tharpe’s claims. Using a trailing stop, regardless of the instrument and time frame tested, inevitably led to devastating losses. The profit factors consistently came in near 0.7, a truly awful number.

We did what most of our clients do when they find abysmal strategies. We flipped the strategy on its head. The new strategy uses only a 3 ATR (50 period) trailing limit.

Trailing Limit Analysis

The early conclusion is that trailing limits seem to offer a great deal of potential. Although Jon sent me the code several months ago, it’s only this evening that I had a chance to properly review and test everything.

The profit factors are very encouraging. It depends on the chart that I tested. The worst that I found was a 1.0 profit factor. Everything else came out with profit factors greater than 1. The small table below contains the initial test results. Before you go off salivating that this is the next hot winner, there are a few considerations to keep in mind:

The results depend entirely on the sequence of random numbers used. Using different sets of random numbers will cause different outcomes.
The better results on the higher time frames likely result from sampling error. The number of trades involved was only ~160, which is not enough to make definitive conclusions on the nature of the performance. I prefer to see 400 or more trades before drawing definitive conclusions.
These results do not include spread costs or commissions.

Currency	Period	Profit Factor	Dates Tested
EURUSD	M1	1.12	9/12/2011-3/12/2012
GBPUSD	M1	1.16	9/12/2011-3/12/2012
USDJPY	M1	1.25	9/12/2011-3/12/2012
EURUSD	M5	1.11	2011
GBPUSD	M5	1.0	2011
USDJPY	M5	1.06	2011
EURUSD	H1	1.26	2011
GBPUSD	H1	1.25	2011
USDJPY	H1	1.48	2011

Random trades sometimes produce solid looking equity curves. Remember that this was generated with purely random numbers and a trailing limit

What made me feel better about the results was reviewing the entry and exit efficiencies of each strategy. The number of trades involved really cluttered the graphic. I ran a backtest on a much shorter time period so that the horizontal, blue line would appear clearly on each graph.

The entry efficiency of a random entry is… random (45.45% entry efficiency)

The exit efficiencies of random entries with traililng limits are outstanding

The exit efficiencies of random entries with trailing limits is off the charts. (77.73% exit efficiency)

The entry efficiency tells us exactly what we would expect to find. Trades which enter at random do not perform better than random (obviously). What’s interesting is how the trailing limit exit strategy actually skews the entry efficiencies to read slightly worse than random. This is a good example of why it’s dangerous to rely purely on statistics. Keeping the big picture in mind reduces the likelihood of making an erroneous conclusion, in this case that there might be something “wrong” with our perfectly random entries.

The exit strategy, which is what we’re truly testing, looks absolutely stellar. It shows that acting as a conditional probability allows for a great deal of adverse movement while capturing extreme points of the move.

Adding Money Management

The percentage of winners for the tested charts ranges from 60-67% accuracy. High percentages of winners often lead to winning streaks. My cursory glance through the chart led to my easily finding a suitable example to cherry pick. Here, 5 trades in a row reach their near maximum take profits.

Picking trades at random occasionally leads to streaks of lucky winners

Testing that I’ve done in the past tells me that it’s often advantageous to increase the position size based on consecutive winners when a strategy is more than 50% accurate. My first idea after reviewing the initial results was to modify the forex money management strategy to pursue the consecutive winners. We unfortunately do not have time to pursue those changes in the code right now. Let me know if you’re interested in seeing this and we’ll make it a priority if enough readers respond.

While increasing the risk after consecutive winners works out statistically in your favor, it does add to the risk. It’s entirely possible for a “winning” set of trades to start losing based solely on the money management strategy. There’s no way to know whether your set of trades will be the luck beneficiary of the extra risk or whether it will be the unlikely loser.

Conclusion

Everyone talks about stops. You always have to have a stop. Blah blah blah. I know it’s going to be the first question that everyone asks.

My research shows that trading with a stop is for suckers. Larry Connors’ book Short Term Trading Strategies That Work was the first trading book where I actually felt like I read valuable information. One of my favorite sections is on stop losses. His research shows that using a stop loss (even a 50% stop loss) always reduces the performance of a strategy. My own independent analysis bears this out.

Not using stops does not mean never taking losses. On the contrary, refusing to exit the market at a loss makes for a 100% odds of blowing up one day. The point of using a stop or limit is to empirically define, and to never waver from, a specific point or points in the market at which you will exit. A trailing limit accomplishes that goal admirably.

Interestingly, the trailing limit is one feature which FAP Turbo incorporates that I have yet to find in any other expert advisors. Although I’m not a fan of FAP Turbo type of strategies, it certainly does interest me that one of its main features aligns with this research. Also notable is the fact that the trailing limit continues down even to the point where it accepts losses.