You absolutely must check your trading system’s performance on a regular basis. You’re going to miss most of the problems from watching your equity curve alone.
That almost happened to me a few weeks ago. When I observed my account, I noticed that the real results had dramatically underperformed the hypothetical results. A quick review showed me that I only took 271 trades over the prior week, whereas my backtest expected to find 360.
I was only trading 75% of the setups! What could explain the missing trades?
Finding the flaw
One feature that I wrote into the MetaTrader version of the Dominari was a maximum spread feature. I’m paying commissions, so the idea of the rare but possible scenario of paying a 10 pip spread to enter a trade seemed intolerable. I added a maximum spread feature to prevent getting ripped off.
I also didn’t put much thought into what happens if the spread is too wide. My initial instinct was to put the EA into hibernation for a few seconds. It would then wake up and check the spread. If the spread narrowed enough, it would send a market order. But in my haste to start trading, I forgot to also require that the price be near my original requested price. That design would have allowed the market to drift up 10 pips and then, if the spread narrowed, dramatically overpay to get in the trade.
The new method for capping the spread paid uses limit orders if the spread is too wide. The advantage to this method is that it solves two simultaneous problems. The first one is easy to understand. A limit order has a limited price. It’s not possible for the price drift described in the above paragraph to occur. I either get the price I want or the market moves without me and I miss the trade.
Equity curve since I made the execution changes on March 16.
The second advantage to using limit orders on entry is the fact that a limit order rests on the broker’s server. The hibernating method could potentially miss fractions of a second where the spread temporarily narrows to an acceptable price. Limit orders catch all price quotes, improving my theoretical likelihood of a fill.
Reality proved the theory after a week of trading. Instead of taking 75% of all possible signals, I’m now taking 87.5% of signals. That’s a result of the new limit method and my willingness to pay a wider spread to enter a trade.
More improvement
The question at the top of my mind was, “Should I be willing to pay even more to enter these trades?” Like a good quant, I immediately decided to calculate the question instead of haphazardly guessing.
I wrote a script in MetaTrader to search for every limit order in my account which was cancelled. I then looked at what the hypothetical performance of those trades would have been if I had simply paid the exorbitant spread.
It turns out that I should be willing to pay a lot more money to enter these trades.
There have been 50 cancelled limit orders within the past week, 44 of which were theoretically profitable. The average theoretical profit per trade was €1.28 compared to €0.33 for all executed trades. That’s a massive 287% difference in profitability!
The other shocker was the percent accuracy. 44 out of 50 implies an accuracy of 88%, compared to 64% accuracy on executed trades. 50 signals isn’t a lot. Am I getting too excited about missed profits or is that bad luck?
Basic statistics gives an answer with a high degree of precision. If the real accuracy is 64%, then you would expect to see 50 * 0.64 = 32 winning trades in a random sampling. My observed, theoretical accuracy with these limit orders was 44 orders out of 50, which is 88% accurate.
It turns out that I should be willing to pay a lot more money to enter these trades.
The standard deviation for 64% accuracy on 50 orders is 0.48, which we can then use to calculate the standard error. The standard error on 50 orders is sqrt(50) * 0.48 = 3.42 orders.
And finally, the standard error gives us enough information to compute the z-score. The z-score is the observed values-expected values/standard error, which is (44-32) / 3.42 = 3.5. A z-score of 3.5 has a probability of 0.000233 occurring due to random chance, or about 1 in 4,299 tests.
Conclusion: The statistics say with high confidence that my non-executed orders are substantially more accurate than my executed orders.
With the orders being both more accurate and having a higher per trade value, I increased the maximum spread that I’m willing to pay by 53%. While that sounds oddly precise, the per trade value might be substantially overestimated. I ball parked a guess that paying 40% in trade costs for a high quality trade seems reasonable. That number may have to go higher in order for me to measure the details.
Ideas for exploration
The amazing extrapolation from the live order analysis is that the spread seems to predict my likelihood of success. Wider spreads make me more likely to succeed and with a better risk:reward ratio. My project over the next few days will be to start logging my spreads at signal generation time to evaluate whether the spread predicts the profitability of my signals.
Oddly enough, there might even be a paradoxical outcome where narrow spreads predict my failure. More on that when I have enough data to answer the question.
I MAY NOT BE ASKING THE RIGHT QUESTION, HOWEVER, WHAT WOULD THE HIGHER SPREAD DO TO THE 75% EXECUTED TRADE PROFIT/LOSS? WELL WRITTEN AND THANKS.
My limit orders stipulate the WORST price that I’m willing to pay, and I only send them out when the spread is too wide.
Say that I’m watching EURUSD where the prices are 1.08500 bid and 1.08505 ask. If I was only willing to pay a 0.3 pip spread on EURUSD and I wanted to buy, then I’d set my limit order to 1.08500 (the bid) + 0.00003 = 1.08503.
Buy orders only use the ask. The current ask would have to decline from 1.08505 to 1.08503 in order for my trade to enter the market.
This is all very interesting and thanks for sharing. But, all of this is based on such a small sample of data it really could easily fall into the category of random chance.
So in answer to the question: “Am I getting too excited about missed profits or is that bad luck?” I’d say yes and no.
I do realize these types of issues are critically important for the kind of high volume, small target trading system you’re trying to employ. Which surprises me that commissions, spreads and the like weren’t REALLY thoroughly researched before launching some iterations ago especially after some—after reviewing your projected theoretical results—issued comments and warnings regarding the broker take with this kind of strategy.
Anyway, thanks again for sharing. Very informative.
That’s the point of doing the article. No, it most definitely is not the result of random chance. Such an occurrence would only happen once in every 4,299 trials (99.97% certainty that it’s not random).
Secondly, you can’t thoroughly research broker costs in advance. You have to push volume through their systems to get real numbers. Anything less than that is just pie in the sky, made up guesses. It’s better to spend some money and get real information instead of speculating.
Interesting results.
With an 88% profitable on just the trades you previously rejected vs 64% with the trades you did take along with a much higher return per trade maybe consider this for the coming week or two…
Trade your strategy unchanged along with a second identical strategy variant that ONLY takes the trades the first one would reject as ‘too expensive’ for a comparative analyses of the results. Trading the same account you would still be able to show the net P&L curve, but could easily differentiate between the expected profitability of the two methods. If your lower frequency but higher expected PPT results maintain a high correlation to your previous sample then you could consider increasing the entry size by a factored amount on the second strategy (bang for buck)
Hey Derek,
Good to hear from you again. I’m glad to see that you’re still lurking around the site.
I could do that, but there are practical considerations. The main one is that the portfolio size is already at the bare minimum. Anything less than $10,000 distorts the results due to rounding the lot sizes. I’m going to accomplish the same thing by recording spreads at the time a signal generates. If there’s a clear relationship between more spread = more profit, I’ll then have a solid, mathematical foundation for whatever course of action that I take next.
hello, Shaun Overton, congratulations for your blog.
I also had this problem on my system.
in backtest results are better, but in actual operations.
are not the same as in the backtest, fixed spreads are configured, even a maximum;
but the reality is different. also use pending orders.
which realized and concludes, it is that the brokers to see any sign of great volatility, almentam their spreads, but why? it is only because the volatility increased, I think not. the question is who with high volatility and rapid price, if you have a better probability of success on the part of traders, as the rapid price is much easier to win. and going in the right direction.
in this case the mechanical systems are at a disadvantage if you set a maximum spread value, I’m in the laboratory to look at the question nomeu mechanical system.
but I think a good logic test, it would be a dynamic spread limit depending on market conditions, creating an adaptive heuristic, if volatility increases, and in times of economic announcements., is just an idea
Thanks, Sandro.
I agree that spreads reflect the underlying market condition. It’s not an idea that I would’ve arrived at from a thought experiment. I would’ve viewed wider spreads as a result of wider volatility instead of a profit indicator. I’m still not convinced that spreads will predict my system profitability, but there’s enough evidence for me to spend the time ruling it out.
In your discussion of Z score
“And, positive Z indicates that the profitable trade will be followed by another profitable one, and a loser will be followed by another loss. This observed dependency lets the forex trader vary the position sizes for individual trades in order to help manage risk.”
That should be negative Z I think.
Where does it say that in the article?