Algorithmic and Mechanical Forex Strategies | OneStepRemoved

  • Articles
  • Sophisticated Web Sites
  • Automated Trading
  • Testimonials
  • Contact

How badly do I want in?

March 22, 2016 by Shaun Overton 10 Comments

You absolutely must check your trading system’s performance on a regular basis. You’re going to miss most of the problems from watching your equity curve alone.

That almost happened to me a few weeks ago. When I observed my account, I noticed that the real results had dramatically underperformed the hypothetical results. A quick review showed me that I only took 271 trades over the prior week, whereas my backtest expected to find 360.

I was only trading 75% of the setups! What could explain the missing trades?

Finding the flaw

One feature that I wrote into the MetaTrader version of the Dominari was a maximum spread feature. I’m paying commissions, so the idea of the rare but possible scenario of paying a 10 pip spread to enter a trade seemed intolerable. I added a maximum spread feature to prevent getting ripped off.

I also didn’t put much thought into what happens if the spread is too wide. My initial instinct was to put the EA into hibernation for a few seconds. It would then wake up and check the spread. If the spread narrowed enough, it would send a market order. But in my haste to start trading, I forgot to also require that the price be near my original requested price. That design would have allowed the market to drift up 10 pips and then, if the spread narrowed, dramatically overpay to get in the trade.

The new method for capping the spread paid uses limit orders if the spread is too wide. The advantage to this method is that it solves two simultaneous problems. The first one is easy to understand. A limit order has a limited price. It’s not possible for the price drift described in the above paragraph to occur. I either get the price I want or the market moves without me and I miss the trade.

Equity curve since I made the execution changes on March 16.

Equity curve since I made the execution changes on March 16.

The second advantage to using limit orders on entry is the fact that a limit order rests on the broker’s server. The hibernating method could potentially miss fractions of a second where the spread temporarily narrows to an acceptable price. Limit orders catch all price quotes, improving my theoretical likelihood of a fill.

Reality proved the theory after a week of trading. Instead of taking 75% of all possible signals, I’m now taking 87.5% of signals. That’s a result of the new limit method and my willingness to pay a wider spread to enter a trade.

More improvement

The question at the top of my mind was, “Should I be willing to pay even more to enter these trades?” Like a good quant, I immediately decided to calculate the question instead of haphazardly guessing.

I wrote a script in MetaTrader to search for every limit order in my account which was cancelled. I then looked at what the hypothetical performance of those trades would have been if I had simply paid the exorbitant spread.

It turns out that I should be willing to pay a lot more money to enter these trades.

There have been 50 cancelled limit orders within the past week, 44 of which were theoretically profitable. The average theoretical profit per trade was €1.28 compared to €0.33 for all executed trades. That’s a massive 287% difference in profitability!

The other shocker was the percent accuracy. 44 out of 50 implies an accuracy of 88%, compared to 64% accuracy on executed trades. 50 signals isn’t a lot. Am I getting too excited about missed profits or is that bad luck?

Basic statistics gives an answer with a high degree of precision. If the real accuracy is 64%, then you would expect to see 50 * 0.64 = 32 winning trades in a random sampling. My observed, theoretical accuracy with these limit orders was 44 orders out of 50, which is 88% accurate.

It turns out that I should be willing to pay a lot more money to enter these trades.

The standard deviation for 64% accuracy on 50 orders is 0.48, which we can then use to calculate the standard error. The standard error on 50 orders is sqrt(50) * 0.48 = 3.42 orders.

And finally, the standard error gives us enough information to compute the z-score. The z-score is the observed values-expected values/standard error, which is (44-32) / 3.42 = 3.5. A z-score of 3.5 has a probability of 0.000233 occurring due to random chance, or about 1 in 4,299 tests.

Conclusion: The statistics say with high confidence that my non-executed orders are substantially more accurate than my executed orders.

With the orders being both more accurate and having a higher per trade value, I increased the maximum spread that I’m willing to pay by 53%. While that sounds oddly precise, the per trade value might be substantially overestimated. I ball parked a guess that paying 40% in trade costs for a high quality trade seems reasonable. That number may have to go higher in order for me to measure the details.

Ideas for exploration

The amazing extrapolation from the live order analysis is that the spread seems to predict my likelihood of success. Wider spreads make me more likely to succeed and with a better risk:reward ratio. My project over the next few days will be to start logging my spreads at signal generation time to evaluate whether the spread predicts the profitability of my signals.

Oddly enough, there might even be a paradoxical outcome where narrow spreads predict my failure. More on that when I have enough data to answer the question.

Filed Under: Dominari Tagged With: execution, limit, quant, slippage, standard deviation, standard error, statistics, Z-score

Monitoring my live trade execution

February 10, 2016 by Shaun Overton 29 Comments

Dominari’s biggest risk is its trading costs. In the midst of losing 6 days in a row, I found myself extremely concerned about Dominari’s performance. Did the signals go bad all of a sudden or is this a normal drawdown? Is Dominari losing because of trading costs?

I decided to start analyzing my FXCM account. Part of the nerves were driven by the fact that it took 2 weeks to setup the account. The compliance process took far longer than usual because I’m a former employee. Two weeks later, I turned on the account just in time to a) miss the biggest equity growth and b) to catch the biggest drawdown.

I felt more hostile to the FXCM account performance because I didn’t have any profits to pad the losses. This is all coming from my original risk capital. And I’m having my third child soon. Giving birth to kids in the US is incredibly expensive. I’ve got better uses for the money than to throw it away in the markets!

So, the real question is: am I losing because it’s just a rough patch or because FXCM is eating my lunch?

backtest-equity-fxcm

This image is a backtested equity curve over the same period of my live performance. I’ve traded live since January 28, but the trading didn’t begin until the afternoon. As you can see, I again missed another patch of strong performance.

The rest shows something of a fairy tale. The backtest shows a return of 19.13% over that period, whereas my live performance is down 10%. How much of that is due to commissions, spread, rollover and slippage?

The backtest shows a profit of $956.65 with no trading costs.

My real results, which 1) show a profit on the backtest but 2) are actually showing a loss in real life, can be used to estimate a floor for my trading costs. The formula for that is
( Total profit and loss + commissions + rollover) / total trades, which is currently $1.58 in costs per trade.

The commissions and rollover are easy to separate out using either Myfxbook or the FXCM account report. The grand total spent so far on commissions is -$239.80 and -$3.05 on rollover.

The hardest part to separate is the spread paid. I’m not recording the spread paid on every trade (maybe that’s a mistake and I need to add it). But I’m going to use the table below to estimate. I took a random sample of 30 trades from the 501 trades completed at the time of my analysis.

Spread PaidSlippage
0.0001985231.49E-05
0.000153951-5.13E-05
0.0004558230.000227912
9.98E-050
0.000161242-0.00313413
2.76E-05-9.19E-06
5.55E-056.94E-06
0.000110898-1.01E-05
9.24E-050
9.91E-05-1.57E-16
6.55E-051.31E-05
4.85E-052.08E-05
8.22E-05-1.67E-16
6.87E-050
6.95E-05-1.65E-16
0.00015173-2.17E-05
9.43E-05-2.36E-05
9.38E-05-0.00225922
7.61E-05-0.0024735
0.0001600381.00E-05
0.000135020
0.0035426254.52E-05
0.000222978-0.00376275
7.62E-050
0.0004327977.73E-06
2.61E-050

The average slippage (the right column) is a stunning -0.044%. I’m getting negative slippage on average with FXCM. That’s outstanding! FXCM is improving my fills even though my entries are requested at a worse price. Whatever misgivings I’ve had about FXCM in the past are alleviated. That’s impressive execution.

Estimating the spread paid is much more difficult. I’ve chosen to take my average trade profit on a $5,000 account as the starting point. The trouble is that the value of an average winner can depend on the account performance. If I use stagnant position sizing, then the drawdown doesn’t effect the value of the average winner. Under that assumption, the average winner is $3.48 per trade.

But if I use compound position sizing, the drawdown eats away most of the profits. That drops the average trade value down to $1.70.

I converted the spread paid from pips into percentages. Using EURUSD as an example, a 1 pip spread works out to 0.0001/1.12727 = 0.000089. The reason for doing this is so that I can compare the spread on EURUSD to something with a much wider spread like AUDNZD. The spread is wider on AUDNZD, but the value of a NZD pip isn’t the same as a USD pip. Percentages allow for an apples to apples comparison.

The average spread paid in my sample was 0.00026157605, which is 0.026%. Putting that back into terms relative to my account balance, I’m paying 0.026% * $5,000 = $1.31 per trade in spread. Across 420 trades, that’s -$550.20 in spreads.

Total costs are spread, commissions and rollover:
$550.20 + $239.80 + $3.05 = $793.05

On a per trade basis, that is $1.78 in costs per trade from my estimates.

The total profit on the backtest was $956.65, but I missed about $550 of it because trading didn’t start until 17:00 on the 28th of January. That leaves the backtest profit somewhere around $406.65.

That puts the re-estimated profit and loss at $406.65-$793.05 = -$386.40. The actual loss is -$469, which I feel is a reasonable discrepancy based on the fact that I’m estimating how much profit was contributed on January 28 instead of knowing for certain.

The conclusion is that I need to turn off this trading at FXCM. Even if I joined their active trader program and traded in the top tier, it would only save me half the commissions. Most of the trading costs are in the spread and not commissions. I’m seriously considering a move to a broker that will allow me to make a market by posting limit orders. But first, I’ll need to go over my Pepperstone account to review the trading costs for myself and clients.

Filed Under: Dominari, Test your concepts historically Tagged With: backtest, FXCM, Rollover, slippage, spread

Limit Order Book

August 28, 2013 by Timothy Lewkow 3 Comments

I remember the first time that I really sat down and thought about it. Why exactly does a stock price change? Shrinking the economy and the number of shares helped. Examples starting with 10 oranges together with supply and demand arguments sparked good ideas. But, expanding a simple scenario into a full blown economy with high volumes and different order types never made any sense.

The story is not complete without considering the information contained in a limit order book. It’s the absolute best source for highlighting buying and selling power in a market in real time. The information within the data often results in more desirable entry and exits points.

A simple example of a limit order book

limit order book example

The orange squares represent units of stock that you can buy at market

Suppose that each block represents one share of stock on both the bid and ask side of the market frozen in time. The volume of shares in the above plot are limit orders waiting for execution or cancellation.

Say that Frank comes along and wants to buy 5 shares using a market order. In that case, his order will be filled immediately.

Remarkably, the current quote displayed  of $20,26 is not where Frank can trade- there are no shares available at that price. The quoting convention reflects the spread rather than tradeable prices.

The 5 empty colored squares represents the 5 shares that Frank bought with his market order

The 5 empty colored squares on the right represents the 5 shares that Frank bought with his market order

The order is filled by sellers in a first in, first out (FIFO) process. Those who waited the longest in the order book receive the first execution.

Frank’s market order for 5 shares receives execution at two different prices. The first 2 shares fill at $20.27. The depth of market at that price is only 2 shares, forcing him to sweep the $20.27 price and move on to the next available price at $20.28.

4 total shares are available at $20.28. Because Frank only needs an extra 3 shares, he completes his total order at this level.

The best offer displayed when Frank placed the trade was $20.27, but his average fill is $27.276 (2 * $20.27 + 3 * $20.28). The slightly worse price doesn’t have anything do with slimy brokers. Slippage is the natural result of buying more shares than there are shares available.

Try making Frank a more aggressive buyer. Say he wants nine shares. Large orders receive worse fills because they suck up most of the liquidity on one side of the market.

Why A Spread Exists

Before answering this question, it is first worth understanding the difference between a quote driven and an order driven market.

Order driven market:

• Displays all of the current bids and asks across the market

• Has complete transparency

Quote driven market:

• Displays bid and ask prices from market makers, dealers, or specialists.This is the norm among retail forex brokers

• Often provides a guarantee that an order is filled

A quote driven market has more moving parts and will likely be involved in any market you wish to trade in. Therefore, it is a good idea to think about the existence of a spread in this setting.

When you post an order in a quote driven market, the dealer will either fill it with their own inventory or match you with another market participant. For this reason, part of your transaction cost goes to the dealer who has done this work for you.

In a simple model, the bid ask spread is the price that aggressive traders must pay to have their order immediately filled– think buying and selling the same security at almost the same instant. The spread is the compensation to a dealer for offering that immediacy.

A good way to think about the size of the spread is to consider a market with several competing dealers. In this case, there are two primary scenarios:

1. If the spread is too high, more dealers will enter the market to gain profit from the large bid/ask spread

2. If the spread is too low, dealers will lose money, and exit the market

These two factors ensure that the liquid market dealers make normal profits, and that spreads are of reasonable size.

Supply and Demand

The existence of a spread is quite natural and leads back to the simplicity of supply and demand. Start the argument small and work your way up! I found a great example in an article by Glenn Curtis on August 19 with the following story.

Suppose that a one-of-a-kind diamond is found in the remote countryside of Africa by a miner. An investor hears about the find, phones the miner and offers to buy the diamond for $1 million. The miner says she wants a day or two to think about it. In the interim, newspapers and other investors come forward and show their interest. With other investors apparently interested in the diamond, the miner holds out for $1.1 million and rejects the $1 mil- lion offer. Now suppose two more potential buyers make themselves known and submit bids for $1.2 million and $1.3 million dollars, respectively. The new asking price of that diamond is going to go up. The following day, a miner in Asia uncovers 10 more diamonds exactly like the one found by the miner in Africa. As a result, both the price and demand for the African diamond will drop precipitously because of the sudden abundance of the once- rare diamond.

Imagine the diamond becomes more popular. More buyers want it. More mines open, and more sellers emerge. In a rational setting, this creates a quote driven a bid ask spread. Add enough volume, and before long, you are back to the first example.

Filed Under: What's happening in the current markets? Tagged With: ask, bid, FIFO, limit, order book, slippage, spread

System Expectations

September 19, 2012 by Shaun Overton Leave a Comment

I found myself outside today looking for an excuse to make a video. It’s 80°F / 27°C, sunny skies and a soft breeze in Dallas. The last place anyone wants to be is in front of the computer when the weather is this nice.

No doubt that some active daytraders or people that hate their jobs are thinking the same thing. I suspect that the motivation for most people making automated expert advisors is the dream of making money without doing anything. Turn on the software and wait for the trading profits to roll in. That was certainly the case with the company Forex Made Sleazy… I mean, Forex Made Easy several years ago.

We do have a handful of customers that trade profitably, but even then, it takes a long time for an automated system to get to the point where it’s largely hands off. The best conceived ideas, which I would define as plausibly worthy of my own investment funds, takes a bare minimum of several months to execute from start to finish. This also presumes the unlikely notion that the idea has genuine potential to start with.

Even the most simple, valid concepts encounter substantial setbacks before the system can truly run hands-free. It’s usually not some kind of epic programming disaster where the client wants black and the programmer makes white. Don’t get me wrong; communication is critical. The smoothest projects are always the ones where both parties understand one another readily.

Nonetheless, even the most well-oiled team experiences countless hiccups in the process of morphing from idea to reality. Simple ideas often fall the most vulnerable to real world problems. Trade execution stands out as the most common obstacle. If anything goes remotely unexpected, a potentially profitable scenario may lead to unexpected losses.

I worked with one client that came up with a simple idea that mathematically showed a heavy positive expectation. Yet when we launched the idea in the real world, the prices that the system absolutely required in order to function never came through. Slippage occurred precisely when it was the most damaging.

We had to go back to the drawing board looking for ways to re-engineer the expert advisor where the importance of execution declined. That setback alone took several months to overcome in any meaningful sense.

The take away here is that it’s totally unreasonable to expect to hire a forex programmer and expect a dramatic shift in profits and life style. The best ideas take several months before they are worthy of running their full account balance. Unfortunately, most of the ideas out there are not good to begin with. That’s why making an EA that is profitable over the long run is so incredibly difficult.

Filed Under: Trading strategy ideas Tagged With: programming, slippage, system, trading

Max spread and slippage

December 21, 2011 by Shaun Overton Leave a Comment

Many novice traders mix up the distinction between the slippage and max spread. The spread refers to the trading cost. Designating a maximum spread forbids an expert advisor from entering orders whenever the cost of doing so exceeds a certain threshold.

Max spread

Forex spreads often widen around news events. It’s frequently a great deal of chaos where the end result is not much different from where it all started. Many traders find it preferable to sit out these events. It’s better to miss a trading opportunity than to pay an arm and a leg for it.

Max slippage

Slippage controls the execution of the order. MetaTrader offers a unique feature in the OrderSend() command called slippage. Most market orders are treated as pure market orders. It’s treated as a command to the broker to execute the order without regard to the price paid. The maximum slippage pulls back the reigns a little bit.

Say that the market price is 50 and an MQL program sets the maximum slippage to 2. The MetaTrader broker knows that it may only execute the price within a range of 2 pips from the requested entry price. Either the price 50, 51, or 52 will do.

The difference between maximum spread and maximum slippage

The easiest way to distinguish the two items is to remember the following two questions.

Does it look like I’m about to pay too much to enter this trade? If so, I should use the maximum spread to prevent expensive trades.

Am I worried about the broker abusing my market order request after I send the order? If so, I should use the maximum slippage setting.

OneStepRemoved.com uses a hidden maximum slippage variable in our expert advisor programming template. We usually set it at 2 micro pips. You can ask us to make it an external variable upon request.

Filed Under: How does the forex market work?, Uncategorized Tagged With: slippage, spread

FREE trading strategies by email

Trending

Sorry. No data so far.

Archives

  • Dominari
  • How does the forex market work?
  • Indicators
  • MetaTrader Tips
  • MQL (for nerds)
  • NinjaTrader Tips
  • Pilum
  • QB Pro
  • Stop losing money
  • Test your concepts historically
  • Trading strategy ideas
  • Uncategorized
  • What's happening in the current markets?

Translation


Free Trading Strategies

Privacy PolicyRisk Disclosure

Copyright © 2023 OneStepRemoved.com, Inc. All Rights Reserved.