Algorithmic and Mechanical Forex Strategies | OneStepRemoved

Order Retry Logic

by 5 Comments

The difference between a backtest and live trading is that nothing ever goes wrong on a backtest. If a strategy traded correctly for EURUSD in 2011 yesterday, you know that the same test will work properly today.

The backtester is not designed nor is it able to catch the types of problems that occur in live trading. I made an effort to list common issues by platform and to detail the most common solutions.

 

MetaTrader

Most older EAs attached stops or take profits to its orders. The NFA rules from around 2009 require all forex trades to enter without any exit conditions attached.

The rule created a nightmare for US MT4 brokers. They were forced to go back, modify MetaTrader and disallow tickets with a stop or limit.

The solution is to confirm correct execution of a trade. Once the trade enters, only then should the expert advisor attempt to add the stop or take profit.

The rule is unfortunate as it requires additional communication time. The process slows order execution, which can cause the trade context is busy error.

order denied

These are not words that you want to see when placing live trades

Our Expert Advisor Programming Template

Kamal O. asked on Friday about the words RETRYCOUNT and RETRYDELAY in our EA template code. Those 2 words are critical for maintaining code that handles all possible situations, or at least 99.9% of them.

We set them by default to 10 attempts and 1,000 milliseconds, respectively. That is, an order will attempt to order up to 10 separate times. Each attempt will wait at least 1,000 milliseconds (1 second) before making another attempt.

In between attempts, we also check to see if the trade context is busy. If it is not, then we proceed. Otherwise, the code waits for an opportunity to submit the new order up to the specified maximum.

The same retry ad wait logic also applies to submitting stop losses and take profits. One of the most horrifying real world events that a trader can discover is an open trade with no exit conditions attached. Such things really do happen.

The same retry logic is in place in our template code to dramatically reduce the chance of that occurring. We do this for every expert advisor that we program. If you have an expert advisor that does not include retry logic, then contact us about making your EA’s source code more robust.

NinjaTrader

NinjaTrader largely handles errors and problems for the programmer. However, there are common situations where NinjaTrader’s solution frustrates the user. Disabling strategies and closing all trades whenever an overfill occurs comes to mind.

Anything but the simplest trading strategies are better handled using an unmanaged approach. Managed approaches using pending orders will almost always create a need for rewriting a strategy.

One of the most common reasons NinjaTrader users contract us relates to something going wrong with their live trading. The best way to avoid programming something twice is by making the code ready for real world trading on the first attempt.

High Frequency NinjaTrader Strategy

by 7 Comments

I’ve been working on a high frequency trading system for NinjaTrader on behalf of a long term client. My live account is with MB Trading. Rather than placing market orders and paying a commission, I changed the order types to limit orders. We want to receive a small commission for the market making strategy rather than paying a commission to accept the displayed prices.

MetaTrader suffers two major disadvantages that make NinjaTrader a superior option for high-frequency trading. MT4 does not offer charts lower than the M1 time frame and the trade context is busy error prevents multiple charts from running simultaneously. NinjaTrader is complex enough to where I can control most details, but simple enough that I don’t need to invest hundreds of hours to test an idea. After extensively testing the strategy on M1 charts as a price taker, I feel very confident that the strategy is sound. The only issue now is determining whether or not whether taking a passive (i.e., market making) approach will result in enough fills to make the strategy worthwhile.

The first issue that I came across wasn’t with NinjaTrader; it was with MB Trading’s API. The strategy worked fine on the simulation account, which only routes orders to NinjaTrader (NT). NT then makes guesses when fills would occur. The goal of that phase was not to test the strategy. I only wanted to test the programming to make sure that it worked properly.

100 trades went off without a hitch in the Sim account. The strategy only made it through 2-3 microlot trades on the live account before the pending orders hung. NinjaTrader pending orders pass through 3 states before they actually hit the market. For the programmers out there, these are the OrderState properties of IOrder objects.

  1. Pending submit – the strategy sent the order to the broker and is waiting to hear back
  2. Accepted – the broker acknowledges receipt of the order, but is still placing the order into the market
  3. Working – the order is available for others to trade

The strategy updated orders on every tick. What often happened was that the pace would go far too quickly, creating a major communications backlog during fast markets. NinjaTrader never threw an exception. The only evidence of a problem was that I would see a hanging order with the PendingChange property. The inconvenient solution was to exit NinjaTrader and reload everything.

I figured that perhaps that the managed order state caused the issue. I changed my approach to unmanaged orders, but that did not make a difference. I eventually came to the realization that the MB Trading API cannot handle more than one order every few seconds.

The strategy found the sweet spot after changing from tick to second charts. Updates of 6 seconds or longer seem to give the MB Trading API enough time to update wihle still preserving something of a high frequency approach. Any trades that need to run faster than that threshold at MB Trading need to use the FIX protocol.

The other element that drove me crazy is that NinjaTrader limit orders automatically delete themselves once per bar. I nearly tore my hair out, and I don’t have all that much hair, for several hours trying to figure out why orders deleted themselves automatically. Many people identify with the school of hard knocks approach to learning. I’m as thick headed as most. I figured out the cause when I revisitied NinjaTrader’s online documentation and discovered a limit entry method that allows good till cancelled (GTC) orders.

The speed problem also manifested with overfills. An overfill is when a strategy requests to cancel a pending order, but the broker fills the order before the cancellation takes effect. The biggest concern with overfills is that NinjaTrader automatically disables a strategy and exits positions at market when an overfill occurs. The only way to programmatically prevent this is to change the entry methods to an unmanaged approach.

The easiest way to develop for a high frequency strategy in NinjaTrader (but not ultra-high frequency) is to use managed orders. Whenever an exit is needed, place the limit entry in the opposite direction. NinjaTrader takes care of placing the exit order for the open market positoin. Limit the updates to every handful of seconds. It allows the broker API to catch up and helps avoid the problem of overfills.

How to handle the Trade context is busy error

by 5 Comments

The trade context is busy error causes nightmares for scalpers and high frequency traders. MetaTrader chokes when they open up too many charts or place too many order. The problems stem from how MT4 processes and receives information from the broker server.

The OrderSend() command, which MQL programmers use to open a trade, also tells the MT4 client terminal to await a server response. Everything locks up. The code cannot do anything until the server permits the client terminal to continue. More importantly, the trading terminal does not allow any additional trades to open.

Example

Consider a scenario where you trade EURUSD, AUDUSD and USDJPY currency pairs on H1 charts. MT4 must await an incoming tick before MQL allows it to make a trading decision. I discussed this issue several months ago with time based orders.

The ticks of multiple currencies often group together during liquid hours. If the terminal receives the incoming ticks within a few hundred milliseconds of each other, MetaTrader creates its own bottleneck.

The AUDUSD tick comes in first. The Expert Advisor sees a trading opportunity and decides to open an AUDUSD trade with OrderSend(). 150 ms later a EURUSD tick comes in and creates another trading opportunity. The EURUSD EA, which runs on a completely different chart, is not allowed to trade.

Retry delay for Trade Context is Busy

Our Expert Advisor programmers check for “Trade Context is Busy” errors before sending orders to the broker. When the situation exits, the EA checks the trade context every 50 ms up to 50 times.

It is not uncommon for MetaTrader orders to take a second or more to execute. The original AUDUSD trade from our example is still hanging out there. The EURUSD EA enters a 50 ms cycle where it constantly asks, “Did the trade context open up?”

Then the USDJPY tick comes in 70 ms after the EURUSD tick. USDJPY encounters the same trade context issue, kicking it into a separate, “Is the trade context busy” loop.

When the AUDUSD trade finally executes, it leaves the EURUSD and USDJPY expert advisors in a frantic race to discover the newly opened trading window. The first EA that discovers that the trading context busy error disappeared will send its request first. The second EA, unfortunately, cannot communicate with the first. Although the second EA will discover that the trade context opened up, it will receive a trade conext is busy error when it attempts to trade; the firs EA already sent a trading request. The second expert advisor allots itself a maximum of 10 failures before it gives up on the trade.

Assuming that the EURUSD discovers the new trading window first, then the USDJPY receives its first failed order attempt. The USDJPY once again inquires about the trading context every 50 ms. The EURUSD trade eventually executes, allowing the USDJPY to finally enter the market.

Most of our customers never notice this delay, which means that we did our job. There is, however, a breaking point where MetaTrader simply cannot issue enough orders at the speed the trader desires. Any combination of charts where 10 or more orders go off near-simultaneously will result in missed trading opportunities.

MetaQuotes designed MT4 with the trade context error in order to limit the processing requirements of the brokers. As I’ve stated repeatedly, the brokers get MetaQuotes’ preferential treatment because they are the ones paying MetaQuotes’ fee. MetaTrader is not a high frequency trading platform. It’s not in the broker’s interest to support high speed trading. It usually opens them up to too many traders taking advantage of arbitrage opportunities.