NinjaTrader

NinjaTrader Order Methods

Ninjascript, the C# API used for programming strategies in NinjaTrader, attempts to stirke a balance between making the strategy development process as simple as possible while preserving flexibility in the types of strategies that it can support.  It handles these competing requirements by splitting strategies into two types of order methods. Many programmers within the NinjaTrader community also refer to these types as approaches.

Managed approach

The first and most common order method is a managed approach. The managed approach is also the default assumption that NinjaTrader assumes you’re using when writing a strategy.

I backtest strategies this way 90%+ of the time. It’s way easier to write them using a managed approach once all of the variables and properties are set up. NinjaScript makes available a number of methods that all inter-relate. The names and functionality of each is obvious. EnterLong(), for example, enters a long trade.

All of these methods have premapped relationships between each other. When the strategy calls ExitLong(), NinjaTrader takes care of the details such as knowing which long it needs to exit. That seems obvious enough, but many trading applications do not connect the dots so easily between trading functionality. MetaTrader 4 & 5 are good examples of this type of simple functionality not being available.

The ease with which trading decisions are handled makes it ideal for developing quick and dirty code to test simple ideas.  Most of my strategy ideas follow the KISS principle; keep it simple, stupid. It’s usually a very simple development process (10-20 minutes) to program and test a trading idea from the moment it enters my mind. Most ideas don’t work out, but at least I get the satisfaction of knowing the answer quickly.

I discovered the hard way that the managed approach makes assumptions about order handling. EnterLongLimit(), for example, automatically deletes a pending limit order after one bar. I remember spending several hours with a live strategy trying to figure out why so many of my orders entered correctly on tick charts, only to have them disappear on the next tick. Carefully poring over the documentation led me to discover an overloaded method with the liveUntilCandelled parameter.

It’s examples like this pre-programmed functionality that you need to watch out for. This is especially true if you have a bug that doesn’t make sense in the context of your strategy.

Unmanaged approach

An unmanaged approach takes away all of the assumptions that NinjaTrader makes while a strategy runs in real time. This option leaves it up to the coder to store the position status in memory and make appropriate decisions.

Every strategy that we’ve written with unmanaged orders uses pending stop and limit orders. A recent example involves a market making strategy at Interactive Brokers. The strategy is 100% in the market during the US and Asian sessions. We originally started with a managed approach but ran into problems with the commissions and overfills.

The strategy is only trading 2 mini lots during the testing phase, which is the minimum trade size. Although the standard commission of 0.2 pips per side is quite low, the minimum commission is $2.50. That’s relatively high for such a small position size. I noticed on the first day that my client was getting charged commissions for the entry leg and exit leg, respectively.

The first trade entered with 20k. When it came time to exit, the managed approach set an exit limit for 20k and a separate entry limit for another 20k. IB charged commissions twice for both orders, which cost $5 per round turn trade. The trading report looked like:

BOT 20k $2.50

SLD 20k $2.50

SLD 20k $2.50

BOT 20k $2.5o

BOT 20k $2.50

SLD 20k $2.50

BOT 20k $2.50

SLD 20k $2.50

That’s effectively 4 trades at a cost of $20.

Grouping the trades into a single order more or less cut the commissions in half. I found it easier to do with an unmanaged approach because I didn’t have to worry about how the Entry() and Exit() methods would interact. The trade report changed to:

BOT 20k $2.50

SLD 40k $2.50

BOT 40k $2.50

SLD 40k $2.50

BOT 20k $2.50

The same sequence grouped together drops the cost of the 4 trades down to $12.50.  The initial entry and the final exit cost $2.50, but the trades in between only pay $2.50 for the entry and exit combined. When the order size increases to standard lots, the commission savings will dramatically improve the client return.

The other reason that we switched to an unmanaged approach was that NinjaTrader kills a strategy whenever an overfill occurs. An overfill is when a strategy requests cancellation of a pending order, but that pending order gets filled before the cancel request arrives. It then exits the new position at market and turns off the strategy.

I found this nearly intolerable with a market making strategy because an overfill would almost certainly occur at least once daily. Unmanaged orders have the option of disabling an overfill and handling it in whatever manner the trader deems appropriate.  In my case, it was to simply increase the amount of position that the strategy was trying to exit.

The main disadvantage to unmanaged orders is that everything is tracked internally. If you need to disable a strategy to change the settings or lose the broker connection, the code is probably not going to interact very well with positions that are already open.  Whatever IOrder objects were stored in the memory, which is how the strategy knows what positions are open, disappear whenever the strategy is removed from a chart.

Unmanaged strategies also take a lot more time to test. The amount of code involved does not differ dramatically from managed orders. It’s larger, but not grotesquely so. The reason that it takes more time is that the interaction between the order types is not polished like it is in managed orders. The programmer must write everything from scratch, which invariably means that the strategy will suffer from many more initial bugs, especially in a live environment.

Strategies that rely on market orders do not need to consider this approach in my opinion. It causes a lot more work and I have yet to encounter a situation where it makes sense. More importantly, I can’t think of a hypothetical situation where it would make sense either.

Toxic Trade Flow in Forex

I work with a number of expert advisor vendors and CTAs that want to sell their products. Whenever they approach a forex brokerage about partnerships, the inevitable first question the broker asks is, “Is the trade flow toxic?”.

Not all expert advisors are cut from the same cloth. Many trade infrequently. Strategies that trend trade or that trade long term ranges are broker favorites. The flow is predictable. The brokerage’s order book does not shift quickly.

Many large brokerages today still run dealing desks. When an order comes in to buy EUR/USD, the broker acts as counter-party to the trade. The customer is long and the brokerage is now short. Dealing desks handle this situation differently depending on their market perspective. If they do not have an opinion on market direction, then they will simply offset their trade by going long with their liquidity provider, which is usually a top-tier or group of top-tier banks. That relieves the broker of his exposure. He also keeps the mark up in the spread or commission for his efforts.

When the broker expects the market to move a certain direction, the dealing desk may elect to maintain the exposure. The trader’s loss is the broker’s profit and vice versa. The dealing desk wants to ensure that it is not overleveraged, but also to ensure that it maintains the desired level of exposure (say, for example, 100 standard lots long EURUSD). Traders that shift their positions rapidly force the dealing desk to constantly and quickly update its own position size.

Brokers with market positions hate scalping EAs. Trades that enter and exit quickly interfere with the broker’s goal of maintaining a position of a certain size. The amount of work required to maintain the exposure generally does not offset the amount of profit earned from the scalping trader. It’s business they don’t want. The industry term for unwanted business is toxic order flow or toxic trade flow.

Toxic orders and low liquidity

Another other kind of toxic expert advisor is one that takes advantage of price movements stemming from low liquidity. FAP Turbo is most infamous of this type. Whenever the price of EURGBP or EURCHF spiked quickly, the FAP Turbo expert advisor attempted to enter quickly on the movement. More often than not, the price quickly retraced to the take profit level of 3-5 pips.

Traders loved it. It was the closest thing to near instant profit. Brokers, on the other hand, despised FAP Turbo. Most brokers don’t care if individual traders make money off of them. When 10% of their customer base earns off of them and does so all at the same time, it’s a massive problem. The order flow was toxic to their businesses or to those of their liquidity providers.

The EURGBP typically offers market depth of 2-3 million euros, which is a pittance compared to the majors. Now consider that there were several thousand FAP Turbo users at the height of its popularity. Although most only traded mini lots, a substantial portion of them traded standard lots. Think about the combined order size of 2,000 traders while there is only 3 million available in liquidity. It just doesn’t work. There are too many dogs chasing a single piece of meat. The first ones to get their orders across got filled, but the back of the pack gets stuck either with scraps or nothing at all.

The traders that did win get on the brokers’ nerves for two reasons. It’s not that the traders won. It’s that they won and it only took them seconds to do so. Brokerages that offset their risk do not have sufficient time to do so. Brokerages that don’t offset their risk do not have sufficient time to enjoy the exposure that they are after.

The liquidity providers that filled the first wave of orders are not happy, either. This is probably the biggest problem of all. All forex brokerages rely on the dealing desks of major banks (i.e., liquidity providers) to some degree. There are only 10-15 major banks worth dealing with. It’s a small, almost incestuous community. If you get a bad name with one of them, you get a bad name with all of them.

The liquidity providers go back to the brokerages and dictate that they either shut out the toxic order flow or threaten to sever the liquidity relationship.  When the choice is between a critical supplier or a single customer, it’s always the customer that gets the boot.

Breakeven Trailing Stop

The trailing stop that we build in most of our custom expert advisors varies somewhat from the generic trailing stops out there. The code uses two inputs. An input is basically a variable that pops up on the screen when you load the expert advisor. You can change the input value without the need for additional programming.

Expert Advisor inputs tab

This is a screenshot of typical inputs for a MetaTrader expert advisor

The unique aspect of our trailing stop is that it does not trail until the trade reaches a certain amount of profit. Delaying the adjustment allows the user to treat the stop order as a take profit tool instead of simply exiting at a loss.  Most traders feel that once a runner appears, only then does it make sense to adjust the initial rules for exiting at a loss. Acting defensively about the trade only when a decent amount of profit is on the table avoids early stop outs, or at least so the theory goes.

TrailStart is the input which controls when the stop moves from losing to breakeven. It is only at this point of profit that the EA adjusts the exit conditions to avoid a loss.

Say, for example, that the TrailStart is set to activate at 20 pips. That means when your buy trade goes up 20 pips from the entry price, the EA automatically adjusts the stop loss to equal the entry price. The EA cannot lose from that point forward, not counting the effects of slippage.

TrailAmount kicks in only after the stop already adjusted to breakeven. This input controls the distance to increments at which the stop loss should update favorably. If TrailAmount equals 5, it means that the stop loss should adjust in your favor 5 pips for every 5 pips of extra profit beyond the TrailStart at 20 pips.

If the stop loss already moved to breakeven at 20 pips, it means that the stop loss is currently at 0 pips. The trade neither wins nor loses if the market hits the stop. If the price advances another 5 pips (TrailAmount ), the expert advisor determines that it must trail the stop again by 5 pips. The price reaching +25 causes the stop to advance from 0 to +5. When the price moves another 5 pips to +30, the stop advances to +10.

Notice how the stop distance remains a consistent 20 pips in the example. It’s the exact same amount as the TrailStart input.

Random Trailing Limit

I got the idea for a trailing limit from Van Tharpe’s market classic Trade Your Way to Financial Freedom.  I made Jon Rackley read through the book as part of his training when I first hired him. He brought my attention to an unusual claim made on page 267. Van Tharpe says that it’s often possible to make money even with random numbers.

Random numbers are a pet theory of mine. I’ve long put effort into figuring out whether I could develop a strategy that trades totally at random and still makes money. As the owner of a programming company for traders, and as part of Jon’s initial training for NinjaTrader in December, I assigned him the task of programming the strategy into code.

The book states that trading with totally random entries and a 3 ATR trailing stop generally leads to making money. Flip a coin. You go long if it lands on heads.  Go short if it lands on tails. One important note is that we elected to use pure random numbers in our programming instead of the pseudo-random numbers that computers generate. Doing so allows us to avoid time biases in the seeding process that generally pop up when the seeds used are close together in time.

The first working version that I reviewed displayed everything contrary to Van Tharpe’s claims. Using a trailing stop, regardless of the instrument and time frame tested, inevitably led to devastating losses. The profit factors consistently came in near 0.7, a truly awful number.

We did what most of our clients do when they find abysmal strategies. We flipped the strategy on its head. The new strategy uses only a 3 ATR (50 period) trailing limit.

Trailing Limit Analysis

The early conclusion is that trailing limits seem to offer a great deal of potential. Although Jon sent me the code several months ago, it’s only this evening that I had a chance to properly review and test everything.

The profit factors are very encouraging. It depends on the chart that I tested. The worst that I found was a 1.0 profit factor. Everything else came out with profit factors greater than 1. The small table below contains the initial test results. Before you go off salivating that this is the next hot winner, there are a few considerations to keep in mind:

  1. The results depend entirely on the sequence of random numbers used. Using different sets of random numbers will cause different outcomes.
  2. The better results on the higher time frames likely result from sampling error. The number of trades involved was only ~160, which is not enough to make definitive conclusions on the nature of the performance. I prefer to see 400 or more trades before drawing definitive conclusions.
  3. These results do not include spread costs or commissions.
Currency Period Profit Factor Dates Tested
EURUSD M1 1.12 9/12/2011-3/12/2012
GBPUSD M1 1.16 9/12/2011-3/12/2012
USDJPY M1 1.25 9/12/2011-3/12/2012
EURUSD M5 1.11 2011
GBPUSD M5 1.0 2011
USDJPY M5 1.06 2011
EURUSD H1 1.26 2011
GBPUSD H1 1.25 2011
USDJPY H1 1.48 2011

Random trades sometimes produce solid looking equity curves

Random trades sometimes produce solid looking equity curves. Remember that this was generated with purely random numbers and a trailing limit

What made me feel better about the results was reviewing the entry and exit efficiencies of each strategy. The number of trades involved really cluttered the graphic. I ran a backtest on a much shorter time period so that the horizontal, blue line would appear clearly on each graph.

The entry efficiency of a random entry is... random

The entry efficiency of a random entry is... random (45.45% entry efficiency)

The exit efficiencies of random entries with traililng limits are outstanding

The exit efficiencies of random entries with trailing limits is off the charts. (77.73% exit efficiency)

The entry efficiency tells us exactly what we would expect to find. Trades which enter at random do not perform better than random (obviously). What’s interesting is how the trailing limit exit strategy actually skews the entry efficiencies to read slightly worse than random. This is a good example of why it’s dangerous to rely purely on statistics. Keeping the big picture in mind reduces the likelihood of making an erroneous conclusion, in this case that there might be something “wrong” with our perfectly random entries.

The exit strategy, which is what we’re truly testing, looks absolutely stellar. It shows that acting as a conditional probability allows for a great deal of adverse movement while capturing extreme points of the move.

Adding Money Management

The percentage of winners for the tested charts ranges from 60-67% accuracy. High percentages of winners often lead to winning streaks. My cursory glance through the chart led to my easily finding a suitable example to cherry pick. Here, 5 trades in a row reach their near maximum take profits.

Random trades on a hot streak

Picking trades at random occasionally leads to streaks of lucky winners

Testing that I’ve done in the past tells me that it’s often advantageous to increase the position size based on consecutive winners when a strategy is more than 50% accurate.  My first idea after reviewing the initial results was to modify the forex money management strategy to pursue the consecutive winners. We unfortunately do not have time to pursue those changes in the code right now. Let me know if you’re interested in seeing this and we’ll make it a priority if enough readers respond.

While increasing the risk after consecutive winners works out statistically in your favor, it does add to the risk. It’s entirely possible for a “winning” set of trades to start losing based solely on the money management strategy.  There’s no way to know whether your set of trades will be the luck beneficiary of the extra risk or whether it will be the unlikely loser.

Conclusion

Everyone talks about stops. You always have to have a stop. Blah blah blah. I know it’s going to be the first question that everyone asks.

My research shows that trading with a stop is for suckers. Larry Connors’ book Short Term Trading Strategies That Work was the first trading book where I actually felt like I read valuable information. One of my favorite sections is on stop losses. His research shows that using a stop loss (even a 50% stop loss) always reduces the performance of a strategy. My own independent analysis bears this out.

Not using stops does not mean never taking losses. On the contrary, refusing to exit the market at a loss makes for a 100% odds of blowing up one day. The point of using a stop or limit is to empirically define, and to never waver from, a specific point or points in the market at which you will exit. A trailing limit accomplishes that goal admirably.

Interestingly, the trailing limit is one feature which FAP Turbo incorporates that I have yet to find in any other expert advisors. Although I’m not a fan of FAP Turbo type of strategies, it certainly does interest me that one of its main features aligns with this research. Also notable is the fact that the trailing limit continues down even to the point where it accepts losses.

Range Trade at High Frequency

Range trading systems make the best candidates for high frequency systems. They are less execution sensitive than trending systems for a simple reason. Range trades “catch the falling knife,” making them suitable for using limit orders.

High frequency prices vary from the normal M30 and H1 charts. The lower the time frame, the better that the chart fits to a normal bell curve. One common theme in systems trading since the 2008 crash has been “tail risk” or “fat tails”, which refer to the edges of a probability distribution like the bell curve. The fatter the tails, the more likely that a range trading system is to crash and burn.

The high frequency bell curve shows the tail risk of important events

The bell curve shows the tail risk of important events. The tails are colored in red. Fat tails mean that important news happens more frequently

The real world events captured in the tails reflect headline news like Bernanke speaking or Ireland announcing another referendum on all this bailout nonsense. The events only happen once, obviously. If you consider the news events in the context of hourly charts, they happen frequently as a percentage of the overall period. If you look at a one minute chart, that same event is now about 1/60th as important. Dropping down to tick charts nearly makes the events disappear in the statistical profile.

My experience is that the news cycle drives trends on a macro basis. “Macro basis” and high frequency are two topics that don’t belong together. Trending systems should focus on long term trading, while ranging systems are far more suited to high frequency. If your system trend trades, you can throw it in the rubbish bin for high frequency trading ideas.

High frequency considerations

Keep in mind that there are effectively two ways to participate in the forex market: you can either act as a price taker or as a price marker. Price takers range across all market participants. A hedge fund or university endowment is just as likely to take a price as they are to make one. CTAs and retail forex traders are much more likely to make their decisions based on the expected market direction. Timing is critical for them, so they don’t want to leave it to chance whether or not they’ll get to enter a trade.

The trader gets filled right away. That’s the major advantage. The main disadvantage to acting as a price taker is that you pay the spread every single time that you want to enter a position.

I sat with AvaFX in Dublin on my last trip. They charge a 3 pip fixed cost spread. I mentioned my concern about how that spread affects my client’s EA performance. His MetaTrader expert advisor trades 4 times per day on 2 currency pairs. If you do the math on a 3 pip spread, it works out to 8 * 260 = 2,080 trades per year. If you’re paying 3 pips and trading a $10,00 account, you would have to earn $6,240 per year – a 62.4% return, just to cover trading costs. I don’t care how good a system is – it will never cover those kinds of costs. Trading on margin will not do anything to resolve the issue. Spread costs are directly proportional to the amount traded, which impacts the profit. There is no way to trade and make money if the transaction costs are too high.

Designing an expert advisor is difficult enough, but it’s even harder when you factor in the trading costs. Say, for example, that I develop a EA that wins 75% of the time with a payout of 0.5:1 before trading costs. When the EA wins, it earns $0.5. It loses $1 whenever a loss occurs. The profit is 75 wins * $0.5 = $37.5. The loss is 25 * $1 = $25. The expert advisor’s profit factor is 37.5/25 = 1.5.

That should sound great. The problem occurs when the total commission outweighs the total expected profit. This example required 100 trades. Let’s say that we were trading mini lots with an average win of 5 pips and the average loss of 10 pips. That puts the gross profit at $375 and the gross loss at $250. The return is $125 for the 100 trades, excpet that we must now subtract the $100 for trading costs. The total profit plummets to a measly $25.

If the expert advisor’s expectations held true for something like a 10 pip take profit and 20 pip stop loss, the trader might be better off to change the exit points. The reason is that the profitability may actually improve. The goal would be to reduce the number of trading opportunities with an eye towards making them more profitable relative to the costs.

A better approach, in my opinion, would be to switch over to market making. Although you usually still pay to trade, the advantage to market making is that you earn the spread rather than paying it. The spread is overwhelmingly most traders biggest cost. Not paying it opens the possibility of applying the strategy where one normally could not afford it.

Market making only works if your forex broker allows you to post best bid/best offer and have the price reflected on the screen. Most brokers claim that they are ECNs. A real forex ECN allows you to post limit orders. Whenever that order represents the best bid or offer, the price and size of your order shows up on the screen. The only retail trader friendly brokers that I know of are Interactive Brokers and MB Trading.

I ran my NinjaTrader license at MB Trading last week to test the execution and order flow. The test only use traded a microlot (0.01) and posted best bid or best offer on the EURUSD. The orders remained valid for anywhere from 1-10 minutes. Despite the small trade size and lengthy time period as best bid/offer, the orders only filled 75% of the time. That meant that I caught 100% of the losers but only 56% of the potential winners. Not good, in spite of getting paid for the limit orders.

Interactive Brokers is the next test candidate. They have been around much longer and should have far more order flow. I’m hoping that the low fill rate that I experienced making a market at MB Trading will improve substantially when I shift the same strategy to Interactive Brokers.

I expect to find a few other changes as well. The spread that I earn should fall from around 0.9 pips on EURUSD to 0.5 pips, which is indicative of Interactive Brokers’ improved pricing. I also will have to pay a 0.2 pip commission, which reduces the net credit from 1.0 pips at MB Trading (0.9 spread + 0.1 commission) to 0.3. Nonetheless, I expect the improved fill rate on winning trades to work more in my favor.

The thing that most people will hate is that you can only test a market making approach with live money. It’s sufficient to backtest a strategy using market orders with a 0 spread assumption. The goal is to weed out the junk from diamonds in the rough. No method exists, however, to accurately determine whether or not a trade would have gotten filled with a limit order. The only way to find out is to test an idea with live money, then to compare the results to a backtest over the same period. If the live, high frequency performance is similar to a backtest, then you probably have a winning approach.

The real motivation here is to get as many opportunities as possible. Just like the casino does everything to help you pull the slot machine faster, the trader should look for as many favorable setups as possible. High frequency stands out in this area. The inherent advantages of a system are more likely to manifest more quickly. Assuming that you get a handle on the trading cost problem, the profit is often limited only by the number of trades that can be squeezed into a day.

Programming options at high frequency

MetaTrader 4 is not a good candidate unless you expect to post orders once per minute or slower. MetaTrader suffers from the Trade Context is Busy error. Running an expert advisor on more than a single instrument could cause orders to enter too slowly or not at all. MetaTrader is only an option with MB Trading. Interactive Brokers does not support MetaTrader.

NinjaTrader works great and offers a lot of the broker portability that comes with programming in MQL. Programming a high frequency strategy in NinjaTrader works at most human speeds (5 seconds or more). For the brokerages where NinjaTrader submits orders using the broker’s API, I find a speed bump affect at work. NinjaTrader processes the orders lightning fast, but the broker API cannot handle the speed and starts to choke. If you want to test any frequency that’s not ultra high frequency, I recommend programming in NinjaTrader.

The FIX Protocol is the best option for the institutional trader that cares about maximal performance and does not suffer from the usual budget constraints. FIX is a fancy way of controlling communications between a custom platform and the broker. It does not involve software, only rules. The FIX protocol allows the trader to write software 100% from scratch. The trades and orders can go out the door literally as fast the machine can process them. It’s the advantage that comes with building everything from scratch.

Market Depth

Moving from small time retail forex accounts to a serious account size comes with some bumps in the road. Most traders see the prices of forex pairs on the screen and assume that they can buy and sell unlimited quantities at any time. Although the forex market is the largest market of any in the world, making it the most liquid, there is still a limited size to what you can trade at any moment.

Reading Forex Market Depth

NinjaTrader and MB Trading both make market depth information available in their trading screens. It shows where all of the nearby liquidity lies. Say, for example, that you wanted to place a big order for EUR/USD. You can see in the screenshot from MB Trading’s platform that the liquidity gets bigger as you get further from the price. They call these “Level 2″ quotes, which is jargon taken from equities traders.

Market depth of EUR/USD

MB Trading's Navigator shows the market depth of the EUR/USD. Light colors show the best prices, darker prices indicate distance from the best price.

I took this screenshot in the late afternoon when liquidity is at its worst. The best bid shows a depth of 200, which is measured in mini lots of 10,000. You could sell up to 2 million EUR/USD and get filled at that price. Notice, however, that most of the liquidity is further away. 3 million sits at the next best price and another 3 million even further from that. The net available liquidity is 8 million on the short side and 6 million on the long side for a total of 14 million.

The FX Pro screen in NinjaTrader makes it even clearer. I took this screenshot several minutes later, which is why liquidity numbers are different. One thing I really like about this screen is that NinjaTrader converts the liquidity depth into more tangible numbers. The formatting makes more sense to me. Plus, it’s much easier to keep track of the varying spread.

FX Pro screen in NinjaTrader

The FX Pro screen in NinjaTrader

What you learned in economics doesn’t apply to trading. Dealing in bulk actually leads to worse pricing instead of improved pricing. The reason is that forex instruments are so standardized that there’s effectively never a lack of customers. It’s really an issue of getting how much much you want at a certain speed.

Only a fool would hit the market with 20M EURUSD instantaneously unless you’re desperate to exit a position. If you push a market order judging solely from the quote on the screen, you may get 2-8 million filled at the displayed price. But, the rest of the order will get filled at progressively worse prices. The traders making a market want their pound of flesh for letting someone into the market so quickly.

Traders cannot see the liquidity depth of most brokers because they elect not to show it. Their platforms encourage the buy this, buy that psychology. The more information that they broadcast, the more bandwidth that’s required, which means that better servers are required.

The EURUSD is the most liquid forex pair in the world. This varies largely by broker, but at any given time you should be able to trade 20-30 million EUR/USD. That doesn’t mean that you’ll get filled at the price on the screen. What that means is that that is the sum quantity available at any given moment.

Some brokers hide their quantity. It’s not like the stock market where if there are 15,000 shares of Microsoft ready to trade at any given moment, you can see 100% of the liquidity available. In forex, as an OTC market, the broker may wish to restrict the viewable book for a few reasons.

If you offer markets from say 5 banks, it’s very rare for the broker to feed the best competitive price and to let the banks fight it out? Why? Because the broker also needs the banks to stick around when nobody wants to trade.

It’s an informal agreement that if I’m retail broker A and UBS is my main liquidity feed, I go out of my way to give UBS the good flow. My customers expect to trade during NFP and other volatile markets (although they really shouldn’t!). If the brokerage simply lets the banks fight it out, then the banks have every reason to let the brokerage rot on the vine when their customers want to trade but it’s bad for the banks. The banks certainly don’t want to take positions in volatile markets. Their only incentive for doing so is if the “good flow” that the forex broker sends during normal markets incentivizes them to accept the risk of losing more than they care to during NFP.

News traders are the most likely to try trading during a thin market. They are also the most likely to complain about not being able to trade. These are the arguments to which I’m least sympathetic. If you’re trading the news, you are overwhelmingly likely to have less than one year of trading experience. The decision has nothing to do with researching systems or evaluating whether or not it’s a good idea. It has everything to do with gambling.

Retail traders are the most likely to trade during volatile events, not just news but really any type of momentum. Almost everyone follows a breakout or momentum strategy. It has everything to do with what traders perceive as the most likely outcome. When the market explodes in one direction, it takes nerves of steel to stand in front of the freight train. Therefore, it’s probably a good idea because it’s counter-intuitive.

Trends happen so slowly that they don’t excite the gambling buzz that most retail forex traders are after. My friend Afshin in Dublin fell victim to this last week. He saw the EUR/USD rising day after day after day. He felt like it was simply overdue for a correction. The urge to participate, rather than coming from a desire for a quick hit, instead came from a desire to be right before there was any clear indication of the opportunity to be right. The point is that what feels natural to do is often precisely the wrong thing to do. It feels natural to every other trader, too.

Market Depth and Direction

One research project that I’d eventually love to do is to study how market depth on any given side of a market affects direction. Some traders run simple liquidity businesses where they receive trading rebates in exchange for accepting the risk of holding a position over the short run. These entities are less likely to concern themselves with picking the direction of the market.

Trading desks that make markets, however, often want the flow so that they can establish a position and earn the spread while doing so. These entities are picking direction – and they are backed by very intelligent math geeks with PhDs and a lot of time on their hands. If those desks make a visibly deep market and it’s sufficiently one sided, then it’s probably safe to assume that they expect to the market to move in the opposite direction.

When you’re buying a forex pair, the bank is selling it to you. So if everyone stacks the liquidity so that you can buy but the liquidity is thin on the short side, it should be telling you that the smart money wants to go short right now.

Trading Time in Programming

The major automated trading platforms such as MetaTrader 4, NinjaTrader and TradeStation all count time in the same way. This makes it quite convenient for ordering trading strategies and expert advisors; you don’t have to do any mental gymnastics to describe the strategy in different platforms. The consistent arrangement of time makes it easy for us to translate trading strategies across multiple platforms.

We tend to think of time as moving in the same direction as when we read. English speakers, who read from left to right, think of time as moving the same direction. If you speak a language like Arabic that reads right to left, you tend to think of time as marching to the left.

All of these charting platforms are written by speakers of left to right languages. The past is anything that’s not on the far right side. The present is the square on the far right. Each square represents an equal time interval. Traders know these as bars.

Time as an Array

A visual display of time and how it's segmented

What tends to confuse everyone ordering expert advisors is that even though time marches to the right, trading programmers count the bars to the left. What makes things more confusing is that programmers always start counting from 0 instead of 1.

How to count time in an array

Even though time moves to the right, we count it from the right and move back left

If you want to trade a moving average cross strategy that waits for the bars to close, what you’re looking at is “bar 2″ and “bar 1″. The way I describe this in the scope of work is the value at two closed bars ago and the value at the last closed bar, respectively.

An expert advisor that uses closed bars ignores bar 0 because it is still open, which causes the moving average values to fluctuate. The only way to know for certain a moving average value at a particular bar is to wait for the bar to close, which means it is no longer bar 0. When a client requests an expert advisor that trades intrabar, they intend to compare the moving averages at “bar 1″ with “bar 0″. In plain language, that means to compare the value at the last closed bar with the currently open bar.

Hopefully, this description makes sense when you open a chart and see the bars already loaded. The final confusing element is when a new bar pops onto the screen. The previous examples showed 5 bars on the chart (bars 0-4). When a sixth bar pops up, the count is reset to the new time period on every update.

Say that we’re looking at an H1 chart in MetaTrader and that the current time is 06:00. When the new hour strikes, the chart loads a new candle to represent 07:00. It’s at this time that the count resets.

Time udpates

When a new bar appears, your charting platform resets the count based on the newest bar.

High Frequency NinjaTrader Strategy

I’ve been working on a high frequency trading system for NinjaTrader on behalf of a long term client. My live account is with MB Trading. Rather than placing market orders and paying a commission, I changed the order types to limit orders. We want to receive a small commission for the market making strategy rather than paying a commission to accept the displayed prices.

MetaTrader suffers two major disadvantages that make NinjaTrader a superior option for high-frequency trading. MT4 does not offer charts lower than the M1 time frame and the trade context is busy error prevents multiple charts from running simultaneously. NinjaTrader is complex enough to where I can control most details, but simple enough that I don’t need to invest hundreds of hours to test an idea. After extensively testing the strategy on M1 charts as a price taker, I feel very confident that the strategy is sound. The only issue now is determining whether or not whether taking a passive (i.e., market making) approach will result in enough fills to make the strategy worthwhile.

The first issue that I came across wasn’t with NinjaTrader; it was with MB Trading’s API. The strategy worked fine on the simulation account, which only routes orders to NinjaTrader (NT). NT then makes guesses when fills would occur. The goal of that phase was not to test the strategy. I only wanted to test the programming to make sure that it worked properly.

100 trades went off without a hitch in the Sim account. The strategy only made it through 2-3 microlot trades on the live account before the pending orders hung. NinjaTrader pending orders pass through 3 states before they actually hit the market. For the programmers out there, these are the OrderState properties of IOrder objects.

  1. Pending submit – the strategy sent the order to the broker and is waiting to hear back
  2. Accepted – the broker acknowledges receipt of the order, but is still placing the order into the market
  3. Working – the order is available for others to trade

The strategy updated orders on every tick. What often happened was that the pace would go far too quickly, creating a major communications backlog during fast markets. NinjaTrader never threw an exception. The only evidence of a problem was that I would see a hanging order with the PendingChange property. The inconvenient solution was to exit NinjaTrader and reload everything.

I figured that perhaps that the managed order state caused the issue. I changed my approach to unmanaged orders, but that did not make a difference. I eventually came to the realization that the MB Trading API cannot handle more than one order every few seconds.

The strategy found the sweet spot after changing from tick to second charts. Updates of 6 seconds or longer seem to give the MB Trading API enough time to update wihle still preserving something of a high frequency approach. Any trades that need to run faster than that threshold at MB Trading need to use the FIX protocol.

The other element that drove me crazy is that NinjaTrader limit orders automatically delete themselves once per bar. I nearly tore my hair out, and I don’t have all that much hair, for several hours trying to figure out why orders deleted themselves automatically. Many people identify with the school of hard knocks approach to learning. I’m as thick headed as most. I figured out the cause when I revisitied NinjaTrader’s online documentation and discovered a limit entry method that allows good till cancelled (GTC) orders.

The speed problem also manifested with overfills. An overfill is when a strategy requests to cancel a pending order, but the broker fills the order before the cancellation takes effect. The biggest concern with overfills is that NinjaTrader automatically disables a strategy and exits positions at market when an overfill occurs. The only way to programmatically prevent this is to change the entry methods to an unmanaged approach.

The easiest way to develop for a high frequency strategy in NinjaTrader (but not ultra-high frequency) is to use managed orders. Whenever an exit is needed, place the limit entry in the opposite direction. NinjaTrader takes care of placing the exit order for the open market positoin. Limit the updates to every handful of seconds. It allows the broker API to catch up and helps avoid the problem of overfills.

Reverse engineer expert advisor

I guarantee that you are not the first trader to consider the idea of reverse engineering an expert advisor. The idea pops up most frequently among traders that don’t want to pay for a commercial expert advisor. Alternatively, they want to use the same strategy when another trader that doesn’t want to share the idea. The motivations for reverse engineering remind me of decompiling EAs, making me leery of the idea.

Reverse engineering a strategy only stands a chance when some parameters are known about the strategy. If, for example, you knew that the strategy involves MACD and moving average crossovers, it at least provides a reasonable starting point. The programmer could write software which combines every possible combination of two moving averages of various types with every known type of MACD. The programmer could then make guess about which types of signals might result in a buy or sell decision. If the guesses are not very good, then the outcome of the reverse engineering attempt is certain failure.

Then you have the problem of guessing on which chart to base the decisions. Many traders use standard charts like the M1 and M15, but many others use less common options like an M3. If the trader uses multiple charts like an M2 and M10, the resulting trade history would clutter together. Good luck trying to pry apart the different series.

Making things worse is if the trader uses a chart that doesn’t depend on time at all. Tick charts are the most common, although you occasionally see less orthodox options like Point and Figure charts and Kagi charts. Time is irrelevant. You wouldn’t have any idea on which charts to run the test.

This approach of making somewhat intelligent guesses while throwing mud at the wall is called a brute force attack. You literally designate every unique possible combination, then see which one opens the metaphorical safe. Some results will bear no resemblance at all to the actual results. If you get extraordinarily lucky and/or have fantastic intelligence, then you might find a close replica of the strategy.

It would be possible to study the correlation of the tested values with the values in the supplied account statement. You would ideally want to find data clusters with similar variable settings that do not dramatically alter the correlation between the reverse engineered strategy and the actual account statement.

If you don’t know very much about the strategy, or if what you think you know turns out to be wrong, then you stand no chance at all of reverse engineering the expert advisor. For all that you know, the EA that you thought used RSI might turn out to run on phases of the moon (yes, there are real strategies that do that) or that make decisions based on coin flips.

Most people assume that they know more than they really do. I would discourage all but the most fool hearty or stubborn from making the attempt, unless you had a very good reason for doing so.

Develop Trading Strategy

One common element unites 99% of the trading strategies that we’ve developed. The final product does not match the initial design.

All programming projects proceed in one of two ways. You can either develop a project from the top-down, or you can build from the bottom up. The former option represents the old way of thinking, especially within the university system. When professors assign a project, the requirements never change. The assignment remains static.

Real life rarely cooperates like that. Instead, human beings change their minds. They see the work in progress and decide that a minor tweak might dramatically alter the outcome. The expert advisor programmer is forced to constantly adjust to the moving goal posts.

I always tell clients that they will change their minds about some of the rules during the process. Most insist that they will not. They do, after all, expect them to work.

Then, the rules inevitably change for most of them. Theory goes out the window. The traders discover the real issues that expert advisors face in the market.

Programmers developed a new methodology to accommodate the constantly shifting rules. It’s called agile development. This problem doesn’t only occur with forex traders. It also happens with businesses making web sites, database programmers storing information, and really any project that involves human beings. People are prone to changing their minds. It’s our nature. Agile development is the programmer’s acknowledgement that getting upset at people for changing their minds is not entirely reasonable.

The agile approach breaks a project into time periods with a ranked list of objectives. The team sorts out which objectives are most important and do-able within the allotted time. The period starts and everyone rushes to accomplish as much programming as possible. Everyone regroups after 2-6 weeks to evaluate the progress and to deliver the updated version to the client.

Most expert advisor projects only last a few weeks, so a precise copy of the agile approach would not fit very well with developing a trading strategy. We flip the strategy development approach on its head by assigning ranked priorities with a deadline of 1-2 business days. The goal is to facilitate communication, which is inevitably the most challenging obstacle in this field. We also make the process less formal to speed things along.

The quick turnarounds lead to a series of yes/no answers. The client either gets excited that it works properly, or more likely, feels that we did not understand him correctly. The honest goal of this approach is to get the user away from explaining the idea for the 1,000th time in the same, identical way. The last explanation did not help the project move along. Client feedback from working software leads the programmer into understanding the concept on his own without making the client feel like he’s repeating himself.

I like to think of it as the client steering the driver’s wheel while the programmer controls the gas pedal. The strategy development process only works when the steering and power move in synch.

Overcoming expert advisor programming challenges

When clients feel like an expert advisor has not made the desired progress, the first step is to assign goals one at a time. Too often, we receive reports with 20 different bugs. The emphasis on priority falls away as the desire to “fix everything and fix it now” drowns out the details of each request. I always like to take a step back and reaffirm our commitment to developing the strategy, but we can only handle so many issues at a time.

I have a very patient client in New Jersey (I never though I’d say “patient” and “New Jersey” in the same sentence”) who ordered a lot of custom elements in a basic expert advisor. Most of the complications arose from the fact that I had to remotely access his computer with LogMeIn to do the programming. His indicator only has one license. Purchasing a second for the EA programming was not economical.

The requirements to remotely access the computer and the amount of custom code turned a simple project into a complicated one. The way I addressed the issue was to remove almost all of the custom components and to replace them with code from our usual template. Now that the client sees that the “strategy” stuff works, he found it much easier to write a checklist of bugs to fix. More importantly, seeing critical components of the code working reinforced his confidence in the company’s programming ability.

Although he remained patient throughout the debugging process, I could hear the flagging confidence in his voice before we switched gears. Changing the direction of the project showed him that the EA actually does work; it’s the little pieces and how they fit together that are the problem. Of equal importance was the fact that he felt confident in my ability to deliver the results that he wants.

The client gets full credit for releasing control of the debugging process, even when he wasn’t entirely comfortable doing so. The idea of papering over the custom steps with pre-programmed template code struck him as a step backwards. He nonetheless followed the programmer’s lead, letting him drive the process with himself providing feedback where it was needed.

Both parties agree that we’re once again heading in the right direction. The client sees obvious progress in his expert advisor. The programmer is able to manage the debugging process in a methodical, organized manner. Everyone is happy.

Realistic strategy development

The most realistic way to develop a strategy is to acknowledge that you don’t exactly know where you’re going. You know it’s likely to involve a certain indicator and that it either trends or ranges. The best way to prepare for the journey is to acknowledge in advance that it is indeed a journey. You more than likely will make changes, most of which you cannot expect.

It’s 100% certain that something is going to go wrong when you program your first project. It’s your strategy, but it’s really a two-man team that builds it. Make sure to pick someone that knows what do when problems come up.

Before you choose your programming partner, make sure it’s someone that you actually trust to get the job done. You ideally want to find someone that offers strong communication skills and appreciates the importance of a timely reply. Organizing the project and overcoming unexpected challenges helps everyone get back to their true passion of trading.

Sign up for EA Package


* = required field

powered by MailChimp!