Algorithmic and Mechanical Forex Strategies | OneStepRemoved

7 Things You Need To Become A Successful Forex Trader

by 1 Comment

If you make the decision to start trading forex to earn extra income it is vital that you set yourself up to succeed. To do so you need to be aware of the 7 things you need to become a successful forex trader.

The desire to succeed

Firstly, it is elementary that you commit fully to the process of becoming a forex trader. That means that you are willing to put the time in to learn all important aspects of currency trading and the global currency market. You will only have the motivation to put the time in to learn, if you truly have the desire to succeed and make money trading currencies.

A genuine interest in forex, economics and the financial markets

Secondly, and this ties in with the first point, if you want to succeed as a currency trader it is vital that you have a genuine interest in the financial markets. If you are doing it just to make a quick buck, without actually putting in the regular work of reading financial news, analyzing charts and reading daily currency market updates, then you will most likely not succeed. Furthermore, you will need to learn about macroeconomics, as economic data and central bank policies are key drivers of the foreign exchange market. Hence, having a genuine interest in what moves the financial markets is a key component to becoming a successful currency trader.

The economist

The right online broker

There is a vast choice of online brokers that charge different spreads and commissions and have different product ranges. Hence, it is important to choose an online broker that is right for you. To do so, you need to choose a broker that covers the asset classes and currency pairs you wish to trade, charges you comparatively low fees, offers tight spreads, has a good reputation and is regulated by your country’s financial regulatory body.

It is also important that the online broker you choose offers easy-to-use chart analysis tools, timely market news updates and possesses good customer service. The best way to choose a broker is to check independent broker reviews and comparisons online.

Trading capital (but less than you might think)

To start trading forex you need a certain amount of capital. However, it is must less than you might think if you choose to trade with leverage. Leverage in the foreign exchange market refers to the ability to move, for example, USD 100 dollars worth of a currency using only USD 1. This would be leverage of 100:1, which is a popular leverage amount in the currency market. Other common leverage amounts are 50:1 and 20:1. Using leverage you can move large amounts of a security by only putting down a small initial amount per trade. This small amount is referred to as the initial margin.

The best way to use leverage is by trading so-called CFDs (contracts for difference) as they allow you to set your leverage, as you require it. By adopting a CFDs trading strategy you are able to profit off small moves in the currency pairs you are trading without putting down a large amount of capital on each trade. Hence, this is the best way to trade currencies if you only have a small amount of available trading capital.

trade cfds

The right trading strategy

Once you feel comfortable with the currency market’s terminology and mechanics and you have deposited your trading capital into your online brokerage account, it’s time for you to apply the right trading strategy.

When it comes to trading currencies there are many approaches you can take. For example, you can apply more of a momentum trading strategy and put on trades just after market moving news, such as economic data announcements, or you can use a technical indicators-based trading strategy and follow a set of indicators that give you buy and sell signals for the currency pairs you follow. Of course, there are many more approaches you can take. It is important for you to find a strategy that suits your style of trading and is in line with your risk-return profile.

The discipline to stick to your strategy

Once you have found a trading strategy that works for you, it is important that you have the discipline to stick to your trading strategy. A great way to ensure you don’t let emotions get in the way of you following your strategy is to set target prices and stop-losses, where your broker automatically buys or sells the currency you hold against another, once these trading levels have been hit.

online trading

The emotional stability to handle losses

Finally, if you truly want to succeed as a forex trader you need to develop the emotional stability to handle losses. No matter how good your trading strategy is you will have days where you will generate losses. It is important to accept down days and not let your losses affect you emotionally, as this could impair you when you put on further trades.

How To Win With Mechanical Trading Systems

by 13 Comments

Much ink has been devoted to pinpointing the causes of mechanical trading systems failures, especially after the fact. Although it may seem oxymoronic (or, to some traders, simply moronic), the main reason why these trading systems fail is because they rely too much on the hands-free, fire-and-forget nature of mechanical trading. Algorithms themselves lack the objective human oversight and intervention necessary to help systems evolve in step with changing market conditions.

Mechanical trading systems failure, or trader failure?

Instead of bemoaning a trading-system failure, it’s more constructive to consider the ways in which traders can have the best of both worlds:  That is, traders can enjoy the benefits of algorithm-managed mechanical trading systems, such as rapid-fire automatic executions and emotion-free trading decisions, while still leveraging their innate human capacity for objective thinking about failure and success.

The most important element of any trader is the human capability to evolve. Traders can change and adapt their trading systems in order to continue winning before losses become financially or emotionally devastating.

Choose the right type and amount of market data for testing

Successful traders use a system of repetitive rules to harvest gains from short-term inefficiencies in the market. For small, independent traders in the big world of securities and derivatives trading, where spreads are thin and competition fierce, the best opportunities for gains come from spotting market inefficiencies based on simple, easy-to-quantify data, then taking action as quickly as possible.

When a trader develops and operates mechanical trading systems based on historical data, he or she is hoping for future gains based on the idea that current marketplace inefficiencies will continue. If a trader chooses the wrong data set or uses the wrong parameters to qualify the data, precious opportunities may be lost. At the same time, once the inefficiency detected in historical data no longer exists, then the trading system fails. The reasons why it vanished are unimportant to the mechanical trader. Only the results matter.

mechanical trading rules

Pick the most pertinent data sets when choosing the data set from which to create and test mechanical trading systems. And, in order to test a sample large enough to confirm whether a trading rule works consistently under a wide range of market conditions, a trader must use the longest practical period of test data.

So, it seems appropriate to build mechanical trading systems based on both the longest-possible historical data set as well as the simplest set of design parameters. Robustness is generally considered the ability to withstand many types of market conditions. Robustness should be inherent in any system tested across a long time range of historical data and simple rules. Lengthy testing and basic rules should reflect the widest array of potential market conditions in the future.

All mechanical trading systems will eventually fail because historical data obviously does not contain all future events. Any system built on historical data will eventually encounter ahistorical conditions. Human insight and intervention prevents automated strategies from running off the rails. The folks at Knight Capital know something about live trading snafus.

Simplicity wins by its adaptability

Successful mechanical trading systems are like living, breathing organisms. The world’s geologic strata are filled with fossils of organisms which, although ideally suited for short-term success during their own historical periods, were too specialized for long-term survival and adaptation. Simple algorithmic mechanical trading systems with human guidance are best because they can undergo quick, easy evolution and adaptation to the changing conditions in the environment (read marketplace).

Simple trading rules reduce the potential impact of data-mining bias. Bias from data mining is problematic because it may overstate how well a historical rule will apply under future conditions, especially when mechanical trading systems are focused on short time frames. Simple and robust mechanical trading systems shouldn’t by affected by the time frames used for testing purposes. – The number of test points found within a given range of historical data should still be large enough to prove or disprove the validity of the trading rules being tested. Stated differently, simple, robust mechanical trading systems will outshine data-mining bias.

If a trader uses a system with simple design parameters, such as the QuantBar system, and tests it by using the longest appropriate historical time period, then the only other important tasks will be to stick to the discipline of trading the system and monitoring its results going forward. Observation enables evolution.

On the other hand, traders who use mechanical trading systems built from a complex set of multiple parameters run the risk of “pre-evolving” their systems into early extinction.

Build a robust system that leverages the best of mechanical trading, without falling prey to its weaknesses

It’s important not to confuse the robustness of mechanical trading systems with their adaptability. Systems developed based on a multitude of parameters led to winning trades during historical periods – and even during current observed periods – are often described as ‘robust.’ That is no a guarantee that such systems can be successfully tweaked once they have been trade past their “honeymoon period.” That is an initial trading period during which conditions happen to coincide with a certain historical period upon which the system was based.

Simple mechanical trading systems are easily adapted to new conditions, even when the root causes of marketplace change remain unclear, and complex systems fall short. When market conditions change, as they continually do, the trading systems which are most likely to continue to win are those which are simple and most-easily adaptable to new conditions; a truly robust system is one which has longevity above all.

Simple algorithmic mechanical trading systems with human guidance are best because they can undergo quick, easy evolution and adaptation to the changing conditions in the environment (read marketplace).

Unfortunately, after experiencing an initial period of gains when using overly-complex mechanical trading systems, many traders fall into the trap of attempting to tweak those systems back to success. The market’s unknown, yet changing, conditions may have already doomed that entire species of mechanical trading systems to extinction. Again, simplicity and adaptability to changing conditions offer the best hope for survival of any trading system.

Use an objective measurement to distinguish between success and failure

A trader’s most-common downfall is a psychological attachment to his or her trading system. When trading-system failures occur, it’s usually because traders have adopted a subjective rather than objective viewpoint, especially with regard to stop-losses during particular trades.

Human nature often drives a trader to develop an emotional attachment to a particular system, especially when the trader has invested a significant amount of time and money into mechanical trading systems with many complex parts which are difficult to understand. However, it’s critically important for a trader to step outside the system in order to consider it objectively.

In some cases, the trader becomes delusional about the expected success of a system, even to the point of continuing to trade an obviously-losing system far longer than a subjective analysis would have allowed. Or, after a period of fat wins, a trader may become “married” to a formerly-winning system even while its beauty fades under the pressure of losses. Worse, a trader may fall into the trap of selectively choosing the testing periods or statistical parameters for an already-losing system, in order to maintain false hope for the system’s continuing value.

An objective yardstick, such as using standard deviation methods to assess the probability of current failure, is the only winning method for determining whether mechanical trading systems have truly failed. Through an objective eye, it’s easy for a trader to quickly spot failure or potential failure in mechanical trading systems, and a simple system may be quickly and easily adapted to create a freshly-winning system once again.

Failure of mechanical trading systems is often quantified based on a comparison of the current losses when measured against the historical losses or drawdowns. Such an analysis may lead to a subjective, incorrect conclusion. Maximum drawdown is often used as the threshold metric by which a trader will abandon a system. Without considering the manner by which the system reached that drawdown level, or the length of time required to reach that level, a trader should not conclude that the system is a loser based on drawdown alone.

Standard deviation versus drawdown as a metric of failure

In fact, the best method to avoid discarding a winning system is to use an objective measurement standard to determine the current or recent distribution of returns from the system obtained while actually trading it. Compare that measurement against the historical distribution of returns calculated from back-testing, while assigning a fixed threshold value according to the certainty that the current “losing” distribution of mechanical trading systems is indeed beyond normal, to-be-expected losses, and should therefore be discarded as failed.

So, for example, assume that a trader ignores the current drawdown level which has signaled a problem and triggered his investigation. Instead, compare the current losing streak against the historical losses which would have occurred while trading that system during historical test periods. Depending upon how conservative a trader is, he or she may discover that the current or recent loss is beyond, say, the 95% certainty level implied by two standard deviations from the “normal” historical loss level. This would certainly be a strong statistical sign that the system is performing poorly, and has therefore failed. In contrast, a different trader with greater appetite for risk may objectively decide that three standard deviations from the norm (i.e. 99.7%) is the appropriate certainty level for judging a trading system as “failed.”

The most important factor for any trading systems’ success, whether manual or mechanical, is always the human decision-making ability. The value of good mechanical trading systems is that, like all good machines, they minimize human weaknesses and empower achievements far beyond those attainable through manual methods. Yet, when properly built, they still allow firm control according to the trader’s judgment and allow him or her to steer clear of obstacles and potential failures.

Although a trader can use math in the form of a statistical calculation of standard distribution to assess whether a loss is normal and acceptable according to historical records, he or she is still relying on human judgment instead of making purely-mechanical, math-based decisions based on algorithms alone.

Traders can enjoy the best of both worlds. The power of algorithms and mechanical trading minimizes the effects of human emotion and tardiness on order placement and execution, especially with regard to maintaining stop-loss discipline. It still uses the objective assessment of standard deviation in order to retain human control over the trading system.

Be prepared for change, and be prepared to change the trading system

Along with the objectivity to detect when mechanical trading systems change from winners into losers, a trader must also have the discipline and foresight to evolve and change the systems so they can continue to win during new market conditions. In any environment filled with change, the simpler the system, the quicker and easier its evolution will be. If a complex strategy fails, it may be easier to replace than to modify it, while some of the simplest and most-intuitive systems, such as the QuantBar system, are relatively easy to modify on-the-fly in order to adapt to future market conditions.

In summary, it can be said properly-built mechanical trading systems should be simple and adaptable, and tested according to the right type and amount of data so that they will be robust enough to produce gains under a wide variety of market conditions. And, a winning system must be judged by the appropriate metric of success. Instead of merely relying on algorithmic trading rules or maximum drawdown levels, any decision about whether a system has failed should be made according to the trader’s human judgment, and based on an assessment of the number of standard deviations of the system’s current performance when measured against its historic-test losses. If mechanical trading systems are failing to perform, the trader should make the necessary changes instead of clinging to a losing system.

When Does Adding A Secondary Filter Increase Performance?

by Leave a Comment

Earlier posts this week addressed the need to keep a strategy simple in order to avoid exposure to curve-fitting bias. While that is true in a general sense, there are also times when adding a level of complication to a strategy can improve its performance without significantly increasing its risk of ruin.

We saw an example of this with Shaun’s Euro Scalping Strategy. When a timing filter is applied that only trades the strategy during slower trading times, the performance metrics shot up, giving us a very interesting system. What would happen if we added a second layer to this filter?

filter

Adding multiple filters can help a strategy single out the very best trades, but it can also expose a strategy to curve-fitting bias.

Jeff from System Trader Success took his research on Shaun’s Euro Scalping Strategy a step further this week. He decided to experiment with introducing a volatility filter to the strategy. Once he determined the optimal parameters for the filter, he tested how it would improve the base scalping strategy as well as the time filtered version.

Optimizing the Volatility Filter

Jeff’s goal was to use daily price action to determine how Shaun’s Euro Scalping Strategy performed on days where volatility was high and compare that with how the strategy performed on days where volatility was low.

The biggest lesson here is that in order to optimize the filter, Jeff went back to the original scalping strategy. In order to get the most amount of data and limit curve-fitting, he scrapped the time filter for now.

The lesson here is that if you are going to use multiple filters, it is a good idea to build, test, and optimize them separately. Then you can stack them together. Stacking the filters while optimizing them can skew the data in a way that confuses which filters are actually working.

The Volatility Filter

Applying the volatility filter to the original scalping strategy slightly improved the strategy. It increased the profit factor from 1.38 to 1.43 while significantly lowering the maximum drawdown. The average profit per trade rose from $24.58 to $25.45.

The win rate was kept pretty much the same, and the overall net profit was reduce because 128 trades were filtered out. As you can see, there was definitely improvement, but not nearly as much as with the time filter.

Combining Filters

While the results of applying the volatility filter to the original strategy were not impressive, the original plan was to apply the filter to the time filtered strategy. Putting the two filters together produces the best version of the scalping strategy yet. 

The combined filters improve the profit factor to 3.44 and the average profit per trade to $56.74. The maximum drawdown is reduced to less than half of the maximum drawdown of the original system.

The new combined filter strategy reduces the total trades to 164, where the time filtered strategy had 233 and the original strategy had 456. Despite how few trades the combined filtered strategy makes, it returns almost as much net profit as the original strategy.

What we have done with these combined filters is find a way to isolate the very best of the best trades from the original strategy.

 

 

Time Frame

by Leave a Comment

The feedback on the group trading strategy was phenomenal! I’ve never had so many emails with such a broad range of opinions. Thank you to everyone for your response and thoughts.

Numerous people pointed out an obvious oversight on my part: which timeframe will the strategy use? Before you have a heart attack, please hear me out. Notice on the graph (republished below) that the curve really does not kick outward until 0.3% or more. On the current EUR\USD price, that’s a movement of 39 pips (0.3% * 1.3000). I used the M1 chart to create the graph. The strategy, too, will use the M1 chart for its signals.

EURUSD price distance from SMA 200

The graphs displays the percent distance of the price from the SMA against frequency (i.e., how often is the price this far from the 200 SMA?)

 

Another reason to consider the M1 chart is the number of trades per year. The average number of bars that the price spends above or below a 200 SMA is… 200 bars. This is based on research that I performed earlier this year. There are approximately 386,000 trading minutes per year. Divide that by an average of 200 M1 bars between crossings and you get a maximum of 1,930 trades per year (386,00/200).

I’m only looking at taking a percentage of those trades. If 20% of the crosses result in desirable opportunities, that is 386 trades per year. While active, it’s hardly crazy. 386 trades per year averages to ~1.5 trades per day.

Leave comments below if you see anything worth adding on the time frame.

After-thoughts

This series eventually led to a profitable trading strategy. If you’d like to read through the journey, then I suggest reading the articles sequentially

The initial strategy idea
Time frame
A research plan
An annoying surprise in the initial backtests
An attempt at range trading
Range trading results
The moving average envelope scalper

Buy and Hold Gold

by Leave a Comment

I came across a paper entitled A Quantitative Approach to Tactical Asset Allocation by Mebane Faber. This paper appears to have been extremely popular over the years. Reading through the contents and reviewing the charts, I loudly wondered if the strategy might apply to gold.

Rules for buying gold

The idea is very simple. Taken directly from the paper, the rules are:

BUY RULE
Buy when monthly price > 10-month SMA.

SELL RULE
Sell and move to cash when monthly price < 10-month SMA.

1. All entry and exit prices are on the day of the signal at the close. The model is only
updated once a month on the last day of the month. Price fluctuations during the rest of
the month are ignored.
2. All data series are total return series including dividends, updated monthly.
3. Cash returns are estimated with 90-day Treasury bills, and margin rates (for leveraged
models to be discussed later) are estimated with the broker call rate.
4. Taxes, commissions, and slippage are excluded.

Test results

The tests were done using Kinetick’s free end of day data. The data extends from August 4, 1997 until December 10, 2012.

One thing which drives me crazy about NinjaTrader is that the percent return calculations are so opaque. The actual numbers used a clearly wrong. The buy and hold return for gold is easy to calculate. The 1997 price was about $350 an ounce. Today, gold trades near $1,700. The buy and hold return should be in the neighborhood of 385%. NinjaTrader claims that the return is somewhere near 150%. I’ve gone through the charts to confirm that the trades are correct. You can download the gold strategy for NinjaTrader and verify the trades for yourself.

The percent return metric is consistent within the platform, so luckily the exact numbers are unimportant. The only thing that really matters is whether or not the strategy returns a bigger number than the buy and hold approach.

Equity curve of buying and holding gold

A chart of gold’s daily equity curve, extending back to June 1997.

Equity curve of the gold trading strategy

The tactical gold strategy dramatically underperforms the buy and hold return.

Reverse gold strategy equity curve

Pursuing the opposite idea of buying on crosses underneath the SMA10 doesn’t help

The strategy severely underperforms the buy and hold return. The idea held in the paper itself is deeply flawed. Buy and hold is a silly concept. Most of the “returns” in the portfolio stem from long term inflation and the devaluing of the dollar. The prices of the securities rose because the dollar, which you can think of as the counter currency, declined enormously in value over this time period.

Additionally, the idea of the price crossing the moving average implies that the trend escapes the moving average. The MA, in effect, gets dragged along with the price. While this is true with a handful of monster trends, the way that price moves usually work is something along the lines of 10 steps forward, 9 steps backward. The results of the test indicate this.

If the proposed strategy returns less than buy and hold and the total buy and hold strategy returned 150%, then it stands to reason that the total buying return is trades taken by the strategy + trades not taken by the strategy. The test includes what are supposed to be trades not taken by the strategy – the reverse signals. Adding up the proposed strategy’s returns with the returns of the opposite signal only accounts for roughly one third (22% + 34%) of the 150% earned. Where did the rest of the money go?

Most of the move happens around the short term moving average. The majority of the move has already happened after the price to crosses and closes above the moving average. The instinct of many novice strategy developers is to move to intrabar signals. Intrabar trading ignores the basic problem; you cannot know whether or not the bar will close above the moving average.

Strategy Decay

by 1 Comment

Ernie Chan wrote a great blog post today on the life and death of a strategy. He talks about some of my favorite points: KISS (Keep It Simple, Stupid!) and the importance of viewing real performance relative to negative performance.

If you like this blog, then you’ll love his. It’s very similar. The only difference is that Ernie tends to write with more assumptions about his readers’ math skills.

NinjaTrader Order Methods

by Leave a Comment

Ninjascript, the C# API used for programming strategies in NinjaTrader, attempts to stirke a balance between making the strategy development process as simple as possible while preserving flexibility in the types of strategies that it can support.  It handles these competing requirements by splitting strategies into two types of order methods. Many programmers within the NinjaTrader community also refer to these types as approaches.

Managed approach

The first and most common order method is a managed approach. The managed approach is also the default assumption that NinjaTrader assumes you’re using when writing a strategy.

I backtest strategies this way 90%+ of the time. It’s way easier to write them using a managed approach once all of the variables and properties are set up. NinjaScript makes available a number of methods that all inter-relate. The names and functionality of each is obvious. EnterLong(), for example, enters a long trade.

All of these methods have premapped relationships between each other. When the strategy calls ExitLong(), NinjaTrader takes care of the details such as knowing which long it needs to exit. That seems obvious enough, but many trading applications do not connect the dots so easily between trading functionality. MetaTrader 4 & 5 are good examples of this type of simple functionality not being available.

The ease with which trading decisions are handled makes it ideal for developing quick and dirty code to test simple ideas.  Most of my strategy ideas follow the KISS principle; keep it simple, stupid. It’s usually a very simple development process (10-20 minutes) to program and test a trading idea from the moment it enters my mind. Most ideas don’t work out, but at least I get the satisfaction of knowing the answer quickly.

I discovered the hard way that the managed approach makes assumptions about order handling. EnterLongLimit(), for example, automatically deletes a pending limit order after one bar. I remember spending several hours with a live strategy trying to figure out why so many of my orders entered correctly on tick charts, only to have them disappear on the next tick. Carefully poring over the documentation led me to discover an overloaded method with the liveUntilCandelled parameter.

It’s examples like this pre-programmed functionality that you need to watch out for. This is especially true if you have a bug that doesn’t make sense in the context of your strategy.

Unmanaged approach

An unmanaged approach takes away all of the assumptions that NinjaTrader makes while a strategy runs in real time. This option leaves it up to the coder to store the position status in memory and make appropriate decisions.

Every strategy that we’ve written with unmanaged orders uses pending stop and limit orders. A recent example involves a market making strategy at Interactive Brokers. The strategy is 100% in the market during the US and Asian sessions. We originally started with a managed approach but ran into problems with the commissions and overfills.

The strategy is only trading 2 mini lots during the testing phase, which is the minimum trade size. Although the standard commission of 0.2 pips per side is quite low, the minimum commission is $2.50. That’s relatively high for such a small position size. I noticed on the first day that my client was getting charged commissions for the entry leg and exit leg, respectively.

The first trade entered with 20k. When it came time to exit, the managed approach set an exit limit for 20k and a separate entry limit for another 20k. IB charged commissions twice for both orders, which cost $5 per round turn trade. The trading report looked like:

BOT 20k $2.50

SLD 20k $2.50

SLD 20k $2.50

BOT 20k $2.5o

BOT 20k $2.50

SLD 20k $2.50

BOT 20k $2.50

SLD 20k $2.50

That’s effectively 4 trades at a cost of $20.

Grouping the trades into a single order more or less cut the commissions in half. I found it easier to do with an unmanaged approach because I didn’t have to worry about how the Entry() and Exit() methods would interact. The trade report changed to:

BOT 20k $2.50

SLD 40k $2.50

BOT 40k $2.50

SLD 40k $2.50

BOT 20k $2.50

The same sequence grouped together drops the cost of the 4 trades down to $12.50.  The initial entry and the final exit cost $2.50, but the trades in between only pay $2.50 for the entry and exit combined. When the order size increases to standard lots, the commission savings will dramatically improve the client return.

The other reason that we switched to an unmanaged approach was that NinjaTrader kills a strategy whenever an overfill occurs. An overfill is when a strategy requests cancellation of a pending order, but that pending order gets filled before the cancel request arrives. It then exits the new position at market and turns off the strategy.

I found this nearly intolerable with a market making strategy because an overfill would almost certainly occur at least once daily. Unmanaged orders have the option of disabling an overfill and handling it in whatever manner the trader deems appropriate.  In my case, it was to simply increase the amount of position that the strategy was trying to exit.

The main disadvantage to unmanaged orders is that everything is tracked internally. If you need to disable a strategy to change the settings or lose the broker connection, the code is probably not going to interact very well with positions that are already open.  Whatever IOrder objects were stored in the memory, which is how the strategy knows what positions are open, disappear whenever the strategy is removed from a chart.

Unmanaged strategies also take a lot more time to test. The amount of code involved does not differ dramatically from managed orders. It’s larger, but not grotesquely so. The reason that it takes more time is that the interaction between the order types is not polished like it is in managed orders. The programmer must write everything from scratch, which invariably means that the strategy will suffer from many more initial bugs, especially in a live environment.

Strategies that rely on market orders do not need to consider this approach in my opinion. It causes a lot more work and I have yet to encounter a situation where it makes sense. More importantly, I can’t think of a hypothetical situation where it would make sense either.

High Frequency NinjaTrader Strategy

by 7 Comments

I’ve been working on a high frequency trading system for NinjaTrader on behalf of a long term client. My live account is with MB Trading. Rather than placing market orders and paying a commission, I changed the order types to limit orders. We want to receive a small commission for the market making strategy rather than paying a commission to accept the displayed prices.

MetaTrader suffers two major disadvantages that make NinjaTrader a superior option for high-frequency trading. MT4 does not offer charts lower than the M1 time frame and the trade context is busy error prevents multiple charts from running simultaneously. NinjaTrader is complex enough to where I can control most details, but simple enough that I don’t need to invest hundreds of hours to test an idea. After extensively testing the strategy on M1 charts as a price taker, I feel very confident that the strategy is sound. The only issue now is determining whether or not whether taking a passive (i.e., market making) approach will result in enough fills to make the strategy worthwhile.

The first issue that I came across wasn’t with NinjaTrader; it was with MB Trading’s API. The strategy worked fine on the simulation account, which only routes orders to NinjaTrader (NT). NT then makes guesses when fills would occur. The goal of that phase was not to test the strategy. I only wanted to test the programming to make sure that it worked properly.

100 trades went off without a hitch in the Sim account. The strategy only made it through 2-3 microlot trades on the live account before the pending orders hung. NinjaTrader pending orders pass through 3 states before they actually hit the market. For the programmers out there, these are the OrderState properties of IOrder objects.

  1. Pending submit – the strategy sent the order to the broker and is waiting to hear back
  2. Accepted – the broker acknowledges receipt of the order, but is still placing the order into the market
  3. Working – the order is available for others to trade

The strategy updated orders on every tick. What often happened was that the pace would go far too quickly, creating a major communications backlog during fast markets. NinjaTrader never threw an exception. The only evidence of a problem was that I would see a hanging order with the PendingChange property. The inconvenient solution was to exit NinjaTrader and reload everything.

I figured that perhaps that the managed order state caused the issue. I changed my approach to unmanaged orders, but that did not make a difference. I eventually came to the realization that the MB Trading API cannot handle more than one order every few seconds.

The strategy found the sweet spot after changing from tick to second charts. Updates of 6 seconds or longer seem to give the MB Trading API enough time to update wihle still preserving something of a high frequency approach. Any trades that need to run faster than that threshold at MB Trading need to use the FIX protocol.

The other element that drove me crazy is that NinjaTrader limit orders automatically delete themselves once per bar. I nearly tore my hair out, and I don’t have all that much hair, for several hours trying to figure out why orders deleted themselves automatically. Many people identify with the school of hard knocks approach to learning. I’m as thick headed as most. I figured out the cause when I revisitied NinjaTrader’s online documentation and discovered a limit entry method that allows good till cancelled (GTC) orders.

The speed problem also manifested with overfills. An overfill is when a strategy requests to cancel a pending order, but the broker fills the order before the cancellation takes effect. The biggest concern with overfills is that NinjaTrader automatically disables a strategy and exits positions at market when an overfill occurs. The only way to programmatically prevent this is to change the entry methods to an unmanaged approach.

The easiest way to develop for a high frequency strategy in NinjaTrader (but not ultra-high frequency) is to use managed orders. Whenever an exit is needed, place the limit entry in the opposite direction. NinjaTrader takes care of placing the exit order for the open market positoin. Limit the updates to every handful of seconds. It allows the broker API to catch up and helps avoid the problem of overfills.

Backtesting for efficiency

by 6 Comments

Your forex backtests are absolutely worthless if you do not test the statistical entry efficiency and exit efficiency of the strategy. Everyone that runs a backtest inevitably reports the dollars earned as the outcome. Other factors exist like the average win to loss, the profit factor and the Sharpe ratio, but they do not tell you anything useful until the final step of designing an automated trading system.

The correct approach to testing a strategy should focus on the question, “is my strategy a piece of garbage?” Most people try to prove themselves right. The real test is to not be able to prove yourself wrong. The only way to do that is through a statistical approach.

Entry and Exit Efficiency

Efficiency puts a hard number to what percentage of an available trading range that a strategy captures. The trading window starts on the bar where a trade entered the market. The window closes when the trade exits.

The total available window is the highest high minus the lowest low in the window. Calculating the entry and exit efficiency simply measures what percentage of that window that your strategy tends to capture. Take the average of all of the trades and you get the overall efficiency.

Entry efficiency formula

Formula for a long trade: (Highest high – entry price) ÷ (Highest high – lowest low)
Formula for a short trade: (entry price – lowest low) ÷ (Highest high – lowest low)

Exit efficiency formula

Formula for a long trade: (Exit price – lowest low) ÷ (Highest high – lowest low)
Formula for a short trade: (Highest high – exit price) ÷ (Highest high – lowest low)

Take an example where you buy a hypothetical currency at 150 and sell it at 170. The lowest low between the time of entry and exit was 140. The price then ran all the way up to 200 before settling back down to 170, which is where the exit took place.

The entry efficiency is (200-150) ÷ (200-140) = 50 ÷ 60 = 83%. Nearly anyone would agree that this makes for a great entry.
The exit efficiency is (170-140) ÷ (200-140) = 30 ÷ 60 = 50%. Most would agree that the exit would have ideally occurred sooner than it did.

Efficiencies do not change by instrument or time frame

One major problem that we encounter with forex backtests is the limited data set. This is especially true for those interested in testing long term strategies like those on the H4 or D1 charts. The wonderful thing about entry and exit efficiencies is that they do not vary from chart to chart or even period to period.

I like to jump down to M1 charts for efficiency testing. The data is nearly endless. I never have to worry about running out. The great thing is that I know when I shift back to the H4 chart, the efficiencies should not change more than ±5%.

If you see the efficiency vary too much, then you may not have enough trades to form a statistically significant group. My experience tells me that 75 trades usually gets very close to the actual efficiency. 100 trades or more is better. When I run tests on M1 charts, I often get several thousand trades over the course of a few months. Numbers that large can tell you with a great deal of confidence just how robust a strategy’s parameters truly are.

Usually, you can assume that any results that fall within 45-55% are the result of a random, stochastic process. When I see backtests that creep right up to those barriers like 54.9% or even 55.1%, the results inevitably tank back to around the 50% mark.

Random trade outcomes and dollar profits

I wish this section was about how to make money with a random efficiency. Alas, we must cover how randomness can result in unjustified eurphoria.

I’ve been interested in the concept of randomness for several years now. Mathematicians refer to it with the more opaque name of a “stochastic process”. Despite the non-sensical name, it’s just a fancy way of saying the study of randomness – how it changes, its distribution, how far it “walks”, etc.

Yesterday, I used the analogy of coin flips to describe how Martingale strategies are probabilistically doomed to failure. One interesting concept that I did not mention relates to Brownian motion. Even with a set of random outcomes, trades will go on a random walk away from the starting point.

Einstein gets the real credit for solving the math behind the concept, even though his name is not on the term. He demonstrated that the distance a random process will follow is the square root of the number of trials. If we decide to flip a coin 60 times, we know that 50% of the time should fall on heads and the other 30 on tails.

It actually turns out that we should expect a very slight bias in the number of either winners or losers, although we do not know which one. It’s random. The precise bias, whichever way it prefers to go, should equal √60, which works out ~7. The heads outcomes should typically range from 23-37, with the tails outcomes making up the difference.

Seven trades out of sixty strongly alters the percentages, even if we know that it’s really supposed to be 50%. If heads only came up 23 times out of 60, that’s 38%. The problem is not with the coin. It’s with the number of trials. As you do an increasing large number of trails, the random bias decreases in significance in terms of the percent accuracy. 50,000 trades, for example, should show a surplus of roughly 223 trades in favor of winning or losing. The accuracy range falls within 1% of 50% on either side, a dramatic improvement.

Risks of curve fitting

Curve fitting a random efficiency relates to the idea of Brownian motion. Let’s say that we use a strategy that I know will never show an entry or exit efficiency: the moving average crossover. I’ve gone through this strategy six ways from Sunday, almost exclusively at the behest of clients. It does not work as a fully automated strategy. There is no secret set of fast and slow periods that will unlock the hidden keys to profit.

Most traders, experienced or not, abuse the backtester by searching for a set of parameters that yield the most dollar profit. They curve fit their test to optimize for maximum profit. What really happens is that the traders optimize for amount of random drift that already occurred.

When I used the example of 50,000 trades creating a natural drift of 223, I cited it with the purpose of showing how little it reduces the error in the real percent accurracy. The other consequence for trading systems is that as the error percentage decreases, the natural bias in your outcomes increases. Blindly running the optimizer only selects the set of combination that yields a combination of two criteria:

  • The drift that happened to work out in favor of that set of parameters
  • The profit and loss that varies with those parameters. The dollar profit naturally changes because the two moving averages cross at different points

You need a tool like efficiency to guard against these types of random outcomes. It’s the only method that I know of that definitively states whether or not a strategy behaves in a random manner. I especially like the fact that it breaks those elements down into two of the three basic components of a trading strategy: the entry, the exit, and the position sizing.

Efficient strategies do not work all of the time

Position sizing marks the final obstacle to building your fully automated trading strategy. A set of rules that yields a statistically efficient entry that is paired with an efficient exit does not necessarily make money. The value of each trading setup can vary, too.

Each strategy contains different sets of winners and losers. Each winner and loser varies in its dollar value. Whatever money management approach that you take requires balancing the ratio of the winners and losers in a way that normalizes the outcome of each trade. You ideally want to eliminate the variation in dollar value. 20 pip trades should earn or lose you exactly as much as the 100 pip trades.

That seems counter-intuitive. Most traders want to win in proportion with the size of the opportunity. It’s better from a system perspective to entirely ignore the size of the opportunity and to make each trade worth the same amount. Betting more or less with each trade effectively normalizes the value of each trade.

Using a stop loss stands out as an obvious candidate to fix how much a trade is worth. The severe disadvantage is that it almost always negatively affects the exit efficiency. Whenever I can get away with it, I always recommend using a market based exit instead of an arbitrary stop loss. Traders usually scream at the top of their lungs when they hear me say this. I’m just speaking as a systems developer. The numbers are what they are.