ninjatrader Archives - Algorithmic and Mechanical Forex Strategies

Coppock Curves : A Straight Line To Trading Success

April 15, 2014 by Eddie Flower Leave a Comment

Coppock Curves, sometimes called Coppock indicators or Trendex indicators, are a type of indicator which offers quant traders a solid foundation upon which to build a simple yet successful mechanical trading system.

As described in more detail below, I use Coppock Curves in my mechanical trading system to generate trading signals in the S&P 500 or any other highly-liquid index. Coppock Curves also work well for trading iShares and ETFs.

What is a Coppock Curve?

Coppock Curves are a momentum indicator. Over time, they oscillate over and under zero. The Coppock Curve indicator was first described in 1962 by the economist and trader Edwin Coppock. In fact, it works so well that the Market Technicians Association (MTA) recognized Dr. Coppock with a lifetime achievement award in 1989.

Its value lies in showing the beginning of long-term changes in price trends of stocks and indexes, particularly at the beginning of upward trends. This indicator can also signal the bottoms of futures and forex markets, yet I’ve found it less reliable there.

Although you can program your mechanical trading algorithms to generate trading signals based on this indicator over any time frame, I typically use it with monthly charts across a wide range of stock and index markets. Still, active traders can certainly use Coppock Curves with daily or even hourly time periods.

Specifically, I use Coppock Curves to generate “buy” signals at the bottom of bear markets. This indicator is especially good for distinguishing between bear rallies and actual market bottoms.

This is a trend-following indicator, so it doesn’t precisely show a market bottom. Instead, it shows me when a strong, bullish rally has become safely established enough to trade confidently.

Best of all, in my experience trades from signals based on Coppocks Curves are fairly resistant to shakeouts and whipsaws. Coppock Curves are slow, but they’re safe.

Coppock Curves signal the end of a “mourning period”

As background, it’s worthwhile to note that the original idea which led Dr. Coppock to develop his indicator was based on the natural cycle of life, death, and mourning before returning to new life again.

He thought that the normal upward march of stock markets (and therefore stock indexes) was like the “life” part of the cycle, which of course was followed by “death” that is, the period of falling prices during a bear market.

Dr. Coppock was particularly interested in calculating the length of a stock market’s “mourning” period, after which it would be safe to re-enter the market “long” again. Logically, this entry point at the end of the mourning period would represent the beginning of the next long-term uptrend.

The apocryphal story says that he asked the bishops at a local Episcopal Church, one of his investment clients, how long people usually spent in mourning after bereavements. He was told that human mourning typically requires between 11 to 14 months, so those were the values he adopted in his original equation to determine when stock prices would begin to rise again.

Coppock Curves were first used as long-term indicators based on monthly charts. Of course, the signals generated with monthly time frames are fairly infrequent. Still, because I use Coppock Curves to trade a variety of markets, I receive plenty of trading signals.

In particular, the monthly time frame is very reliable for stock and index trading. Studies have shown that, since 1920 in the U.S. stock markets, Coppock Curves have generated winning signals with about 80% frequency.

Nowadays, with the rapid turnover in modern markets, it seems that trading cycles have become faster. In addition to monthly time frames, some traders have found that daily time frames work very well in generating successful Coppock Curve signals.

A trader can program a mechanical trading system to recognize and respond to signals based on a daily or hourly time frame, although additional algo trading parameters should be added to reduce the chance of overtrading.

If you want to use Coppock Curves to generate signals on shorter time frames, you could experiment with your mechanical trading system using a variety of make-sense “mourning periods” for your particular marketplace.

How to calculate Coppock Curves

The Coppock indicator is based on three variables: A shorter-term rate of change (abbreviated as ROC), and a somewhat longer-term ROC. Coppock Curves are developed by using the weighted moving average (WMA) derived from the chosen time periods of a given market index.

The classic equation stated in words:

Coppock Curve = The 10-period WMA of a 14-period ROC plus an 11-period ROC

Or, as a formula for programming:

Coppock Curve = WMA[10] of (ROC[14] + ROC[11])

When ROC = [(Close – Close n periods ago) / (Close n periods ago)] * 100

Where n is the number of time periods.

In the classic scenario, 11 and 14 time periods. Be sure to make separate ROC calculations.

As you can see, the basic setup is very simple – On a moving basis, I program my mechanical trading system to calculate the percent of change in a given index (say the S&P or DJIA) from fourteen months ago.

Then, my mechanical trading program calculates the percentage change in the same index from eleven months ago. Next, the mechanical trading system adds together the two different percent changes. Then, it calculates a 10-period weighted moving average of the above total.

It’s important to note that you can use different time periods for the ROC calculations and the WMA calculations. I sometimes program my mechanical trading system to use the classic 11- and 14-month time periods for ROC while using time periods for the WMA which are shorter than the classic 10-month period.

So, I often use using a 2-month or 3-month WMA (instead of 10 months) while the ROC is calculated using the 11- and 14-month prices.

Or, you can modify your mechanical trading system to employ shorter time periods for some or all of the calculations, i.e. use daily or hourly prices instead of monthly price charts. It generates more signals, but in my experience they’re less reliable unless you add additional filters, as discussed below.

As well, you can add additional embellishments to suit your own needs. In any event, the general method remains the same. When charting the basic inputs, you’ll see that the output is a fairly smooth arc, hence the name of this indicator.

In any event, the classic Coppock Curve equation for programming a mechanical trading system can be stated as: The sum of the 14-month rate of change and the 11-month rate of change, with smoothing by applying a 10-month weighted moving average.

The Coppock Curve “buy” signal

On Coppock Curves, the zero line is the trigger. When the price line rises from below the 0 line it signals a low-risk buying opportunity. My mechanical trading system executes a buy when the Coppock indicator is first below 0, then heads upward from the trough.

Since this is most effective as a bullish indicator, I ignore the opposite (“sell”) signals. Still, some traders, especially those using short time frames, use Coppock Curves with algo trading systems to generate sell signals and execute trades that close out long positions. Active traders can both close long trades and open shorts when the Coppock Curve crosses below the zero line.

The figure below shows the classic Coppock Curve trading strategy using monthly time periods. The buy signal came in 1991. The sell signal came ten years later, in 2001. Note that this long time frame helped me avoid the slump in late 2001 and 2002.

The next buy signal came in 2003 and the sell signal was in 2008. This helped me escape the slump in 2008 and into 2009. Note, also that the current “buy” position, signaled in early 2010, continues to remain open, at least through the date of this chart.

The Coppock Curve on an S&P 500 monthly chart

Next, for more-active traders here’s a screenshot showing the strategy applied with shorter time periods, as shown on a daily S&P 500 chart. Of course, many more signals are generated, although in general they are less likely to be winners.

Coppock Curve on a daily S&P 500 chart

Importantly, the longer the time period, the safer the buy signal. Since my mechanical trading system based on Coppock indicators is a trend-following system, I don’t necessarily capture the immediate gains from the exact moment of a trend reversal. Instead, my mechanical trading system gets me “long” just before the beginning of a profitable advance in a bull market.

Adjusting and filtering signals from Coppock Curves

I’ve found Coppock Curves to be highly reliable when used for monthly time periods. In my experience, using weekly, daily or hourly time periods usually means that my entries and exits aren’t as “tight” as I would like, meaning that I don’t capture all the gains I had hoped for, and I also have more losses.

However, active traders can decrease the ROC variables, which has the effect of increasing the speed of fluctuation in Coppock Curves and will therefore generate more trading signals. Of course, even though monthly time periods are my favorite, an ultra-long-term trader could also increase the ROC time periods to slow fluctuations even more, thus generating fewer signals.

As I’ve said above, in order to receive earlier entry signals, I usually decrease the WMA downward from 10 months, sometimes to 6 months, and often to as little as 2 months. By programming my mechanical trading system carefully with just the right WMA period, and filtering the signals, I maximize my profitability in a given market.

If you want to use Coppock indicators for active trading, I recommend that you filter the trade signals generated by your mechanical trading system so that you only accept trades which are in the same direction as the current dominant trend. You’ll find this mechanical trading strategy to be the most profitable, since you can avoid many losing trades by filtering the signals.

Which markets show reliable Coppock Curves?

I use my Coppock curve-powered mechanical trading system to trade a range of indexes, especially those based directly on stocks, such as:

Dow Jones Industrial Average
S&P 500
NASDAQ Composite
EURO STOXX 50
FTSE 100
Nikkei 225
Hang Seng

As well, if you’re focused on ETFs you’ll find that a mechanical trading system using Coppock Curves will allow you to catch the beginning of trends in specific market niches, such as biotechnology, energy, and international or regional equities niches.

The key is to make sure you trade only the liquid indexes. Otherwise, you may run the risk of being shaken out during “fake” trend changes.

Trading Coppock Curves in non-equity indexes

As well, for the sake of diversification and to avoid issues with correlation, I also program my mechanical trading system to spot and trade Coppock Curves in non-equity indexes as well. Again, I focus on markets which have sufficient liquidity.

There are some profitable non-equity indexes, including iShares and ETFs, which can be traded using Coppock indicators:

Bloomberg US Treasury Bond Index
Bloomberg Canada Sovereign Bond Index
Bloomberg U.K. Sovereign Bond Index
Bloomberg US Corporate Bond Index
Bloomberg GBP Investment Grade European Corporate Bond Index
Bloomberg EUR Investment Grade European Corporate Bond Index
Bloomberg JPY Investment Grade Corporate Bond Index
iShares Barclays 7-10 Year Treasury Bond Fund
iShares Barclays 20 Year Treasury Bond Fund
Schwab Short-Term U.S. Treasury ETF
Vanguard Short-Term Government Bond ETF
PIMCO 1-3 Year U.S. Treasury Index ETF

I’ve seen reliable signals from Coppock Curves when trading all the above-listed non-equity indexes. As always, the key is to use a mechanical trading system in only those markets which are highly liquid, so that the algorithms are reasonably sure that a confirmed signal is legitimate before trading it.

Coppock Curves show a straight line to success

In recent years, Coppock Curves have been drawing renewed interest from traders who are turning once again to this tried-and-true trading tool. See, for example, these recent mentions of Coppock indicators in the financial press: Jay On The Markets, and the follow up article, as well as in various trader musings.

In summary, I can say that Coppock Curves can lead you straight to success, as long as you have the patience to let your mechanical trading system do the work for you. If you use the length of variables’ time periods which are most appropriate for your chosen markets, you should do very well with Coppock Curves.

Walk Forward Optimization

January 13, 2014 by Shaun Overton 6 Comments

If you were walking and randomly it started to rain, would you consider carrying an umbrella tomorrow? Of course you would.

The reason I ask a rhetorical question like that is when people observe a behavior, they respond accordingly. If they expect that something might happen again, they change their behavior to accommodate the change in outcomes.

When you think about forex robots, everybody has the dream of developing a strategy that works forever. It requires no changes. The initial settings always work. Turn it on and move to the beach.

Reality, of course, is more complicated than that.

Walk forward optimization continually optimizes throughout time instead of looking for one set of static settings

That leads to expectations of what you need to do when your strategy inevitably goes awry. It’s very possible that you come up with a strategy that works and does amazingly well on the current market. However, a past genius doesn’t mean future genius. There’s always the chance that your strategy will no longer work in the future.

Why is that? It’s the same reason that you might carry an umbrella tomorrow if it rains today. People observe the market performing in a consistent manner. As more and more people make the observation, people start trading on it. The market responds to those changes, and eventually the opportunity completely washes out as too many people have eared about it.

Walk forward testing is the process of determining whether or not your strategy has washed out. By testing on one set of data, and then testing it on a blind set, you can give yourself an indication of whether your strategy is bad or not. The goal of walk forward isn’t to prove that your strategy is good. It’s to prove that your strategy is not known to be bad.

The process of walk forward testing is very simple. You identify a set of information that you want to use for your testing and optimization. Using a real example, right now it’s the beginning of 2014. So maybe you want to look and test data from 2011 through 2012. That would be your in sample data, and then your out of sample data might be all of 2013.

In order to conduct a walk forward test, you would test and analyze your strategy 2011-2012. Then, to determine if it’s “not known to be bad”, you then walk forward to 2103 to see review the performance.

What you’ve done is a blind test. You didn’t know what how the strategy would perform in 2013 when you tested it in 2011-2012. By putting it on a blind sample, you give it the opportunity to fail.

The reason so many traders put their faith in walk forward testing is because it’s the absolute best tool to identify weaknesses in your optimization. When you’re testing a strategy, it is very likely that you’ve overfit to past opportunities.

Self feedback loops in the current market

Let me give you an example. In the current markets, a lot of traders have been banging gold on the market open where every day at market open., they sell as much gold as they possibly can. Sometimes it’s several multiples of the annual production in a span of a few minutes. What you see is an absolute freefall for five or ten minutes. That state persists for days at a time. But that doesn’t last forever. When enough traders start seeing that people bang gold on the open, they start doing the same thing.

Effectively, whoever wants gold to falloff on the market open has taught other traders to do that trade for them. As people expect gold to fall in the first five minutes of the open, they then change their behavior. Some try to jump on banging the open and go short.

Others start modifying their behavior. They notice that gold free falls for five minutes. Then, all of a sudden it stops, and more than like it reverts to the mean. They’ll start changing their tack and buying after so many minutes have elapsed from the open. They expect that the heavy volume that preceded the selling will eventually return to normal. As people change their behavior, other people respond in kind.

If enough people start selling on the open and then buying on the open five minutes later, you can see that a pattern is forming where one person responds to the actions of another. It’s a self feedback loop where the state that was working for the first couple of days no longer works in the future.

If you can identify a strategy that is able to survive those conditions, and is able to survive conditions where you didn’t do any testing and optimization, you give yourself better odds of succeeding in the future. It means that not very many traders have clued into this trading opportunity that you’ve discovered.

The approach to to walk forward testing is the antidote to the problem known as curve fitting. Curve fitting is the ultimate woulda coulda shoulda strategy. It’s akin to opening a chart from yesterday and saying I would’ve bought here and I would’ve sold here, already knowing what transpired.

Of course you’re going to “make money” in that situation. You know with perfect information what the market did. In the future, you don’t know the perfect information. The goal of a strategy is to deal with that ambiguity.

Curve fitting means that you’ve fit everything so perfectly to past market conditions that when new situations inevitably arise, sort of akin to the phrase, “history doesn’t repeat itself, but it rhymes,” your strategy does the same thing.

You want a strategy that does well on past performance, but you’re not coming up with a strategy to make money on historical markets. The purpose of developing a strategy is to make money in future markets. When you’re backtesting, you’re trying to strike the balance between solid historical performance and, most importantly, making sure that that historical knowledge extrapolates to future performance. Your goal is to make money.

Rolling Walk Forward Optimization

Rolling walk forward optimization takes the walk forward idea and continuously improves the strategy by exposing it to new data. So let’s say that you have a twenty four month sample period. One way to go about it would be to optimize your strategy for a period of two months, then to walk it forward to the third month. You observe the behavior and you reoptimize for the second and third month, then walk it forward to the fourth month.

By doing so continuously, you eliminate the decay time of the strategy and give it a chance to adapt to ongoing market conditions. It is sort of the redheaded stepchild to machine learning. Experience and losses give the strategy the opportunity to improve and adjust to the market changes through walk forward optimization.

…you eliminate the decay time of the strategy and give it a chance to adapt to ongoing market conditions

Another important consideration for walk forward analysis is the degrees of freedom within a system. For example, let’s say that you are analyzing a moving averaage cross. You’re using two moving averages and use a fixed stoploss and take profit. That would give you four degrees freedom. The fast moving average is the first degree. The slow moving average is the second degree. The third is the stoploss and the fourth is the take profit.

The more degrees of freedom that you allow in a system vastly increases the chances 0f curve fitting your systems to historical data. The absolute best systems maintain twelve degrees of freedom or less. You want to find trading opportunities that have large numbers of trades and that offer performance that you find satisfactory.

Another element to consider in your optimization is what are you optimizing for. Most people focus on the absolute return. Returns are great, but most traders care much more about how they make their money instead of how much. Let me give you an example. If I had a system that made $25,000 last year, would you want it? Almost everybody says yes.

If I have a system that made $25,000 last year, but you had to lose to $15,000 before you made any money. Most people don’t want that system. What this means is that you care a lot more about the performance on a day-to-day basis rather than end result. The problem with optimization and even walk forward optimization is that you’re not necessarily focused on what you care about in the real world: the way that you’re making your money.

Most charting packages focused on the net outcome and that can cause some weaknesses in your system. If you’re range trading, what you’ve really done is cherry pick the results that are the least affected by substantial news. In effect, you’ve chosen the settings that have not yet been affected by fat tails.

If you’re trend trading, you’ve done the exact opposite. You intentionally pick the settings that maximize the fat tailes that have happened in the past. With trend trading strategies, you probably aren’t going to find consistent performance. Instead, what you’ll find is that the optimization frequently causes long, ongoing droughts of incessant drawdown. Then suddenly, almost out of nowhere, it finds a mega monster winner that returns several multiples of the drawdown that you experienced. This is fine for a hypothetical backtests, but in the real world where you’re suffering losses on a near daily basis, most traders can’t take the pain. The weakness I find with most optimizations is that they don’t look at the consistency of performance. A potential substitute for optimizing a strategy would be looking at the linear regression of the equity curve over time. The best equity curve has the strongest linear regression slope.

Popular charting packages that implement rolling walk forward optimization are Amibroker, TradeStation, Multicharts and NinjaTrader.

Walk forward optimization in NinjaTrader

Open the Strategy Analyzer from the Control Center. Click File / New / Strategy Analyzer.

Open the strategy analyzer in NinjaTrader

Left mouse click on an instrument or instrument list and right mouse click to bring up the right mouse click menu. Select the menu item Walk Forward. You can also click on the “w” icon in the Strategy Analyzer toolbar. If you prefer hot keys, you can also use CTRL + W. Lastly, you can also push the “W” icon at the top left of the Strategy Analyzer.
Select a strategy from the Strategy slide out menu
Set the Walk Forward properties (See the “Understanding Walk Forward properties” section below for property definitions) and press the OK button.

There are many ways to select walk forward optimization in NinjaTrader

The Walk Forward progress will be shown in the Status Bar of the Control Center.

NinjaTrader Screenshot

July 26, 2013 by Shaun Overton Leave a Comment

Taking a screenshot of a chart in NinjaTrader is a 3 step process. You don’t need any outside software or special skills.

Open a chart
Right click. Select “Image” from the menu, then “Save As”
Select the location where you’d like to save the file

Right click on the chart to take a screenshot.

I always recommend saving files to the desktop if you’re not very savvy with computers. You can find them easily.

Build a Trading Robot

April 10, 2013 by Shaun Overton 1 Comment

I talk a lot about the importance of building your trading plan. The same thing applies for building a trading robot or an expert advisor.

Most people approach EA development as digging frantically looking for huge gold nuggets. That approach is a good way to waste a lot of time and money.

Most people dive into the process without considering the details. The purpose of this post is to slow everything down so that you can develop some sort of business plan for your trading.

Steps to building a trading robot

Three long, difficult steps are involved with deploying a trading robot. You first need to obtain data for testing. A good strategy, once discovered, then needs to be programmed to automatically place trades. Finally, you need to select a broker to execute the orders in the live market.

The importance of data

A good number of traders brag about their best trades. One trade gave them a million dollar profit and other such and such.

What newbie system traders don’t realize is the tedious process of how these people reached their status. Knowing whether or not a system has an edge or not entails researching your idea using historical price data. Here are some tips that you can use when looking for data:

• You need to get data, you need to analyze it, and then you need to trade it. It is really simple on the surface, but when you are trying to go about this, every step creates huge obstacles. The easiest work around is to use the trading platforms listed at the bottom of the article.

• If you are looking for free options, you might go look somewhere like Yahoo! Finance where you can get data on lots of stuff that is mostly end of day prices. It’s no good for high frequency. You can get some options data, you can get forex and you can get some indices and some futures data.

• The reason why data quality is important is when we go on to analyze and come up with potential ways that you might want to trade algorithmically, if you have garbage data, you have a garbage analysis. Be very careful about the data that you decide to accept.

Problems gathering data

Now that we learned how important data is, we now discuss the other side of the picture in data gathering which is its disadvantages.

• Unreliable. Sometimes it’s just wrong. Sometimes there are duplicate entries. Sometimes there are gaps in the data that are unexplained. And if you don’t really know what you’re looking for, what kind of problems there might be in the data, you can come up with some weird discoveries.

• Delays. Technology bridges the information gap. But sometimes, due to time differences and the most annoying part when your broker or you are having problems on your internet connection at times, delays are inevitable.

• Clutter. There are a gazillion sources of information on the internet. This equates to a huge amount of data. Different brokers have their own set of websites and blogs which displays various analyses on a single instrument. It will now be a problem on what data will you consider credible and useful in your trade. Sometimes, analysts just anticipate most especially technical analysts posting price forecasts and so on. Those who waits for news like Unemployment rate has their own views on figures that will show up prior and during the announcement which basically creates clutter.

The Trading Platform

When you trade online, a trading platform is like your playing field. It is the software through which you manage your trades when you open, close or set limits or stops. Usually, a trading platform is provided by the broker.

There are platforms, APIs and all sorts of different companies that offer data and trading capabilities all in the same product. The advantage to those types of software is that it makes your life a lot easier because if you go about this on your own it’s a monumental task.

A lot of people that like to play with R, and you can just custom program your own research platform. Matlab and R are the most common tools in this category. The problem is that you have to build the trading components entirely on your own.

Here are some popular platforms MetaTrader, NinjaTrader, ThinkorSwim and Multicharts. All these platforms use their own different language and solve the data problem. They also include the ability to trade automatically. Most of the heavy lifting has already been done.

MetaTrader uses a custom language called MQL4, which is really a C Scripting Language. NinjaTrader uses C# .NET 3.5. Trade Station and Multicharts is a language called EasyLanguage, which for programmers will probably bore you to tears. To understand more about these platforms, you can check different brokers and their offerings.

Trading Platforms and data

We’ve covered the languages, the markets they cover and then, the other problem is really data. These platforms offer very different set-ups and they all handle the data problem differently.

MetaTrader

MetaTrader is kind of like the AK-47 of Trade Platforms. You download it and it doesn’t matter how novice or not good with computers you are. You will have a hard time not figuring out how to work the platform. It is simple, it’s very friendly. It’s also not sophisticated.

If you’re trying to do something sophisticated, that’s probably not the place to be. And for your analysis, you have to be very careful, because when you download MetaTrader, charts just pop up. You think, “Oh this is great, I got my data and everything looks good”, but the problem is that most of the time the data is junk. You can’t actually rely on it and do any serious analysis.

Getting the data and getting it formatted to MetaTrader is the most convoluted, difficult problem that traders face everyday. This is the platform that we deal with and everybody has problems with their backtesting and getting familiar.

This is good enough if you want to trade every couple of minutes and you’re not super execution sensitive and you are just trying to get something out cheaply. If you try every 4 hours and you trade 3 currencies, MetaTrader is fantastic for that.

But if you try to day trade or trade 20 different currencies at the same time, that’s a disaster, because MT4 isn’t multi-threaded. Every time you push an order into the market, MT4 can only handle it one at a time.

If you have 5 orders firing off together, this one has to finish, and you have all the latency in the middle where they connect and then bounce and the trade confirms okay. Now you repeat the process four more times to get all 5 trades filled. If you’re pushing too many orders through, MT4 will choke.

NinjaTrader

You have to find somebody to give you the data. There are some paid options, there are some, there’s one that’s free called Kinetick, and they give you end of day data. Brokerages also provide limited historical data. The quantity varies substantially from broker to broker.

The problem when you’re programming all this stuff and you’re trading in the live market, if you program to a broker specific platform, you probably spend 4 to 6 months developing and testing it and getting it working, and a lot of money and time. If you go to start trading live and you’re not happy with the broker, too bad. You married them.

What NinjaTrader did is, let’s say that there are Brokers A, B and C. NinjaTrader sits on top to bridge everything together.

NinjaTrader is an API shoved on top of multiple broker APIs so that you can write your strategy in NinjaTrader. They’ve done all the integration with every broker partner they have.

It’s a different way to handle the same problem as MetaTrader. MetaTrader just goes to these brokers and say “You should use our platform”. If you developed in MetaTrader, you can go switch on a whim.

If you have NinjaTrader, you can go switch on a whim. The difference is that you don’t just have NinjaTrader. You have to download the broker platform, then you download NinjaTrader. Then, you get everything hooked up and make sure everything plays nicely together.

There’s a steep learning curve with getting this all set-up to the point where you can actually download historical data and start trading with your broker. Once you have the data set-up, though, NinjaTrader is awesome.

TradeStation and Multicharts

TradeStation and Multicharts offer the same quality of analytics as NinjaTrader. They are easier to develop in. Obtaining data quickly makes these platforms easier for testing trading robots.

If you program with TradeStation, you trade with Trade Station. You can’t go trade with anybody else. However, if you’re ever unsatisfied with the broker because they slip you or because they charge bad commissions or whatever goes wrong with your trading, you don’t have any alternative. You’d have to move to MultiCharts or to redevelop the programming completely from scratch in another platform.

The Broker

Remember that your broker is your trading partner. It provides you the platform, data and most importantly the access in the market. Brokers are also service providers, and traders are their customers. It’s common for a customer to encounter some difficulties regarding the provider. There are two common reasons why a customer sometimes feels dissatisfied.

• First, bad service. When you are having difficulties opening your platform to execute a trade, you call on a customer service representative. And just like any other companies, a structured way of handling complaints will be given to you. Sometimes they can’t just give you what you need because they do not want to understand what you need.

• The second reason is bad execution. The broker is the market maker. They set the buying and selling prices at every single second. When you place a trade, they can present prices that are favorable to them. As a trader, when you know that the trend is on your side, sometimes you do not mind trading few pips higher or lower than your order. So you grab the offer. Until you realize that the broker made some hefty spread on your trades. This affects the trader emotionally and emotions should be out of trading. Emotions spoil strategies nut as a human being, it’s natural to feel upset.

Conclusion

Building your own trading robot is not as easy as ABC. It entails allot of effort in research, trying and testing trading signals and detaching every trade from your emotions. All of these steps are built on the foundation of your trading experience.

Remember that building a trading robot is anchored with the basic steps:
• Gathering and identifying the data you will use. Be very keen on what data to retain and what data will go directly to the bin. Not all data fed by your broker, articles written by analysts or data given to you by someone you know are reliable. Make sure to get the right data, analyze the data and trade using the data.

• A trading platform you are comfortable with. Before you start making your own trading robot, feel free to try different trading platforms. It’s like trying on some new pair of shoes before buying them. Do not go for something very complicated and you cannot even decode some basic functions. Remember that your platform is your playing field. It is more fun to trade through a platform you know how to operate than to get the most technical one and read through the help section while some other traders are gaining fast from a current trend.

• A reliable broker that will assist you on your trades. Your broker is your partner. It gives you data, platform and sometimes analysis in real time. Make sure to choose the broker that can give you what you demand and what you need.

If you will follow some of the above tips, you are on the right track on making your own trading robot.

Import a NinjaTrader Strategy

February 15, 2013 by Shaun Overton 17 Comments

It’s easy to import a NinjaTrader strategy or indicator. Popular places for finding a new strategy or indicator include the NinjaTrader forums, Big Mike’s Trading and passing files amongst friends.

The first step is to go to the Control Center, the main screen in NinjaTrader. Click on File, then select Utilities. A new menu will fly out to the right. Select the top option, which is Import NinjaScript.

import ninjatrader strategy or indicator

A screenshot from the NinjaTrader Control Center. Client File, Utilities, Import NinjaScript to bring a new strategy or indicator into NinjaTrader.

Zip Files

Once selected, the program asks you to locate a zip file. You’re probably used to programs using their own funny extensions. MetaTrader, for example, uses .mq4 files for its strategies.

NinjaTrader sticks to using .zip files for both a strategy and indicator. The process to import a NinjaTrader strategy is exactly the same as an indicator.

Select the zip file wherever you saved it. The desktop is always a safe place to download files. If you downloaded an indicator but aren’t sure where you put it, then check the downloads folder. You can find that in C:\Users\YOUR USER NAME\Downloads\. Internet Explorer, Firefox and Chrome all download files to this folder as part of their default settings.

The nice thing about NinjaTrader using zip files is that you don’t see all sorts of funny looking icons. The disadvantage is that it’s impossible to tell whether the file is an NT7 strategy or indicator, or if it’s something entirely different.

If you see a zip file and can’t tell whether or not it belongs to NinjaTrader, the simplest way to find out is to open the file. Double click it.

Exported NinjaTrader zip files contain Info.xml and potentially two folders: Indicator and/or Strategy

When the folder opens, you should see a file called Info.xml. The number of folders that you’ll see depends on the file type. If the zip file only contains an indicator, then only the Indicator folder appears. A zip file containing a strategy will more than likely show both the Indicator and Strategy folders.

Conclusion

Common errors that pop up when importing files are compile errors. Read through that article if you get stuck trying to import a new file.

If you run into any frustrating errors during the process, then please leave a comment on the blog below. I’d be glad to help you out.

Order Retry Logic

January 21, 2013 by Shaun Overton 5 Comments

The difference between a backtest and live trading is that nothing ever goes wrong on a backtest. If a strategy traded correctly for EURUSD in 2011 yesterday, you know that the same test will work properly today.

The backtester is not designed nor is it able to catch the types of problems that occur in live trading. I made an effort to list common issues by platform and to detail the most common solutions.

MetaTrader

Most older EAs attached stops or take profits to its orders. The NFA rules from around 2009 require all forex trades to enter without any exit conditions attached.

The rule created a nightmare for US MT4 brokers. They were forced to go back, modify MetaTrader and disallow tickets with a stop or limit.

The solution is to confirm correct execution of a trade. Once the trade enters, only then should the expert advisor attempt to add the stop or take profit.

The rule is unfortunate as it requires additional communication time. The process slows order execution, which can cause the trade context is busy error.

These are not words that you want to see when placing live trades

Our Expert Advisor Programming Template

Kamal O. asked on Friday about the words RETRYCOUNT and RETRYDELAY in our EA template code. Those 2 words are critical for maintaining code that handles all possible situations, or at least 99.9% of them.

We set them by default to 10 attempts and 1,000 milliseconds, respectively. That is, an order will attempt to order up to 10 separate times. Each attempt will wait at least 1,000 milliseconds (1 second) before making another attempt.

In between attempts, we also check to see if the trade context is busy. If it is not, then we proceed. Otherwise, the code waits for an opportunity to submit the new order up to the specified maximum.

The same retry ad wait logic also applies to submitting stop losses and take profits. One of the most horrifying real world events that a trader can discover is an open trade with no exit conditions attached. Such things really do happen.

The same retry logic is in place in our template code to dramatically reduce the chance of that occurring. We do this for every expert advisor that we program. If you have an expert advisor that does not include retry logic, then contact us about making your EA’s source code more robust.

NinjaTrader

NinjaTrader largely handles errors and problems for the programmer. However, there are common situations where NinjaTrader’s solution frustrates the user. Disabling strategies and closing all trades whenever an overfill occurs comes to mind.

Anything but the simplest trading strategies are better handled using an unmanaged approach. Managed approaches using pending orders will almost always create a need for rewriting a strategy.

One of the most common reasons NinjaTrader users contract us relates to something going wrong with their live trading. The best way to avoid programming something twice is by making the code ready for real world trading on the first attempt.

Automated Trading Part II

January 7, 2013 by Shaun Overton 1 Comment

The second part in Nathan’s interview series with me focuses on the role of high frequency trading in the markets and testing trading strategies. If you missed the first automated trading interview in the series, you can read it here.

(Nathan):
Do you have any specific thoughts or opinions on HFT (High Frequency Trading)? This is such a hot topic among traders and I would imagine you have a unique insight into algo trading in general.

Do you see machines eventually replacing the “human” trader or could HFT eventually get banned from the markets? At least for day traders it seems to give quite an unfair advantage to the HFT camp for executions?

(Shaun):
There is a huge difference between algorithmic trading and HFT. HFT is obviously automated due to the speeds involved, but that does not imply that all automated trading is HFT. It’s only a subgroup.

(Nathan):
Sure, there are plenty of automated systems that are not HFT. I bring it up under the context that HFT uses deceptive algorithms for their order posting tactics.

(Shaun):
HFT is uniformly destructive to capital markets and their purpose. It nickels and dimes investors and traders through market manipulation. Just last week Nanex detected a single organization that pushed through 4% of all the order flow on US equities quotes. Even worse, not a single one of the orders executed. Posting orders without the intent to trade is blatantly illegal.

The other negative consequence of HFT comes from the rebates that the dark pools and exchanges pay to “liquidity providers,” which are really the HFT bots. The arrangement tangibly alters the motivation for participating in markets. Rather than investing or even speculating on price, the HFT algos generally do not care about market movements. They just want the liquidity rebate.

(Nathan):
This to me is the bigger issue. The whole arrangement is shady and as you said it alters the motivation for market participation. What steps or changes do you recommend?

(Shaun):
The Market Ticker blog is one of my favorites on that subject. Karl Denninger advocates a regulatory rule of a two second minimum order time. I support the idea. Nobody can plausibly claim that an order placed for such a short duration is for any trading purpose. If an order is not intended to be filled, it should be not permitted.

(Nathan):
I cannot argue with that logic. Back to testing, how important is accounting for commissions and slippage to the integrity of any back-testing data in your opinion? To me, as you go down the scale from longer term trading to day-trading the importance grows exponentially.

(Shaun):
I fully agree. The consequences of trading costs pile up with increased frequency. Shorter time frames multiply the frequency, which as you pointed out, grows exponentially.

My personal preference is to skip trading costs and commissions on short time frames so that I can obtain a sufficient sample size for my analysis. I do not foresee myself ever trading on one minute charts, but I almost always use one minute tests to analyze randomness within a strategy. Unlike most systems developers, the profitability of a system is a backseat concern.

I read a newsletter this morning written by a multi-million dollar businessman. He concluded today’s article saying that if you start a business to make a lot of money, you’ll more than likely fail. You have to excel at providing a quality product and service in order to succeed over the long run. When the inherent business excels, only then does the long term money follow.

Trading is a business in precisely the same sense. Most traders rush through the system development process to spit out quick profits. They rarely, if ever, consider a strategy’s performance over a lengthy period of time. Everything is about the here and now. Additionally, good systems frequently lose money. You need something more in the toolkit besides the random scorecard of profit and loss.

(Nathan):
Good systems do have losing periods, yet many traders seem to be convinced there is a “holy grail” approach out there that will buck this fact. If some of the most successful traders ever have posted losing periods (or even years) and have been around for 20+ years it seems hard to fathom beating their performance from day 1.

Regarding testing platforms, you were one of the first people that really explored the issues with back-testing Forex – can you provide more detail on the problems for those that are interested in developing and back-testing a system for FX (MetaTrader platform)?

(Shaun):
You’re opening a can of worms on this one. MetaTrader is hands down the worst platform available for backtesting. The data is notoriously unreliable. Even when good data is at hand, the instructions for importing it and turning it into something usable fill a dozen pages of instructions.

You’re much better off doing real analysis in NinjaTrader, TradeStation or MultiCharts. The metrics are vastly superior and require a tiny fraction of the effort. I still think that MetaTrader is sufficient for live trading most strategies.

(Nathan):
I am a huge fan of live testing/trading alternate strategies. One of my biggest “A Ha” moments came during live testing alternate exit strategies. I traded my account with my original exit approach but also demo traded alternate exit strategies in real time. There is value gained that you don’t always get from a back-testing print out. How do you compensate for slippage when testing a strategy?

(Shaun):
Forex is thankfully unique in that it doesn’t come with unique problems other than rollover. The markets are the most liquid in the world. As a retail forex trader looking at charts longer than five minutes, you can generally assume that the historical prices are reasonably reflective of executable prices for the strategy.

I compensate for slippage and bad ticks by doubling my expected transaction costs. For example, I pay 1.5 pip spread on the EURUSD. When I test a strategy, I demand that it must hold up on 3 pips transaction cost on every trade.

(Nathan):
Before we wrap this up, are there any specific strategies or common parameters that you have noticed in systems that make it? We both know how small of a percentage of traders become successful, but as it relates to mechanical systems are there recurring themes for those that are profitable?

(Shaun):

No, there are no recurring themes that I see. The lowest common denominator is that they do not overtrade and that they use low leverage. Other than those two items, each successful strategy differs substantially from all the others.

The most important ingredient in system development is the developer. I have yet to program a successful trading system for someone without years of full time trading experience under their belt. You have to go through the school of hard knocks if you’re going to make it. Almost all of us are too stubborn to listen to good advice.

(Nathan):
Shaun, I can’t thank you enough for providing such honest responses and sharing your insight. If you are interested in learning more or considering coding your system, go to MetaTrader Programming for more information.

Group Trading Strategy

January 4, 2013 by Shaun Overton 42 Comments

Francis D. from Australia likes to bounce different EA ideas off of me. He mentioned in the latest emails a fear of getting whipsawed from a signal that is either long or short.

This type of problem occurs all the time. I first encountered it with a simple strategy that fades price crosses over the moving average. Whenever the price crosses and closes below the moving average, go long. Shorts follow the same rules. It’s the kind of stupid simple range trading strategy that I always advocate.

Buy when price crosses below the 200 SMA. Sell and go short when the price crosses back above

The example above highlights the same thing that Francis complains about. The price floats around the moving average without going anywhere.

Random Outcomes

Trades based on the price crossing above the SMA come out near breakeven. Winners occur approximately 66% of the time and are one third the size of losers. Such a strategy neither makes nor loses money when ignoring trading costs.

The vast majority of the winners in the strategy are teeny tiny. The strategy encounters its maximum opportunity whenever it is closest to the SMA. The further away it goes, the more likely it keeps going the wrong way.

Flipping the trades still yields a random outcome. The only difference is that winners drop to 33% accuracy, but the average winner is now 2:1.

The Birth of a Trading Strategy

I always wonder how a scaling strategy might affect the outcome. If the strategy is most likely to win when the opportunity is smallest, what happens when the strategy attempts to reduce its position size as the market moves adversely?

Alternatively, what happens if it takes a pass on all the small winners and scales into positions? Yes, it will scale into losers, but it should also make resulting winners bigger. The question then becomes how to decide the rate of scaling and when to bail out of losers.

Thank you, Francis! A new blog series is born. Now that I’ve decided to scale into trades, I need to choose how and when to do it.

Nothing insightful or special jumped to mind. I’m a visual person, so I spent the better part of this afternoon creating the little chart in Excel using NinjaTrader. What starts out as a simple project always grows on itself. It took nearly 4 hours to get the information and formatting correct.

The graphs displays the percent distance of the price from the SMA against frequency (i.e., how often is the price this far from the 200 SMA?)

I care about scaling into trades as the price moves further from the 200 SMA. My instinct from looking at the graph above says I should focus on the inflection point. The curve forms a nice bend around 0.3% away from the SMA. Maybe I can start buying at the inflection point until I get to 0.6% or so.

What do you think I should do?

After-thoughts

This series eventually led to a profitable trading strategy. If you’d like to read through the journey, then I suggest reading the articles sequentially

Selecting an appropriate time frame
A research plan
An annoying surprise in the initial backtests
An attempt at range trading
Range trading results
The moving average envelope scalper

MetaStock First Impression

December 21, 2012 by Shaun Overton 2 Comments

It’s not that developing for MetaStock is hard, it’s just different. I am used to working with MetaTrader and more recently NinjaTrader. MetaStock offers all of the great features one would look for in a trading platform. Navigating around is a bit difficult, though. The biggest obstacle for me is how differently MetaStock handles tasks when compared to other platforms

Of course I recognize comparing anything to MetaTrader isn’t fair in some respects. MetaTrader is a much simpler platform to be sure. MetaStock is significantly more robust in its features and abilities. Where MetaTrader instantly opens charts for you right after installation, MetaStock opens to a kind of passive mode. The platform does not display anything until the user requests it.

When I develop for MT4, I only have to locate and copy an EA for delivery to our customers. That’s not the case with MetaStock. Thankfully MetaStock, like NinjaTrader, has an import/export tool to help. In MetaStock you simply run the Organizer and the wizard will guide you through all of the details of creating an export for backup or delivery to another user.

Each of the major trading platforms has its own set of unique features and each has its own quirks. As noted earlier MetaTrader feels like it’s quicker and easier to get started. The learning curve is short. However, in comparison to NinjaTrader and MetaStock, MT4’s feature set is limited. The best analogy that I can think of is a bike doesn’t take as much skill and practice as driving a car on the highway.

My biggest beef with MetaStock at this point would be the language used for the indicators and EAs. Compared to MQL and C#, the language feels limited and somewhat clunky. It requires DLL programming much more often than other platforms.

NinjaTrader Backtest

December 19, 2012 by Shaun Overton Leave a Comment

Running backtests in NinjaTrader is relatively straight forward, once you learn the ropes. NinjaTrader uses a special window, called the Strategy Analyzer, to run all backtests and optimizations. The first step to finding this window is to click on File \ New \ Strategy Analyzer.

When the Strategy Analyzer opens, the window is divided into 3 vertical panes. The left window allows the user to select the instrument to test. The middle window contains the statistics and backtest information. The final window on the right is not visible; it slides out when you put the mouse over the word “Backtest” in the top right corner.

Finding the symbol that you want to test

It’s important to emphasize that you need a data connection before diving into any of this. NinjaTrader is not a standalone platform. The information that you would like to test is not available by default. Instead, NinjaTrader uses the Account Connection to go out to the broker or data provider to obtain the historical data. Everything below is a moot point if you have not already downloaded historical data and/or do not have an open Account Connection.

The left pane lists all lists that have been created using the Instrument Manager. NinjaTrader supplies lists of the most common instruments available: the DOW, S&P 500 and major forex pairs. Testing an instrument like a junior gold mining stock or something less common requires creating a new list in the Instrument Manager.

One cool feature is that you can test multiple instruments at the same time. If you want to view your strategy’s historical performance on all S&P 500 stocks, then select the list name. It is not necessary to select them individually. Every stock in the list will be included in the analysis.

If this all sounds terribly confusing (Account Connection, Instrument Manager, etc), you’re right. It’s very confusing, which is why the learning curve for NinjaTrader is so steep. It takes a long time to figure out how all of these pieces fit together. Even my staff of professional programmers experienced an ugly few weeks when I started training them. If programmers have trouble figuring out how to use the software, you don’t have to feel bad about your own difficulties.

Strategy options

The right pane opens when the mouse appears over the word “Backtest” on the far right. Within it, the menu contains multiple sections that allows the user to define the strategy. Options near the top under “Parameters” are the inputs or variables that the strategy uses. Common examples include the number of shares to trade, the stop loss distance and others.

The Data series section controls the chart period. Say, for example, that you would live to backtest AAPL on 5 minute charts. The necessary steps are:

Select APPL in the left pane
Choose Last for Price Based on
The type is Minute
The value is 5, which here stands for 5 minute charts

Time Frame controls the period over which the backtest runs. Running an AAPL strategy for 2011 would cause a trade to enter 1/1/2011 for the Start Date and 12/31/2011 for the End Date.

The remaining sections largely do not apply. When they do cause problems, most stem from the Order Handling section. If you want to trade several signals in the same direction, then the option for Entries per direction must change from 1 to a predefined maximum.

Other sections

Anyone with trading experience in other platforms will find the middle pane intuitive, especially TradeStation users. Tabs at the top of the middle pane include the Summary and Graphs. Most of my personal trading analysis concentrates on these two tabs. The others are helpful for more detail oriented folks.

The Strategy Analyzer contains a number of buttons at the top left of the screen. There are four which I find to be useful.

The floppy disc icon stands for saving a file. When the backtest completes, and especially if it takes several minutes to run, then option to save results may save a significant amount of time. If you have multiple backtests and would like to share them, the only way to do so is to send someone all of your backtests. NT stores all of its saved results in a local database. The exact location is Documents\NinjaTrader 7\db\NinjaTrader.sdf. Sending your friends or colleagues this file includes all saved backtests to date. Make sure that the recipient backs up his own NinjaTrader.sdf file before using yours. Otherwise, the information will be lost.

Right clicking in the middle pane allows the user to export the summary grid to an Excel file. Although it does not look as convenient as the NinjaTrader format, the above problem highlights the need for sending and saving individual test results.

NinjaTrader doesn’t give the b, o and w enough visual importance in my opinion. The buttons are tiny, yet they control the most important feature of the backtest – the type of test to run. Do you want to run a backtest, optimization or a walk-forward optimization? These little buttons control the type of test run.