Algorithmic and Mechanical Forex Strategies | OneStepRemoved

Automated Trading Part II

by 1 Comment

The second part in Nathan’s interview series with me focuses on the role of high frequency trading in the markets and testing trading strategies. If you missed the first automated trading interview in the series, you can read it here.

Nathan Orange(Nathan):
Do you have any specific thoughts or opinions on HFT (High Frequency Trading)? This is such a hot topic among traders and I would imagine you have a unique insight into algo trading in general.

Do you see machines eventually replacing the “human” trader or could HFT eventually get banned from the markets? At least for day traders it seems to give quite an unfair advantage to the HFT camp for executions?

Shaun Overton(Shaun):
There is a huge difference between algorithmic trading and HFT. HFT is obviously automated due to the speeds involved, but that does not imply that all automated trading is HFT. It’s only a subgroup.

 

Nathan Orange(Nathan):
Sure, there are plenty of automated systems that are not HFT. I bring it up under the context that HFT uses deceptive algorithms for their order posting tactics.

 

 

Shaun Overton(Shaun):
HFT is uniformly destructive to capital markets and their purpose. It nickels and dimes investors and traders through market manipulation. Just last week Nanex detected a single organization that pushed through 4% of all the order flow on US equities quotes. Even worse, not a single one of the orders executed. Posting orders without the intent to trade is blatantly illegal.

The other negative consequence of HFT comes from the rebates that the dark pools and exchanges pay to “liquidity providers,” which are really the HFT bots. The arrangement tangibly alters the motivation for participating in markets. Rather than investing or even speculating on price, the HFT algos generally do not care about market movements. They just want the liquidity rebate.

Nathan Orange(Nathan):
This to me is the bigger issue. The whole arrangement is shady and as you said it alters the motivation for market participation. What steps or changes do you recommend?

 

 

Shaun Overton(Shaun):
The Market Ticker blog is one of my favorites on that subject. Karl Denninger advocates a regulatory rule of a two second minimum order time. I support the idea. Nobody can plausibly claim that an order placed for such a short duration is for any trading purpose. If an order is not intended to be filled, it should be not permitted.

 

Nathan Orange(Nathan):
I cannot argue with that logic. Back to testing, how important is accounting for commissions and slippage to the integrity of any back-testing data in your opinion? To me, as you go down the scale from longer term trading to day-trading the importance grows exponentially.

 

Shaun Overton(Shaun):
I fully agree. The consequences of trading costs pile up with increased frequency. Shorter time frames multiply the frequency, which as you pointed out, grows exponentially.

My personal preference is to skip trading costs and commissions on short time frames so that I can obtain a sufficient sample size for my analysis. I do not foresee myself ever trading on one minute charts, but I almost always use one minute tests to analyze randomness within a strategy. Unlike most systems developers, the profitability of a system is a backseat concern.

I read a newsletter this morning written by a multi-million dollar businessman. He concluded today’s article saying that if you start a business to make a lot of money, you’ll more than likely fail. You have to excel at providing a quality product and service in order to succeed over the long run. When the inherent business excels, only then does the long term money follow.

Trading is a business in precisely the same sense. Most traders rush through the system development process to spit out quick profits. They rarely, if ever, consider a strategy’s performance over a lengthy period of time. Everything is about the here and now. Additionally, good systems frequently lose money. You need something more in the toolkit besides the random scorecard of profit and loss.

Nathan Orange(Nathan):
Good systems do have losing periods, yet many traders seem to be convinced there is a “holy grail” approach out there that will buck this fact. If some of the most successful traders ever have posted losing periods (or even years) and have been around for 20+ years it seems hard to fathom beating their performance from day 1.

Regarding testing platforms, you were one of the first people that really explored the issues with back-testing Forex – can you provide more detail on the problems for those that are interested in developing and back-testing a system for FX (MetaTrader platform)?

Shaun Overton(Shaun):
You’re opening a can of worms on this one. MetaTrader is hands down the worst platform available for backtesting. The data is notoriously unreliable. Even when good data is at hand, the instructions for importing it and turning it into something usable fill a dozen pages of instructions.

You’re much better off doing real analysis in NinjaTrader, TradeStation or MultiCharts. The metrics are vastly superior and require a tiny fraction of the effort. I still think that MetaTrader is sufficient for live trading most strategies.

Nathan Orange(Nathan):
I am a huge fan of live testing/trading alternate strategies. One of my biggest “A Ha” moments came during live testing alternate exit strategies. I traded my account with my original exit approach but also demo traded alternate exit strategies in real time. There is value gained that you don’t always get from a back-testing print out. How do you compensate for slippage when testing a strategy?

 

Shaun Overton(Shaun):
Forex is thankfully unique in that it doesn’t come with unique problems other than rollover. The markets are the most liquid in the world. As a retail forex trader looking at charts longer than five minutes, you can generally assume that the historical prices are reasonably reflective of executable prices for the strategy.

I compensate for slippage and bad ticks by doubling my expected transaction costs. For example, I pay 1.5 pip spread on the EURUSD. When I test a strategy, I demand that it must hold up on 3 pips transaction cost on every trade.

Nathan Orange(Nathan):
Before we wrap this up, are there any specific strategies or common parameters that you have noticed in systems that make it? We both know how small of a percentage of traders become successful, but as it relates to mechanical systems are there recurring themes for those that are profitable?

 

Shaun Overton(Shaun):

No, there are no recurring themes that I see. The lowest common denominator is that they do not overtrade and that they use low leverage. Other than those two items, each successful strategy differs substantially from all the others.

The most important ingredient in system development is the developer. I have yet to program a successful trading system for someone without years of full time trading experience under their belt. You have to go through the school of hard knocks if you’re going to make it. Almost all of us are too stubborn to listen to good advice.

Nathan Orange(Nathan):
Shaun, I can’t thank you enough for providing such honest responses and sharing your insight. If you are interested in learning more or considering coding your system, go to MetaTrader Programming for more information.

NinjaTrader Backtest

by Leave a Comment

Running backtests in NinjaTrader is relatively straight forward, once you learn the ropes. NinjaTrader uses a special window, called the Strategy Analyzer, to run all backtests and optimizations. The first step to finding this window is to click on File \ New \ Strategy Analyzer.

When the Strategy Analyzer opens, the window is divided into 3 vertical panes. The left window allows the user to select the instrument to test. The middle window contains the statistics and backtest information. The final window on the right is not visible; it slides out when you put the mouse over the word “Backtest” in the top right corner.

Finding the symbol that you want to test

It’s important to emphasize that you need a data connection before diving into any of this. NinjaTrader is not a standalone platform. The information that you would like to test is not available by default. Instead, NinjaTrader uses the Account Connection to go out to the broker or data provider to obtain the historical data. Everything below is a moot point if you have not already downloaded historical data and/or do not have an open Account Connection.

The left pane lists all lists that have been created using the Instrument Manager. NinjaTrader supplies lists of the most common instruments available: the DOW, S&P 500 and major forex pairs. Testing an instrument like a junior gold mining stock or something less common requires creating a new list in the Instrument Manager.

One cool feature is that you can test multiple instruments at the same time. If you want to view your strategy’s historical performance on all S&P 500 stocks, then select the list name. It is not necessary to select them individually. Every stock in the list will be included in the analysis.

If this all sounds terribly confusing (Account Connection, Instrument Manager, etc), you’re right. It’s very confusing, which is why the learning curve for NinjaTrader is so steep. It takes a long time to figure out how all of these pieces fit together. Even my staff of professional programmers experienced an ugly few weeks when I started training them. If programmers have trouble figuring out how to use the software, you don’t have to feel bad about your own difficulties.

Strategy options

The right pane opens when the mouse appears over the word “Backtest” on the far right. Within it, the menu contains multiple sections that allows the user to define the strategy. Options near the top under “Parameters” are the inputs or variables that the strategy uses. Common examples include the number of shares to trade, the stop loss distance and others.

The Data series section controls the chart period. Say, for example, that you would live to backtest AAPL on 5 minute charts. The necessary steps are:

  1. Select APPL in the left pane
  2. Choose Last for Price Based on
  3. The type is Minute
  4. The value is 5, which here stands for 5 minute charts

Time Frame controls the period over which the backtest runs. Running an AAPL strategy for 2011 would cause a trade to enter 1/1/2011 for the Start Date and 12/31/2011 for the End Date.

The remaining sections largely do not apply. When they do cause problems, most stem from the Order Handling section. If you want to trade several signals in the same direction, then the option for Entries per direction must change from 1 to a predefined maximum.

Other sections

Anyone with trading experience in other platforms will find the middle pane intuitive, especially TradeStation users. Tabs at the top of the middle pane include the Summary and Graphs. Most of my personal trading analysis concentrates on these two tabs. The others are helpful for more detail oriented folks.

The Strategy Analyzer contains a number of buttons at the top left of the screen. There are four which I find to be useful.

The floppy disc icon stands for saving a file. When the backtest completes, and especially if it takes several minutes to run, then option to save results may save a significant amount of time. If you have multiple backtests and would like to share them, the only way to do so is to send someone all of your backtests. NT stores all of its saved results in a local database. The exact location is Documents\NinjaTrader 7\db\NinjaTrader.sdf. Sending your friends or colleagues this file includes all saved backtests to date. Make sure that the recipient backs up his own NinjaTrader.sdf file before using yours. Otherwise, the information will be lost.

Right clicking in the middle pane allows the user to export the summary grid to an Excel file. Although it does not look as convenient as the NinjaTrader format, the above problem highlights the need for sending and saving individual test results.

NinjaTrader doesn’t give the b, o and w enough visual importance in my opinion. The buttons are tiny, yet they control the most important feature of the backtest – the type of test to run. Do you want to run a backtest, optimization or a walk-forward optimization? These little buttons control the type of test run.

Backtest Spread

by 1 Comment

A common question among MetaTrader users comes from backtest results that appear to change for no reason. Without changing any settings like inputs, the currency pair, or the date ranges used, the backtest results change with every click of the start button. As I mentioned in previous posts on MetaTrader backtesting, MT4 assumes the spread on historical data rather than using recorded data. All charts in the software, and all trading software for that matter, display bid data. The ask is not recorded.

Trading incurs spread costs on every execution. MetaTrader has no choice but to make assumptions trading costs because the historical ask data is unavailable. The way it does this is by taking the current spread of the currency pair under study.

If you find this especially annoying, the easiest way to lower your frustration is by disconnecting from the broker. Severing the connection prevents the assumed spread from updating. The backtest results will stop jumping around.

Renko Backtests

by 9 Comments

Exotic bar types, as NinjaTrader likes to call them, create unique challenges when backtesting strategies. The primary problems is that the backtests are usually bogus. The trader often has no idea that the profitable backtest calculated from errant data.

Renko bars form based on the order of incoming ticks to create specific box sizes. Say, for example, that a trader creates a box size of 5 pips. If the price rises 5 pips from the close price of the last Renko bar, then the chart creates a new bar 5 pips tall. Every 5 pip increment, whether up or down, draws a new Renko bar.

Using increments that easily fall within normal market gaps creates the false impression of trade-able prices where none existed. Minor news events frequently result in 5-10 pip market gaps. In the case of the 10 pip gap, a box size of 5 pips creates 2 Renko bars. The two bars do nothing to communicate the fact that the prices never existed. Their presence merely indicates the direction of a move and eliminates the idea of time altogether. Time, or more specifically the absence of it, strikes me as rather important.

Small box sizes more commonly lead to questions about wildly inaccurate backtests. I received two questions last week inquiring why NinjaTrader showed $19,000 returns in a backtest, but the same forward test lost nearly an identical amount.

The backtests rely on a selected data set to generate the Renko bars used for testing. Users nearly always overlook the data source option in NinjaTrader. It defaults to one minute charts. One tick bid is the only type of data that will form perfectly accurate charts. Any other increment risks creating Renko bars that never existed.

Screenshot of settings to create an accurate Renko backtest

Change your backtest settings to use not only Renko charts, but also Bid data.

Take an extreme example of one minute chart data drawing Renko bars with a 3 pip box size. Say that the over-all height of the bar is 10 pips, the low is 1 pip from the open and the M1 bar closes 8 pips higher. How many Renko boxes does the chart need to draw? The correct answer is that there is no way of knowing.

Examples:

  • The market goes down 1 pip, then up 10 pips and settles at the close price. This draws 3 total box with one box in progress.
  • The market goes down 1 pip, then up 3 pips, then down 3 pips, then up 10 pips. This draws 5 boxes total with one box in  progress.
  • The market does down 1 pip, then up 3 pips, then down 3 pips, then up 3 pips, then down 3 pips, then up 10 pips. This draws 7 boxes total with one box in progress.

As you can see, we have no way of knowing which of the above options is correct, if any of them are correct at all. Summarizing price over time inevitably papers over what happens in the middle (information entropy). NinjaTrader has no option but to guess the unknowable.

It’s done in good faith, but NinjaTrader is essentially making up Renko data to cover up gaps in the price data. When you’re running a backtest, the whole point of the exercise is to eliminate guessing and deliver solid answers.

Most people make the hand waving assumption that it all averages out in the end. The two clients asking me this week about why their Renko backtests came out so screwy, and the reason that I’m writing this post, is that the hypothetical versus real performance was as different as night from day. It most certainly does not average out. Rather, it introduces so many errant points as to make the tests worthless.

Don’t make assumptions in your backtest. Get tick data and, if you’re using Renko bars, make sure to set the test up properly.

Signifigance Testing

by Leave a Comment

I recently had someone from Portugal approach me with an Expert Advisor that he programmed. He felt that it was successful and wanted to get my opinion on its viability. I only know a small amount of information about the system, yet I was able to confidently reject it as unproven.

Statistics are the key. It provides a relatively straightforward toolbox to easily dismiss the simplest of systems. The measure in particular that I care about is called a student’s t-test.

Let’s first talk about when this test is appropriate. If you have fixed take profits and stop losses in place on 100% of all trades, then it is appropriate to use this test. The reason is that these values provide fixed limits on the outcome of trades. The distribution of your trades is likely to form a nice bell curve. In math terms, this is called a Gaussian distribution.

If your strategy uses dynamic exits based on market conditions, then this test is not appropriate. The distribution of forex, stock and futures prices do not follow a bell curve. The test depends on the assumption that the distribution under study follows a bell curve. If you make assumptions that don’t match what you’re measuring, then results are useless at best and dangerously misleading at worst.

If you’d like to go into the mathematics involved, then you can find numerous sources on the internet that explain t-tests. I also really enjoy the book Statistics by Freedman, Pisani and Purves.

Most trading systems do not follow bell curves, which makes the details of the t-test largely irrelevant. What’s useful about it is to show that you can generally feel better about the outcomes and predictions based on the number of results in the backtest.

Restrictions and Degrees of Freedoms

The goal of any test is to ensure that the results are accurate. The more frequently that you test a concept, the more confident that you feel about the outcome repeating itself consistently. The idea of predictability largely matches our intuitive expectations. If my co-worker shows up on time regularly, then I feel confident about him showing up on time tomorrow. If he shows up late regularly, then I know that he’s likely to show up late in the future.

The increase in experience increases the level of confidence. Eventually, the number gets so big that we feel very comfortable with the probabilities.

The Portuguese client presented a system based on moving averages with 3 filters, a stop loss and a take profit. Let’s assume that each filter only had one parameter. The number of restrictions for the buy trade is the moving average period (1), the one parameter for each of the three filters (3), the stop loss distance (1) and the take profit distance (1). This yields a total of 6 restrictions for buy trades. Assuming that sell trades use the same inputs, then we have a total of 12 restrictions.

We can’t begin counting our total number of trades (i.e., degrees of freedom) until the backtest shows at least 12 trades to account for our restrictions. It’s a good idea to not infer anything about a trading system until 30+ trades elapse. With the 12 restrictions in place, that sets the threshold for the minimum number of trades to reach a conclusion at 30 + 12 = 42.

Generally, it’s a good idea to see 300-400 trades before drawing conclusions about any system. The reason for this is that some events are very rare. They may only occur once every couple of hundred trials. Allowing the amount of information to approach this threshold allows the trader to more comfortably evaluate what hidden risks may be present.

The backtest that the Portuguese individual submitted only contained 27 trades. Knowing what we know about basic analysis, I comfortably decided that there is nowhere near enough information on the system to consider evaluating it.

A Word of Caution

An algorithm’s trading statistics are almost certain to change with time. Unless you have a mathematically sophisticated model for evaluating volatility, best practice demands that you evaluate the trading results in light of the type of volatility experienced. Making money in 2008 when the markets nose dived does not mean that you would have made money in 2010, which was much quieter in comparison. Any signal in 2008 that indicated a short would almost certainly show returns that include a handful of monster winners. If those monsters fail to show up because the volatility does not cooperate, then your expert advisor will more than likely flop.

Optimize an Expert Advisor

by 1 Comment

One of the lesser known features of the MetaTrader backtester is the optimization feature. It’s so small that you could be forgiven for overlooking it.

Optimization is the process to maximize a certain outcome. In this case, it’s profit. Any EA developer wants to maximize the amount of profit made over a given period of time. The MetaTrader optimizer allows the trader to search for the combination of inputs that yielded the maximum profit over a given period of time.

The process is identical to running a backtest, except that MT4 runs multiple backtests at the same time. It then organizes the results and offers up the best combination.

Telling the backtester to run in optimization mode is easy. Simply put a check next to the word Optimization. MetaTrader will then sort through the combinations that you tell it to consider.

MetaTrader EA Optimization option

Place a check in the box next to Optimization in the MT4 backtester

The next step is to click on the Expert properties button to the right. A new window appears that contains three tabs: Testing, Inputs and Optimization. These screens allow the trader to inform MetaTrader which variables to consider for testing and how to weight the results.

Testing

The top of the testing section applies to every type of backtest. Here you can select the starting balance. MetaTrader defaults the option to $10,000, although you can make this any amount of your choosing.

The second default option allows the trader to restrict the direction of trades. It’s a frequent expert advisor programming request. It’s also one that is unnecessary. Both the backtester and expert advisor options screen allow the trader the option of restricting trades to long only or short only without additional programming. If the EA is not well programmed, this setting may cause errors 4110 or 4100 to appear all over the trading journal. It’s harmless. The only effect should be that the backtester slows down. It’s the result of writing to the journal hundreds of times or more.

The testing tab of the MetaTrader backtester

The testing tab of the MetaTrader backtester

A groupbox appears underneath these options that inexplicably relates to the optimization process. You’d think it would make more sense to place it in its namesake tab. That’s typical MetaQuotes logic at work.

The first line contains numerous parameters for choosing the best option. User overwhelmingly select for the largest account balance, but other options include the profit factor, expected payoff, maximum drawdown and drawdown percent.

The last line automatically uses a genetic algorithm. Optimization processes use either brute force methods or genetic algorithms. Brute force strikes most people as intuitive although obviously exhausting. The software tests every combination possible. Genetic algorithm’s attempt to make the process more intelligent. When the software sees that certain parameters almost inevitably lead to a losing performance, the algorithm skips similar tests where it expects to lose.

This is a great idea if you have a quality genetic algorithm. My opinion of the MetaTrader backtester is less than stellar. I don’t feel very confident about the algorithm at all. If you don’t mind spending extra time waiting for test results then I suggest unchecking this option. You don’t want to miss a potentially important combination.

Inputs

Most people find this screen confusing. The first column, called value, strictly controls inputs for simple backtests. The Value column is totally ignored during an optimization run.

The inputs tab of the MT4 backtester expert settings

The inputs tab of the MT4 backtester expert settings

The important columns for this task are Start, Step and Stop. Start is the lowest number that the MT4 backtester will consider. Step refers to the interval between the lowest value and the highest value. Tightly controlling this setting allows the user to gain quick insights into how changing the variable values affects performance without dragging the tests out for a full week. Stop is the highest number that the expert advisor will use.

The most obvious candidate for testing in this example is the Take Profit value. The default setting is listed at 50. If you trade the majors, you might want to consider settings ranging between 10 pips and 200 pips. That means that you set Take Profit row to 10 for the Start column and 200 for the Stop column. The real trick here is selecting the Step. If you choose Step = 1, then MetaTrader will run a separate test for every value between 10 and 200. That’s 190 tests, which is overkill. A step of 10 cuts the total number of tests down to 19.

Optimization

This section is the nit-picky part. If a trader feels it’s unacceptable to have 10 consecutive losses in a row, he can place a check next the the Consecutive wins box. MT4 automatically discards any tests which yield a result that contains anything checked off.

The optimization tab in the MT4 backtester expert properties

The optimization tab in the MT4 backtester allows users to discard tests with undesirable traits.

When you finish going through each of the tabs, push OK in the bottom right corner. It’s time to launch the tests.

Curve fitting in the MT4 Optimizer

A word of warning: my personal opinion is that optimizing an expert advisor is usually a very bad idea. The unique settings that yield the most profit in 2012 are unlikely to yield the most profit in 2013. If you don’t control for random chance, there’s a good probability that the 2012 best combination may result in catastrophic losses in 2013.

I recommend that traders pursue any strategy development work in NinjaTrader. I don’t like the idea of optimizing at all. Instead, I always focus on testing strategies for entry and exit efficiency. I know from years of experience that these values never fundamentally change on instruments of the charts traded. Entry and exit efficiencies make wonderful metrics for automated trading because they are so stable.

MetaTrader Logs

by 14 Comments

Log files are written records of all trading activity in MetaTrader. Whenever a trader submits an order, modifies a stop loss or connects to the broker, MetaTrader notes everything that happens along with the time.

We use log files to reconstruct a sequence of events when expert advisors run on a live account. Knowing the order in which things happened helps us to determine why an EA may not work properly.

Consider the logical steps where you want to go to the grocery store. If I were writing software to do this, my log file might read something like this:

1) Find my car keys
2) Find my wallet
3) Get in the car
4) Drive to the grocery store
5) Buy groceries

If my log file only says “find my wallet” and “get in the car”, I intuitively know that something is wrong. Why would the software not work when I get in my car? Because I don’t have my keys.

The log file helps your MQL programmer think along the same lines. When the log says, “calculate the entry rules” and the log only talks about the exits, it’s clear that the flow of the program doesn’t match the design intention.

Log files only come from the computer where the expert advisor runs. When a problem inevitably arises, your programmer will request that you send the log files from the computer where you’re running the EA. This is unfortunately a necessary part of the debugging process.

We do all of our quality assurance testing in the MetaTrader backtester. Although this usually catches the most obvious bugs, new errors will always pop up while forward testing the EA. The log file is what helps connect us to the problem, even though it happened on another computer.

Find your MetaTrader 4 log file

MetaTrader keeps two sets of log files. The most basic logs are located in YOUR BROKER NAME\logs. Most of our clients navigate here accidentally and assume it’s the file that we need. Expert advisors cannot write to this log file, so it unfortunately doesn’t do us much good.

Locating the correct log file for your programmer will vary based on your operating system. Anything in capital letters changes based on your personal information.

Windows XP and Windows Server 2003 users can find the logs in C:\Program Files\YOUR BROKER NAME\experts\logs

Expert advisors that run on Windows Vista or 7 have to put in more effort. Those log files are found in

C:\Users\YOUR USER NAME\appdata\local\virtualstore\Program Files (x86)\YOUR METATRADER INSTALLATION\experts\logs.

The part that confuses most Windows 7 users is that when they navigate to the folder with their USER NAME, the appdata does not appear as an option. The easiest trick is to double click on the current folder name at the top of the screen. Once you’ve clicked, the entire directory name will appear. If you type “appdata” after the final backslash and push enter, the window will navigate to the correct location. You can then continue clicking until the log file appears.

The name of the file corresponds to the date. The format is year, month and date (YYYYMMDD). Today is January 23, 2012, so the log file for today is named 20120123.log.

Finally, log files are often enormous – 50 MB or more. Please right click on the file and select “Compress” or “Send to, Zip File”. Sending the zip file can reduce the file size by 80% or more.

Backtesting for efficiency

by 6 Comments

Your forex backtests are absolutely worthless if you do not test the statistical entry efficiency and exit efficiency of the strategy. Everyone that runs a backtest inevitably reports the dollars earned as the outcome. Other factors exist like the average win to loss, the profit factor and the Sharpe ratio, but they do not tell you anything useful until the final step of designing an automated trading system.

The correct approach to testing a strategy should focus on the question, “is my strategy a piece of garbage?” Most people try to prove themselves right. The real test is to not be able to prove yourself wrong. The only way to do that is through a statistical approach.

Entry and Exit Efficiency

Efficiency puts a hard number to what percentage of an available trading range that a strategy captures. The trading window starts on the bar where a trade entered the market. The window closes when the trade exits.

The total available window is the highest high minus the lowest low in the window. Calculating the entry and exit efficiency simply measures what percentage of that window that your strategy tends to capture. Take the average of all of the trades and you get the overall efficiency.

Entry efficiency formula

Formula for a long trade: (Highest high – entry price) ÷ (Highest high – lowest low)
Formula for a short trade: (entry price – lowest low) ÷ (Highest high – lowest low)

Exit efficiency formula

Formula for a long trade: (Exit price – lowest low) ÷ (Highest high – lowest low)
Formula for a short trade: (Highest high – exit price) ÷ (Highest high – lowest low)

Take an example where you buy a hypothetical currency at 150 and sell it at 170. The lowest low between the time of entry and exit was 140. The price then ran all the way up to 200 before settling back down to 170, which is where the exit took place.

The entry efficiency is (200-150) ÷ (200-140) = 50 ÷ 60 = 83%. Nearly anyone would agree that this makes for a great entry.
The exit efficiency is (170-140) ÷ (200-140) = 30 ÷ 60 = 50%. Most would agree that the exit would have ideally occurred sooner than it did.

Efficiencies do not change by instrument or time frame

One major problem that we encounter with forex backtests is the limited data set. This is especially true for those interested in testing long term strategies like those on the H4 or D1 charts. The wonderful thing about entry and exit efficiencies is that they do not vary from chart to chart or even period to period.

I like to jump down to M1 charts for efficiency testing. The data is nearly endless. I never have to worry about running out. The great thing is that I know when I shift back to the H4 chart, the efficiencies should not change more than ±5%.

If you see the efficiency vary too much, then you may not have enough trades to form a statistically significant group. My experience tells me that 75 trades usually gets very close to the actual efficiency. 100 trades or more is better. When I run tests on M1 charts, I often get several thousand trades over the course of a few months. Numbers that large can tell you with a great deal of confidence just how robust a strategy’s parameters truly are.

Usually, you can assume that any results that fall within 45-55% are the result of a random, stochastic process. When I see backtests that creep right up to those barriers like 54.9% or even 55.1%, the results inevitably tank back to around the 50% mark.

Random trade outcomes and dollar profits

I wish this section was about how to make money with a random efficiency. Alas, we must cover how randomness can result in unjustified eurphoria.

I’ve been interested in the concept of randomness for several years now. Mathematicians refer to it with the more opaque name of a “stochastic process”. Despite the non-sensical name, it’s just a fancy way of saying the study of randomness – how it changes, its distribution, how far it “walks”, etc.

Yesterday, I used the analogy of coin flips to describe how Martingale strategies are probabilistically doomed to failure. One interesting concept that I did not mention relates to Brownian motion. Even with a set of random outcomes, trades will go on a random walk away from the starting point.

Einstein gets the real credit for solving the math behind the concept, even though his name is not on the term. He demonstrated that the distance a random process will follow is the square root of the number of trials. If we decide to flip a coin 60 times, we know that 50% of the time should fall on heads and the other 30 on tails.

It actually turns out that we should expect a very slight bias in the number of either winners or losers, although we do not know which one. It’s random. The precise bias, whichever way it prefers to go, should equal √60, which works out ~7. The heads outcomes should typically range from 23-37, with the tails outcomes making up the difference.

Seven trades out of sixty strongly alters the percentages, even if we know that it’s really supposed to be 50%. If heads only came up 23 times out of 60, that’s 38%. The problem is not with the coin. It’s with the number of trials. As you do an increasing large number of trails, the random bias decreases in significance in terms of the percent accuracy. 50,000 trades, for example, should show a surplus of roughly 223 trades in favor of winning or losing. The accuracy range falls within 1% of 50% on either side, a dramatic improvement.

Risks of curve fitting

Curve fitting a random efficiency relates to the idea of Brownian motion. Let’s say that we use a strategy that I know will never show an entry or exit efficiency: the moving average crossover. I’ve gone through this strategy six ways from Sunday, almost exclusively at the behest of clients. It does not work as a fully automated strategy. There is no secret set of fast and slow periods that will unlock the hidden keys to profit.

Most traders, experienced or not, abuse the backtester by searching for a set of parameters that yield the most dollar profit. They curve fit their test to optimize for maximum profit. What really happens is that the traders optimize for amount of random drift that already occurred.

When I used the example of 50,000 trades creating a natural drift of 223, I cited it with the purpose of showing how little it reduces the error in the real percent accurracy. The other consequence for trading systems is that as the error percentage decreases, the natural bias in your outcomes increases. Blindly running the optimizer only selects the set of combination that yields a combination of two criteria:

  • The drift that happened to work out in favor of that set of parameters
  • The profit and loss that varies with those parameters. The dollar profit naturally changes because the two moving averages cross at different points

You need a tool like efficiency to guard against these types of random outcomes. It’s the only method that I know of that definitively states whether or not a strategy behaves in a random manner. I especially like the fact that it breaks those elements down into two of the three basic components of a trading strategy: the entry, the exit, and the position sizing.

Efficient strategies do not work all of the time

Position sizing marks the final obstacle to building your fully automated trading strategy. A set of rules that yields a statistically efficient entry that is paired with an efficient exit does not necessarily make money. The value of each trading setup can vary, too.

Each strategy contains different sets of winners and losers. Each winner and loser varies in its dollar value. Whatever money management approach that you take requires balancing the ratio of the winners and losers in a way that normalizes the outcome of each trade. You ideally want to eliminate the variation in dollar value. 20 pip trades should earn or lose you exactly as much as the 100 pip trades.

That seems counter-intuitive. Most traders want to win in proportion with the size of the opportunity. It’s better from a system perspective to entirely ignore the size of the opportunity and to make each trade worth the same amount. Betting more or less with each trade effectively normalizes the value of each trade.

Using a stop loss stands out as an obvious candidate to fix how much a trade is worth. The severe disadvantage is that it almost always negatively affects the exit efficiency. Whenever I can get away with it, I always recommend using a market based exit instead of an arbitrary stop loss. Traders usually scream at the top of their lungs when they hear me say this. I’m just speaking as a systems developer. The numbers are what they are.

Accurate NinjaTrader backtests

by 21 Comments

Yesterday, I wrote a bit about how the backtester simulates ticks. When NinjaTrader encounters a bar where the stop and take profit fall within the bar’s range, it always assumes that the stop was hit first.

I had this issue come up for a client that wanted to sell his strategy to an institutional client. The institution wanted the ability to backtest in order to understand how the strategy performs in broader market conditions. The live results only covered 9 months, which made it difficult for the investor to make an informed decision. A reliable, accurate backtest would allow my client to facilitate the sale.

When we first programmed the strategy, it looked like an abysmal failure. The results showed a 45 degree equity plunge from every trade. He placed his stops at 1/3 of ATR, so naturally, NinjaTrader assumed that they were hit all the time.

The solution that we developed for him used the high time frame chart that the strategy actually used, but we then modified the code to do include what NinjaTrader calls intrabar granularity. Basically, we add tick charts on the backend to force the strategy to update on a tick by tick basis.

It’s actually pretty simple to program. In the Initialize function, you need to add bid and ask tick charts.

protected override void Initialize()
{
Add(string instrumentName, PeriodType.Tick, 1, MarketDataType.Bid);
Add(string instrumentName, PeriodType.Tick, 1, MarketDataType.Ask);
}

Then, in the OnBarUpdate() function, you leave the code unchanged. All you need to do is to put it within a simple if-then statment

protected override void OnBarUpdate()
{
if( BarsInProgress == 0 )
{
// Do stuff
}
}

The backtest results came within $5 of the actual trading results over a period of 9 months! Although backtesting frequently leads to inaccurate results, we were thrilled to reproduce the result with such a minor error.

The major disadvantage to this method is that you must manually type in the instrument name inside of the source code, then compile it, every time that you want to run a backtest on a new instrument. It’s not a great solution, but NT7 unfortunately does not support programmatically calling the InstrumentName from within Initialize().