Reverse engineer expert advisor

January 27th, 2012 by Shaun Overton

I guarantee that you are not the first trader to consider the idea of reverse engineering an expert advisor. The idea pops up most frequently among traders that don’t want to pay for a commercial expert advisor. Alternatively, they want to use the same strategy when another trader that doesn’t want to share the idea. The motivations for reverse engineering remind me of decompiling EAs, making me leery of the idea.

Reverse engineering a strategy only stands a chance when some parameters are known about the strategy. If, for example, you knew that the strategy involves MACD and moving average crossovers, it at least provides a reasonable starting point. The programmer could write software which combines every possible combination of two moving averages of various types with every known type of MACD. The programmer could then make guess about which types of signals might result in a buy or sell decision. If the guesses are not very good, then the outcome of the reverse engineering attempt is certain failure.

Then you have the problem of guessing on which chart to base the decisions. Many traders use standard charts like the M1 and M15, but many others use less common options like an M3. If the trader uses multiple charts like an M2 and M10, the resulting trade history would clutter together. Good luck trying to pry apart the different series.

Making things worse is if the trader uses a chart that doesn’t depend on time at all. Tick charts are the most common, although you occasionally see less orthodox options like Point and Figure charts and Kagi charts. Time is irrelevant. You wouldn’t have any idea on which charts to run the test.

This approach of making somewhat intelligent guesses while throwing mud at the wall is called a brute force attack. You literally designate every unique possible combination, then see which one opens the metaphorical safe. Some results will bear no resemblance at all to the actual results. If you get extraordinarily lucky and/or have fantastic intelligence, then you might find a close replica of the strategy.

It would be possible to study the correlation of the tested values with the values in the supplied account statement. You would ideally want to find data clusters with similar variable settings that do not dramatically alter the correlation between the reverse engineered strategy and the actual account statement.

If you don’t know very much about the strategy, or if what you think you know turns out to be wrong, then you stand no chance at all of reverse engineering the expert advisor. For all that you know, the EA that you thought used RSI might turn out to run on phases of the moon (yes, there are real strategies that do that) or that make decisions based on coin flips.

Most people assume that they know more than they really do. I would discourage all but the most fool hearty or stubborn from making the attempt, unless you had a very good reason for doing so.

Scripts in MetaTrader

January 26th, 2012 by Shaun Overton

Scripts are executable files in MetaTrader that only run one time. They are perfect for tasks that are routine but time consuming or unpleasant to do. The most advantageous use of scripts is that they do not rely on incoming price ticks in order to run. The script executes the moment that the trader drops it onto a chart.

An expert advisor runs continuously, but it relies on incoming ticks to know that it should update itself. The frequency that the market changes price varies with the time of day. This means that the period between updates for an expert advisor is highly unpredictable. The predictability of a script’s timing – it runs immediately – makes it more suitable for some trading tasks than an EA.

Script examples

A scalper wants to open a trade quickly. He routinely applies a 20 pip stop loss and a 3 pip take profit. His usual process would involve:

  1. Click inside of MetaTrader to open a trade
  2. Select the correct forex pair
  3. Wait for the trade to open
  4. Frantically add the take profit as quickly as possible

Alternatively, the trader could keep his chart open and the scripts window nearby. Whenever he decides to trade, he drags the script onto the chart. The steps above still occur. The critical difference is that they happen in a fraction of the time. The script runs once, then it removes itself from the chart.

Some tasks require a tiresome amount of clicking rather than speed. Scripts are also useful in that scenario. A lot of MetaTrader users like to stack multiple pending orders above and below the market in a grid pattern. An example would involve placing 10 orders above and below the market at different prices. Doing this manually would take a few minutes.

The alternative is to run a script that does it one time for you. Scripts can display input screens just like expert advisors. That way the user can control the settings.

When the trader is ready to bracket his orders, he drags and drops the script onto the chart. The orders show up on the screen at the requested prices. The total time takes a few seconds instead of several minutes.

Other script uses

The most common, though unorthodox, use of a script is to feed prices into the History Center for a custom offline chart. The script is programmed to run at a set interval such as every half second. The script samples the price, then records the information where the historical prices are kept. The offline chart then re-reads the information and updates the price.

MQL programmers elect to use a script instead of an expert advisor because they are setting an infinite loop. Although scripts technically run once, this script is never allowed to finish its first run. It keeps waiting every set interval to update the price. EAs would not work well here because new, incoming ticks would create a backlog of occasions where the EA is supposed to have run. I would expect MT4′s memory use to eventually get out of control and crash the program if someone elected to take this approach. Even though it is not an ideal solution, scripts update historical prices successfully without the tick backlog concern.

MetaTrader Logs

January 23rd, 2012 by Shaun Overton

Log files are written records of all trading activity in MetaTrader. Whenever a trader submits an order, modifies a stop loss or connects to the broker, MetaTrader notes everything that happens along with the time.

We use log files to reconstruct a sequence of events when expert advisors run on a live account. Knowing the order in which things happened helps us to determine why an EA may not work properly.

Consider the logical steps where you want to go to the grocery store. If I were writing software to do this, my log file might read something like this:

1) Find my car keys
2) Find my wallet
3) Get in the car
4) Drive to the grocery store
5) Buy groceries

If my log file only says “find my wallet” and “get in the car”, I intuitively know that something is wrong. Why would the software not work when I get in my car? Because I don’t have my keys.

The log file helps your MQL programmer think along the same lines. When the log says, “calculate the entry rules” and the log only talks about the exits, it’s clear that the flow of the program doesn’t match the design intention.

Log files only come from the computer where the expert advisor runs. When a problem inevitably arises, your programmer will request that you send the log files from the computer where you’re running the EA. This is unfortunately a necessary part of the debugging process.

We do all of our quality assurance testing in the MetaTrader backtester. Although this usually catches the most obvious bugs, new errors will always pop up while forward testing the EA. The log file is what helps connect us to the problem, even though it happened on another computer.

Find your MetaTrader 4 log file

MetaTrader keeps two sets of log files. The most basic logs are located in YOUR BROKER NAMElogs. Most of our clients navigate here accidentally and assume it’s the file that we need. Expert advisors cannot write to this log file, so it unfortunately doesn’t do us much good.

Locating the correct log file for your programmer will vary based on your operating system. Anything in capital letters changes based on your personal information.

Windows XP and Windows Server 2003 users can find the logs in C:\Program Files\YOUR BROKER NAME\experts\logs

Expert advisors that run on Windows Vista or 7 have to put in more effort. Those log files are found in

C:\Users\YOUR USER NAME\appdata\local\virtualstore\Program Files (x86)\YOUR METATRADER INSTALLATION\experts\logs.

The part that confuses most Windows 7 users is that when they navigate to the folder with their USER NAME, the appdata does not appear as an option. The easiest trick is to double click on the current folder name at the top of the screen. Once you’ve clicked, the entire directory name will appear. If you type “appdata” after the final backslash and push enter, the window will navigate to the correct location. You can then continue clicking until the log file appears.

The name of the file corresponds to the date. The format is year, month and date (YYYYMMDD). Today is January 23, 2012, so the log file for today is named 20120123.log.

Finally, log files are often enormous – 50 MB or more. Please right click on the file and select “Compress” or “Send to, Zip File”. Sending the zip file can reduce the file size by 80% or more.