Algorithmic and Mechanical Forex Strategies | OneStepRemoved

Become an Algo trader in Baby Steps

by 10 Comments

The world of traders is divided into two groups; those who trade using algorithms and those who don’t. Those who trade using algorithms, aka algotraders, are well aware of the advantages of trading with an algo… And for those who don’t use algos? They are equally aware of the algo advantages but are reluctant to dive into its complexity; they are deterred from learning how. But the road to success in trading is not by avoiding challenges, but overcoming them, perhaps in baby steps.

First Baby Step to becoming an Algo trader

Let me ask you a question. What do you think is the first thing you’d need to do to become an algotrader? You might respond, “Learn programming, of course! How else?” Well, you’d be wrong.

If you plan to learn programming for the sole purpose of becoming an algotrader, you’re likely to get lost. Eventually, in despair or frustration, you’ll give up on algotrading.

There are numerous languages, from MQL to R to Python, and you have to decide which one to start with. You might find yourself wasting valuable time learning far too many trivial functions. It may take quite a long time before you even create a trading algo, let alone a profitable one.

But there’s another way, which I call reverse engineering.

The first step is to figure out which trading algo you want to make. In other words, what are the functions and strategy it should implement? Then you test it and finally, move on to the programming part. This way, you’re focused like a spotlight. You know exactly what your algo should do and can focus on the exact functionalities you need to learn to make it happen.

Everything will just come intuitively; which language, how and in what. All the pieces will fall much quicker into place because you already know what to look for.

The best way to start is by using a flowchart. It is actually one of the first things you learn in programming schools.

And what determines the flow? Of course, it’s the conditions. What we call the “ifs.” “Ifs” can be one condition or have many “ands” and “ors.” That’s how you decide what your algo should do in any given circumstances. The conditions to the algo are what the brain is to the body; they do the thinking.

Second Baby Step: Use Excel

Once you’ve made a flowchart of conditions, rather than using a complex tool, use Microsoft Excel or some other spreadsheet software.

Algo

Source: MT4

Extract historical data of a price data; start by using only the closing price. If you want to trade on a daily interval, extract daily data and so on. Use a separate column for each of the conditions in the flowchart, that way it’s easy to write and figure out. Add another column for a buy and sell output to mark when you “opened” and when you “closed” a position. And finally add one column of accumulated profit/ loss.

Use separate columns for each conditions and fill the chart. All you need now is to run a linear chart on one column, accumulated profit/loss, and look at the algo’s historical performance.

Once you master the basics you can move on to move advanced back-testing and curve fitting. But for a start that will do.

Algo

Finally, Learn to Program

Ok, you’re not a “baby” anymore; you have a good idea of what an algo trader needs to do. Now that you’ve got the concept down pat, you’re ready to begin learning programming.

In terms of which language to start with, that will depend on the circumstances. If you are already using an MT4 platform, it’s a no brainer; learn MQL which runs on MT4. The MQL website is filled with “how to” materials. And since you already have a diagram of your algo and an idea what you want to do, finding your way should be simple.

But if you trade with a different broker, you’ll have to decide which best suits. Personally, I recommend starting with MQL, just because it’s easier. Then, you can move into Python, which is a rather easy language to learn or C++ if your algo needs to work fast. If you algo is heavy on the math then perhaps R would fit.

Here are some places you can learn online:

MQL

Coursera

Code Academy

EDX

Or, of course, you can learn from a book. I am more of a book person, but that’s just me.

Then there is another issue—API. API is the mechanism that enables your algo to communicate with your broker and execute your trades. Some brokers work better with certain programming languages than others. Most large brokers have communities and forums that can go into detail as to the best way to use an API.

Unlike the first two baby steps, no one would ever say that learning to program was a baby step. Rather, it’s more of a giant leap. It takes time to learn and to master, though it is worth it in the long run. Meanwhile, you can always use libraries of codes on the web for more complex algos. The thing is that once you’ve accomplished the first two baby steps, making the final leap into programming is a no brainer.

Too Many Charts

by 2 Comments

A million charts on the screen. It looks like a blob of every known currency pair with a mix of every time frame added for good measure. You, my friend, suffer a nightmare juggling the EA settings on all these charts.

The idea of running a single Expert Advisor that trades all charts and multiple time frames stands out as an obvious solution. It’s not a good one, though. The way that MetaTrader 4 is designed prevents this setup from working smoothly.

The start() function in MQL4

MQL4 Expert Advisors only run in one of three different ways. The init() function is called when you load the EA onto a chart. The deinit() function operates whenever you remove the EA. That leaves the start() function. It’s like the beating heart of all MQL4 programs.

Incoming ticks tell the chart to notify an attached expert advisor about the updated price. When that occurs, the expert advisor runs the start() function. That function contains all of the code that traders associate with expert advisors: placing trades, implementing trailing stops, etc. All of those actions depend on incoming ticks.

The organization of MQL4 creates a problem for the goal of creating an EA that places trades for all charts. Expert advisors only update when a tick comes in. If I wanted to trade the GBP/JPY from a chart attached to the EUR/USD, any delay in EUR/USD ticks causes a delay in executing GBP/JPY trades.

This probably does not seem like a big deal. It isn’t a big deal – most of the time. It can, however, create a nagging problem of wondering why a noticeable percentage of trades seem to execute late. Traders on one minute charts would even notice missed signals. Although it’s not frequent, even the major pairs often go longer than one minute in between ticks.

Work-arounds

One idea to reduce the number of open charts while ignoring the incoming tick problem is to create an EA that trades on multiple time frames for the currency pair where it’s attached.  If I wanted to trade the AUDCAD on M30, H1, H4 and D1 charts, then I could place the EA on one of the those charts but have it look for trades on the selected time frames. This type of solution could decrease the number of open charts by 75% or more.

The idea of controlling everything from a single chart is very similar to the market scanning indicator that we created several months ago. There is really no difference between that indicator and the expert advisor proposed here. I feel that missing an indicator signal is of far less consequence than missing trade signals. Expert advisors are more likely to perform extra, ongoing operations like trailing stops that indicators completely ignore. The consequence of delayed ticks is far less significant for an indicator than it is for an EA.

MQL Organization

by Leave a Comment

All MQL expert advisors and indicators contain a few essential components. The general organization of MQL programs does not vary too often.

Files usually start with a declaration of #defines (pronounced pound define) global variables and external variables, also known as an extern data type. They appear near the top of the code to help the read gain an understanding of the variables that will run in the program. Ideally, the names of the variables and how they are organized should assist the programmer with form a general understanding of what the expert advisor or indicator might do.

The next section is usually the init() function, which is the word initialize abbreviated. This section of the code is particularly relevant to programming custom indicators. Most of the general indicator settings like declaring the indicator buffers, the colors to use and other basic features are set within this section. I use init() in every expert advisor that we build to convert the inputs into an appropriate setting for the broker’s pricing. If a client inputs a stop loss of 50 into an EA, I don’t need to do anything if it’s a 4 digit broker. I do, however, need to convert the input to work with a 5 digit broker. I run a quick check within init() to see if Digits == 3 || Digits == 5. If so, then I multiply inputs affected by that setting by 10.

deinit() is the least important section; it’s pretty easy to deinitialize an MQL file because it usually does not take up any system resources. It’s rarely used for anything important. The only uses that I ever have for deinit() are to close an open file handle or to make some sort of closing note. This is often done either on the chart directly through the Comment() function or more often by writing directly into the log file.

The start() function is the real meat of an MQL expert advisor or indicator. Whenever MetaTrader detects an incoming tick, it alerts any MQL programs. Those programs then call the start function so that it can do whatever needs doing. All trading operations, calculations, account monitoring, etc, occur within this section.

All of the other custom functions within the program appear below start(). I usually prefer to rank them in order of their importance or the frequency with which I call them throughout the program. The order of placement of functions does not affect performance in any way at all. It’s strictly a cosmetic practice that makes programming code more legible.

MQL Data Types

by 2 Comments

MQL4 supports seven data types within the program. Each type is associated with different trading tasks that programmers need to perform. The goal of this article is to provide a brief reference for when to use each data type.

The double data type is probably the most common type found in MQL programs. This is because it is the type responsible for calculating floating point numbers. Say for example, that an expert advisor needs to determine when to adjust a trailing stop. The expert advisor looks at the current price and subtracts it from the current stop loss to maintain the appropriate distance (1.3230-1.3209=0.0021). The distance requires a decimal point. When the expert advisor saves the distance to memory, it needs to save the information after the decimal point. That forces the programmer to choose a variable of type double.

Integers, or int, is the simpler version of double. Double values require a decimal place to hold the number’s value accurately. An integer, or whole number, does not have a decimal place. Integers are appropriately used when the MQL programmer knows for a certain fact that the number will never contain a decimal. An example would be if you wanted to implement a max trades feature. If the number of open trades in the account exceeds a certain number, then prevent trades. We know in advance that there is no such thing at 4.76 trades being open. There can only be 4 trades open or 5 trades open. This clearly indicates the need to use an integer.

Datetime values are just what they sound like. They represent both the date and time. More specifically, a datetime variable represents the number of seconds that have elapsed since January 1, 1970. This is where it gets a little tricky. The number of seconds that have elapsed is actually an integer. Datetimes store integer values but then associate them with a date and time.

A value of 0 would indicate that the time is 00:00 on January 1, 1970. A value of 60 stands for one minute later at 00:01 1/1/1970, and so on. One benefit of knowing that the datetime type stores information as integers is that you can easily determine the amount of time that happens between an event. If the event starts at 15:35 and ends at 18:12, you can simply subtract 18:12 – 15:35 and wind up with the number of seconds between those values. That information can then be used to determine the number minutes/hours/days between the two events.

The color data type, not surprisingly, holds color information such as black, yellow, red and so on. Much like datetime types, color also uses integers to store the information. The difference, though, is that extracting the color information from the integer is not at all obvious. Increasing a color type from 32768 by one will not necessarily make it more or less green. Colors use the integer information to retrieve the red, green and blue components of the color in hexadecimal format. Explaining hexadecimals is well beyond the scope of this article. It’s unlikely to come up in your MQL programming. I’ve been doing this for over five years and only came across one project that required manipulating a color in way more complicated than alternating between two set colors.

A string is anything that resembles a word or sentence. It always uses quotes to contain the information. My favorite use of strings is to gather information to display on the chart or in a log file whenever I need to debug an expert advisor.

Char is the final data type. It’s so closely related to a string that I wasn’t even aware this type existed until I looked up information for this article. If we study the word “trade”, then we will find that it is composed of the five characters t, r, a, d and e.

A final note on data types. There are two ways that types are held in memory. An extern variable is one that shows up in the inputs screen whenever an expert advisor or indicator loads. Static variables are the opposite. They remain within the MQL program and never visible outside of it.

MQL Debugging

by 1 Comment

MQL is a very simple scripting language. Debugging MQL, unfortunately, is not easy at all. The MetaEditor compiler that MetaQuotes supplies simply does not contain the sophisticated tools that most programmers are accustomed to using.

MQL debugging problems

Visual Studio and other sophisticated IDEs (integrated development environments) contain a number of features that make it easy to debug code while the programmer writes it. The biggest example of this are break points. A break point is a point in the code where the compiler tells the computer to cease running the program when it arrives at a certain line of code.

Consider the example where a trailing stop sets the new stop incorrectly. The natural instinct for most programmers would be to run the expert advisor on the visual backtester, then insert break points on the lines of code shortly after the trailing stop calculations. Break points halt the code, allowing the programmer to peer inside the brains of the software to see what it thought at the time it made a decision. The key advantage in Visual Studio is that the values of all of the variables are clearly visible. It is possible to walk through the program step by step. Whenever one of the steps does not follow the desired rules, the required change is usually obvious. MetaQuotes thankfully included break points in MQL5. They are not available in MQL4.

The lack of full intellisense support affects my programming speed more than anything. Intellisense detects the use of reserved words like OrderSelect() or ObjectGet(). The MetaEditor includes a rudimentary intellisense, but it lacks the fine details that make it so convenient in Visual Studio.

I am used to programming in C# where I can type the first few letters of a variable or class, then the IDE fills out the rest. When I type “Mes” in C# and push the space bar, I know that the MessageBox option will appear (assuming that I declared the appropriate namespace). The MetaEditor provides a list of candidates for reserved words. The programmer must then either select the option with the mouse or press enter.

I know it seems trivial to require pressing enter instead of the space bar, but think about how many times code resuses the same reserved words or variables. The extra key presses really do add up to a lot of unnecessary typing motions. That’s doubly true for a thirty year old that already wears a wrist brace for carpal tunnel pain.

The MetaEditor’s biggest weakness is that it does not detect variable names. We often write expert advisors that contain several thousand lines of code. Tracking the names of tens of variables poses its own challenges. When the coder types in the same set of variable names repeatedly, it would be nice to simply type the first three letters and move on. Copy and paste might provide a decent alternative. The problem is that variables usually group together. You cannot keep 5 different copy and paste items readily available.

The MetaEditor allows functions to return invalid types. Functions declared as double can return strings, integers or nothing at all. The MQL4 compiler does not track whether or not these are valid. It leaves it up to the programmer to discover the invalid type during real time testing. This oversight is a nightmare for the unwitting programmer that mistakenly returns the wrong type.

This is doubly true when a double function is mistakenly returned to an integer variable. MQL4 does not prevent illegal double to int casts. Even worse, the expert advisor continues running with a 0 value for the interger instead of throwing an exception or error message. I cannot count how many hours that I’ve wasted tracking down variables that look perfect, only to realize that I declared the wrong data type. This usually happens when I’m on autopilot, pounding out code. What seems efficient at the time usually costs several hours of hair pulling frustration.

MQL debugging techniques

The MQL programmers on staff here usually resort to any of the following techniques. You may find that using them in combinations helps improve the debugging process even more.

Debug compiler error

This one can be the most frustrating. The MetaEditor attempts to hint at which line of code causes the compiling error. I say attempts because it gets it wrong more often than it gets it right. There’s nothing more irritating than looking at a perfectly legitimate line of code that the compiler flags as problematic.

I almost always resort to commenting out increasingly large sections of code until the error disappears. I start with commenting out a single line of code. If that doesn’t work, then I comment out ten lines. If that doesn’t work, I might comment out entire functions or sections of code. When the compiler finally runs properly, you know that the last section of commented out code contains the bug.

Next, you backtrack. Start making the offending commented-out section smaller and smaller until the error reappears. Now, you have finally zeroed in on the actual source of the problem.

Debug in real time or on the backtester

My preferred method of debugging is to comment most of the relevant decision information onto the screen, which is done using the Comment() function. I then run the visual backtester, watching how the data behaves in relation to the visual information.

On screen comments are essentially jury-rigged break points. Controlling how and when they appear allows the programmer to step through the code to uncover the issue. The only difference is that comments do not forcefully prevent the code from running. The text which appears is very small. Aside from that, I really like that fact that it’s so robust. The comment function always works without a hitch, making it the best friend of a programmer that’s debugging code.

Taking screenshots takes this to the next level. Whenever customers ask questions about why an expert advisor behaved a certain way, the easiest answers come with screenshots. Commenting the imitation break points usually provide bullet proof answers – the programmer and customer can literally see what the EA thought at the time it made a decision. MQL4 offers the WindowScreenShot() function to do this.

The EAs that we program always take screenshots during critical actions, such as entering a trade or adjusting an exit condition. The goal is to provide a visual record of every decision with an eye to answering future questions about the behavior.

Our default template includes a true/false variable called WriteScreenshots. Traders control whether they want to bother with this debugging feature or not. The only downside to it is that every recorded action eats up about 50kb of hard drive space.

Log files represent the final debugging option. The major drawback is that they are so ugly and difficult to read. I almost always prefer screenshots for this reason.

Nonetheless, log files do have their place. The main way to use them is as error catchers. Whenever a process goes awry due to a problem with either MetaTrader itself or with a broker transaction, the log file is the easiest place to catch it.

Debugging MQL files is a skill that takes awhile to learn. The tools at the programmer’s disposal are much different from those available to higher level languages. Once you get used to working with the much simpler tools in the MetaEditor and MetaTrader, the bug clearing process usually gets much easier.

Decompile MQL ex4

by 48 Comments

Decompiling a file means to break open the compiled source code to view the contents. The goal of decompiling is to access the original source in order to modify it in some manner.

MQ4 and EX4 differences

When you install an expert advisor, you may have noticed two different types of files listed in the experts or indicators folder. .mq4 files hold the original programming code in the MQL language. Programmers can access this file type in order to make changes to the EA or indicator.

The computer does not save ex4 file information in human readable format. Instead, it compiles the code into a series of zeros and ones, called bits, that the computer reads in real time. If you open an ex4 file in the MetaEditor or in NotePad, all that comes up are wiggly characters and gibberish.

The best analogy is to consider the idea of a game plan. If you know that the opposing team’s defense moves slowly, you might try passing more frequently. You don’t know all of the specifics of how the game will play out. The purpose is to develop a general response to a set of circumstances. That’s the mq4 file.

The ex4 file is the series of decisions made in the game that implement the plan to pass more often. The idea is ready to go, even though the individual plays could not have been known because they were in the future. When the plays develop, the players take the generic game plan and implement it in real time.

Reasons to decompile

Most requests to decompile an ex4 file come from traders that purchased a commercial expert advisor and are not happy with the product. Breaking the compiled ex4 file yields an mq4 file. An expert advisor programmer can then take the MQL contents and make the desired changes.

Many would be decompilers want to make harmless changes to a file. A successful indicator may predict the direction of the market, yet it may not generate alerts when it’s a good time to trade. New trading setup signals might save the user from staring at charts too long. Traders mistakenly assume that they need access to the source code to program an MQL indicator. It’s easier to ask a programmer to use the iCustom function rather than breaking into someone else’s code. It’s a legitimate use of the code without requiring unauthorized access to the code.

Some frustrated traders approach me with expert advisors that lose money hand over fist. They want to modify the file contents in the hopes of salvaging a potentially useful concept from the strategy. I’ve also spoken to forex traders that bought “life time licenses” to expert advisors, only to have the vendor go out of business. They want to decompile the ex4 to remove the licensing restriction on a product they rightfully own but that they can no longer access.

I sympathize with the situations as both are common and ethical responses, in my opinion. I suspect, however, that the legal issues with modifying a commercial EA, are not in favor of the customer.

Then, there are the thieves that like someone else’s work but don’t want to pay for it. I unfortunately get this request all the time. OneStepRemoved.com does not offer decompiling services. Too many of the requests revolve around stealing other people’s work.

If you sell expert advisors or sell custom indicators, the most appropriate approach is to program the MetaTrader file in a DLL. It’s even better to encrypt the DLL.

The MQL language was not written with anything in mind other than trading. Code security and the risk of decompiling were not concerns of the MetaQuotes development team. That lead to a preponderance of MetaTrader programmers from Eastern Europe to develop MQL decompiling software and to release it on the internet. Going back and forth between the ex4 and mq4 formats is very, very easy.

Information loss after decompiling

Decompiling ex4 files does yield an mq4 file. That is not, however, the same thing as the original mq4 file. The decompiled ex4 file strips the variable names and replaces them with new machine names.

Good programming practices require naming variables after their purpose. When a variable maintains the distance of a stealth stop, the MQL programmer might name it stealthStop or stealthStopDistance. The decompiled version renames it to something hideous like i_594 or g_12. The decompiling process completely destroys the logical flow of the original mq4 file.

Now imagine a program with 30-50 variables. The name of every single one of those variables changes. The first overview of the code makes no sense. The variable names do not reflect their functionality in any way.

Function names and inputs, also called externs or external variables, are preserved. They do not make the structure of the code clear on their own. Together, they usually form enough information for the programmer to recreate the variable names.

The process of marking decompiled ex4 code and renaming the ugly variables takes at least 1.5 hours. It can many hours if the code is longer than a few hundred lines or the number of variables is unusually high. The review process is necessary if you intend to make any changes to the mq4 file. There’s no way to go about modifying existing MQL code unless its current functionality is clear.

Changes to the mq4 file are exactly like any other MT4 programming project once the review process ends. The coder must program the changes and debug the file to ensure that it works properly.

Find an ex4 decompiler

You can run a basic search for “mql decompiler”, “ex4 mq4 decompile” or any similar combination. The results pages usually show stiff competition among the vendors. It’s not uncommon to get files decompiled for $8-15 per file.

The file delivery times vary between individual vendors. I’ve heard reports that it can take anywhere from a few hours to a week to receive decompiled files. Assuming that you’re decompiling a file for a legitimate reason, it’s hard to know who to trust. The industry, after all, largely targets customers that are stealing. Asking for a trustworthy decompiling service is akin to asking for a trustworthy thief.

Robust Expert Advisors

by Leave a Comment

When you want to program an Expert Advisor, it is important to consider how you are likely to use the file. The most common mistake that I see new MQL programmers make is that they rely too heavily on memory. Events like a power failure, a hard drive crash or weak internet connection can wreak havoc when trading with an Expert Advisor. The EA should ideally be able to pick up and run whenever you are able to restore conditions to normal.

Memory issues

Problems sometimes result from something mundane like changing the settings. I learned that the hard way when a client came to our Dallas office for two weeks to order a very complex EA. Everything worked perfectly in the backtester. The overnight tests, however, never lasted for more than a few hours before something went haywire. I instructed the client not to change the settings; just turn it on and leave it alone. For whatever reason, the importance of those instruction never sank in. It took me ages to realize that his definition of “leaving it alone” didn’t quite mesh with mine. The little tweaks that he made during trading messed with the settings that I stored in memory.

I can instruct a client to use software in a certain manner. More often than not, they deviate from the instructions to some degree. It’s human nature. I learned from those challenges, like any professional should. Our programming templates changed in structure to minimize the need for special instructions.

Global variables

Global variables are a common tool to avoid the issue of relying on memory. When an EA is removed from the chart, the global variables that it created are accessible for you, the EA or any other EA to read from anywhere inside of MetaTrader.

Although the use global variables sounds sophisticated, it’s actually a very simplistic approach. MetaTrader maintains a hidden text file in its installation directory with the variable name, the value that it is storing and the time it was last modified. Whenever more than 6 weeks elapsed since the last time MQL used the global variable, MetaTrader automatically deletes it.

A Martingale system might opt to use a global variable to track a simple statistic. Martingales are wild with their approach to risk, so a trader could realistically want to know at a glance the maximum level traded. Global variables make a trivial task out of that need. If the next level to trade is greater than the global variable, then the MQL code should update the global variable.

The only item to remember is that the global variables expire after six weeks. The EA might need to refresh the information by resetting the global value to its current value. Doing so would keep the variable fresh from MetaTrader’s perspective.

A screen shot of global variables in MetaTrader 4

Read information instead of storing it

I prefer to rely on MQL‘s OrderSelect() command to recreate whatever information I need. Although it is more complex, the advantage is that the EA can function any time, any place, exactly as designed. If the hard drive crashes, it’s not a problem from a recovery perspective. You are still in trouble during the down time, though. You can load the same EA on your MT4 account and have it immediately reconstruct the information.

Taking the above Martingale scenario, we could reconstruct the number of levels traded by comparing the open and close times of trades. If trade #1 has a close time of 2011 Dec. 5 00:00 and trade #2 also has a close time of 2011 Dec. 5 00:00, it tells us that they are part of the same trade group. So far, we have 2 levels traded. If trade #5 closed at 2011 Dec. 5 00:00 but no trades did thereafter, then we know that we traded 5 levels total. The EA can then loop through all possible combinations and pick out the highest number from that.

The approach of reading from the order history is a bit of overkill for this simple example, but it comes in handy when you need to calculate more complicated information dynamically and with a minimal risk of error.