Algorithmic and Mechanical Forex Strategies | OneStepRemoved

Things that go wrong programming

by 2 Comments

I started this business to focus on designing better trading systems. Programming obviously plays a large role in the process.

What most people don’t realize is that the programming experience can be quite challenging. When a project takes more time than expected, it tends to take far more time than the original estimate.

Programming is like Air Travel

Many of you travel regularly. Flying is pretty much a given when you travel any significant distance.

Programming is like flying

Programming is like air travel. Small problems compound into big ones.

How many times have you traveled and the flight arrived 5 hours early? The question is laughable. It doesn’t happen.

20 minutes early to the gates makes most frequent fliers ecstatic. They know that arriving early, even if only by a few minutes, is as good as it gets.

Performance does relate to the airline to some degree. Checking for maintenance problems prevents surprises 20 minutes before takeoff or, heaven forbid, in the air.

The crew arriving on time helps. The last time that I flew from Dulles to Dallas, the replacement crew arrived at the gate an hour late.

The last two times that I flew to Dublin, United Airlines lost my bags… both times. Sometimes, it really is 100% the airlines’ fault.

Force majeure

Those experiences aside, how many times do airlines goof up so badly that travelers arrive days late? Travelers do arrive with severe delays, but those circumstances are usually weather related. It’s outside of the airlines’ control.

I remember the volcano in Iceland that erupted a few years back. People were literally stuck in Europe for a week.

The sequester is a great example. In what’s certainly a willful choice to inflict pain on fliers, the FAA decided to furlough air traffic controllers at major airports.

Those airports are the same ones that I frequent. When I fly to Dublin on Tuesday and I’m potentially 4 hours behind schedule, I’ll be angry. But, I’ll also know to direct that anger at Congress instead of the airline.

Programming and Travel are Fragile Systems

The idea for this article came from Antifagile, where Nassim Taleb discusses how small changes create exponential problems.

Travel is familiar to all of us, so when we think about the delta, which represents the small changes, picture it as the time delay or increase in transit time.

Consider the effects of 3 different deltas

Consider my layover in Newark. How late can I be before I miss the connecting flight. If I miss the connection, how long does it delay me?

20 minutes – The change here is minimal. I will suffer a great deal of (probably unnecessary) stress. My wife and I might jog across the terminal, looking slightly foolish in the process. Nonetheless, the chance of making the connection is near certain.

60 minutes – This is scenario is right on the verge of disaster. My poor wife will listen to me groan and bite my nails as I flip out about missing the connection.

If we do make the flight, it’s only because the airline decided to hold he flight at the gate. Doing so inconveniences hundreds of waiting passengers while a handful of travelers scurry to board the flight.

If they don’t hold the flight, well, then I’m screwed.

The best scenario the can occur after missing the connection is that the airline transfer us to another European destination. The airline then needs to put us on a partner airline to fly us into Dublin, backtracking where we just came from. A one hour delays causes us to

  1. Wait for another European flight
  2. Fly an extra hour to a different destination
  3. Wait for a Dublin connection in a different airport
  4. Fly an hour backtracking

A delay like this could easily result in a an extra 6-8 hours of travel time- all from a 1 hour delay.

3 hour delay – Catching another flight to Europe looks really optimistic. The best case is that the airline put us up in a hotel for the night and sends us on tomorrow’s Dublin flight. A 3 hours delay expands to a 24 hour wait, plus the remaining flight time.

Programming

Ok, Shaun, Ok. What does his have to do with programming?

Just like traveling, a programming project can only go so well. Whenever something unexpected occurs, the problems compound themselves exponentially.

The Evil Delta

Time is the enemy of the traveler. In programming trading robots (or programming anything, really), the delta is the degree of surprise.

Operating system changes: We developed a custom MT4 plugin for a client that likes to trade price ladders. One week after delivering the software, Microsoft released an operating system update. The update broke code in the software that we provided.

Communication: You believe that you asked for one thing, but you get another. Items that seem like minor oversights can blow up into major problems.

Chris worked on a project last month that sought to execute a trading grid at precise intervals. Chris’ original version used market orders. A handful of bugs popped up, but the core of the original version worked well. The client, however, assumed we would use pending orders and requested that it be changed.

The change ruined the original design. More importantly, we discovered that achieving exact execution was fundamentally impossible because we couldn’t precisely control the execution time.

What started off as a 5 hour project blew up to 30 hours of work. The delta from communication surprises is evil.

Basic market mechanics: Sometimes we get asked questions where the trader should know the answer. A common trader-induced question that we get asks why trades suddenly close at market. Traders should have enough knowledge and experience to avoid such basic problems.

The delta on these issues varies, but they’re not as severe as communication issues. They can go anywhere from 20 minutes spent researching the issue to several hours.

Things that can go right in a programming project

  1. Deliver code on time. The time requirements for on time delivery are the easiest to predict. Projects start with a goal. The coder has a good idea for the amount of time required to build a working version.
    I view this as analogous to a flight crew arriving on time. The bar is pretty low here.
  2. The code works bug free the first time – no doubt your first response here is, “That’s the way it should be!”. It’s certainly the way that I’d like it to be, but it often doesn’t work out that way.
    Most software problems result from communication. When we write a scope of work and program an expert advisor, we believe that we fully understand the requirements.

    It often turns out that some of the requirements were not communicated. The product literally follows the order. It’s only when viewing the trades enter the market that the client realizes that they did not ask for something – just like the client that wanted pending orders instead of market orders. They incorrectly assumed that was understood when it was not. The experience of seeing the missing features is the only way the user recognizes the oversight.

  3. Treat people nicely – Programming is a service, but nobody wants to feel like the person on the other end only cares about money. I genuinely care about designing trading systems and helping people. When a customer does business with OneStepRemoved, I want them to trade better and to know that we care about their long term success.

    You can always email me personally if you feel you’ve been treated otherwise.

    4. What kind of surprised have you dealt with when programming your trading robot? Share your experiences in the comments section below.

Selling EAs in the MQL5 Market

by Leave a Comment

MetaQuotes decided to follow the Apple iTunes model for the forex market through its MQL5 market. The company, which is the developer and owner of MetaTrader, has taken the numerous ideas of independent web sites and attempted to put them all under its umbrella.

A lot of traders have been screaming about this, especially in the past few weeks. MT4i is a popular web site that built unauthorized plugins that modified the behavior of MT4. The plugins were wildly popular. Unfortunately, they also violated the software usage agreement that forbids modifying the software.

If you want to work with MetaTrader in a commercial environment, then you are forced to play by the rules that they set. With that in mind, I wanted to write some of the pros and cons of why you might consider using the MQL5 Market.

MQL5 market web site

A screenshot of the MQL5 market web site.

Advantages to using the MQL5 Market

I read Nassim Taleb’s Antifragile a few weeks ago. A predominant theme that he loves to hit on is finding opportunities that have little downside and infinite payoffs. Selling an expert Advisors very much fits into this category.

The cost to develop one ranges from $360-540 for a normal EA. The payoff for most of my customers is the excitement of finding out the performance of their idea. After spending the development costs, the downside is 100% capped.

The thing I like about MQL5 market is that you can take advantage of a free option: a business in a box. The trader only needs to supply an EA and to trade it. MetaQuotes literally does everything for you. They market products, track the performance, do the customer service, collect the money and write the check. Not bad.

What I also like is that MetaQuotes handles the encryption and licensing of the product. Hundreds of customers casually inquired about the possibility of selling their expert advisor over the past 5 ½ years. Very few of them decide to go forward.

The reason is that it takes tons of times (i.e., it’s expensive) to develop the security protocols and encryption to control access to the file. Integrating with a payment processor like PayPal makes it even more expensive.

Most people ask about selling EAs on a whim. It’s not worth spending several thousand dollars on.

MetaQuotes’ system eliminates the hurdle. Potentially selling your expert advisor is as simple as registering the product on their site and filling out a form.

Are your odds of success very good? No.

But you’ll at least have the opportunity to succeed and don’t have to do anything beyond filling out forms for 5 minutes. The better question is, “why not to do it?”

VAT

EU based traders might seriously consider using MetaQuotes simply because of the absurd VAT rates in most of Europe. MetaQuotes charges a 20% commission for selling trading products.

VAT in most of the EU countries that I know of are already higher than 20%. Starting in 2015, EU businesses will be required to collect and track VAT payments for each country where the end customer resides.

Tracking all of those details is insanely complex and time consuming. It’s cheaper to pay MetaQuotes the 20% commission. Doing so also avoids the hassle of reporting and collecting different VAT across the whole EU.

Cons

The disadvantages only affect people with experience running online businesses. I own a successful online business. The challenge of marketing a product online doesn’t intimidate me the way it does to someone who has never done online marketing.

My sales come from the relationships with people. I build those connections online through the web site and the company Facebook and Twitter accounts. When a customer makes a purchasing decision, it’s because they know and trust me as a person.

I see that relationship as the absolute, most critical element in running any kind of business. So, what happens if I lose control over that relationship and hand it off to MetaQuotes?

The business owner blends in with the crowd. MetaTrader 5 isn’t even that popular, but there are already 250 different expert advisors for sale. There’s slim hope of a product standing out from the crowd if setting up a page on MQL5.com is the only marketing strategy.

Customer Profile

The majority MQL5.com visitors come from countries where Western prices seem exorbitant. Yes, a handful of those visitors can afford the $1,000+ price tag that most good strategies charge. As a percentage, though, it’s tiny.

Alexa, the traffic ranking site, is only good for rough approximations. Click the audience tab and scroll the visitors by country section. Alexa says that the US and Australia only form 10% of the MQL5 site’s total traffic. Not good.

Russia makes up the biggest traffic source with 18% of the traffic. Unless you’re thinking of a Muscovite billionaire oligarch, it’s not a country that you associate with disposable income. Most of the other countries in the list have similar profiles.

Vendors on the site more than likely come from similar countries. I can’t help but think of my friend Ardi, who comes from a city in Indonesia where a couple hundred bucks of month is a good income. He’d be willing to sell a quality expert advisor for $99. That’s a lot of money to him.

There are many Ardis out there for every Aussie or Brit that wants to sell his quality product for 900 quid. The price points look ridiculous when you are 5x more expensive than the average seller.
The 20% commission and loss of control over payments doesn’t seem appealing, either. If the strategy is to put up a page and get sales that you otherwise would not have gotten, then fair play.

Any sane business owner is happy to pay a commission on sales he couldn’t do on his own.
That said, using MQL5 as the exclusive marketing strategy doesn’t seem like a good idea, either. I flipped through a couple of the EAs for sale. Not one page contained a link to a sales page with more information. You can put up your one page description, and then that pretty much has to be enough to get someone to pull the trigger.

Conclusion

One of the guys in my bible study owns an online marketing company. We were chatting about web sites the other day and how important social media is to driving traffic.

He mentioned how so many customers approach him the goal of selling their products online. The first question that he asks a new customer is, “what is your marketing strategy?”

When they usually respond, “Um, I don’t have one,” he likes to joke that it’s a Field of Dreams web site. If you build it, they will come.

Anyone with online marketing experience gets the joke. Nobody shows up at your web site, and they certainly don’t buy products, just because you have a page. It takes time and effort to build a relationship with the customer.

I don’t see that happening on the MQL5 market, either now or in the future. MetaQuotes is a large company, so I’m sure that somebody will buy something. It probably won’t be your product.
I wouldn’t put more time into the site beyond the 5 minutes that it takes to fill out the forms. It is, after all, a free option.

Expert Advisor Assumptions

by 3 Comments

Many of our clients are ordering their first custom programming projects. They know the general framework of the expert advisor that they hope to create, but they are not sure of what to ask for specifically. The goal of this post is to outline how an EA functions in relation to the most commonly asked questions.

How do I select the currency pair and time frame that my expert advisor should use for trading?

An Expert Advisor runs on the chart where the user applies it. When you wish to trade an EA on the EURUSD M15 chart, open a EURUSD chart. Change the time frame to M15. Apply the EA. All decisions will be based on the chart.

What are inputs?

Inputs are variables that the user can change without additional programming. Traders often know the indicators that they like using. The settings are often less clear. Inputs allow for tweaking or experimenting without needing to email the programmer every time that you change your mind. A more concrete example is to consider an EA that uses a moving average. Is a 50 period MA better than a 55 period MA? Making the period an input allows the trader to experiment quickly and easily.

How does my EA choose how many lots it should trade?

I assume that you wish to trade fixed lot sizes unless you tell me otherwise. Most of our SOWs include a Lots input. Any trading signals that appear would enter with the amount that you enter for Lots. Clients frequently request other types of money management. You need to explicitly state the type of money management that you wish to include or else it will not be in the expert advisor.

Why does every SOW include a stop and take profit?

Mostly because it does no harm being in there. Setting them to zero will disable each of their functionalities. The vast majority of clients want the ability to select stops and limits. Our programming templates automatically include them to help speed up the development process and to reduce costs for our clients.

What is the difference between a generic trailing stop and a breakeven trailing stop?

A generic trailing stop is what most people think of when the word trailing stop appears. It maintains a set distance from the most favorable price seen. A breakeven trailing stop is more complicated. It was covered in a previous blog post.

What is the purpose of an SOW? Why are you so insistent on pinning down all of the details?

All of our projects are pre-paid. Before we required SOWs for all projects, the expectations between OSR and the client often differed. The customer expected one thing; we expected another. It’s worth taking the extra time to make sure that everyone understands the project and what is expected.

The SOW also allows us to develop a working relationship before any money changes hands. You feel more comfortable with the purchase when you already know that we understand the project from top to bottom.

I want to trade 10 currency pairs at a time. Why can’t a single EA manage all 10 currency pairs?

A single expert advisor technically could manage multiple time frames and multiple currencies. I recognize that the idea is nice; why manage 10 EAs when you could only manage 1 EA? The problem is that MT4 EAs depend on incoming ticks to update. If the expert advisor wants to trade the AUDUSD but it’s applied on the EURUSD, the trade won’t fire off until the EURUSD receives an incoming quote. That’s not good, especially with short time frames. There’s the risk of trades firing off later than expected, trailing stops taking too long to update, etc. More importantly, the execution will always bottleneck because of the trade context is busy error.

I welcome questions on these blog posts. If you’ve always wondered how EAs work and would like your question answered on the blog, then send me an email.

MQL Organization

by Leave a Comment

All MQL expert advisors and indicators contain a few essential components. The general organization of MQL programs does not vary too often.

Files usually start with a declaration of #defines (pronounced pound define) global variables and external variables, also known as an extern data type. They appear near the top of the code to help the read gain an understanding of the variables that will run in the program. Ideally, the names of the variables and how they are organized should assist the programmer with form a general understanding of what the expert advisor or indicator might do.

The next section is usually the init() function, which is the word initialize abbreviated. This section of the code is particularly relevant to programming custom indicators. Most of the general indicator settings like declaring the indicator buffers, the colors to use and other basic features are set within this section. I use init() in every expert advisor that we build to convert the inputs into an appropriate setting for the broker’s pricing. If a client inputs a stop loss of 50 into an EA, I don’t need to do anything if it’s a 4 digit broker. I do, however, need to convert the input to work with a 5 digit broker. I run a quick check within init() to see if Digits == 3 || Digits == 5. If so, then I multiply inputs affected by that setting by 10.

deinit() is the least important section; it’s pretty easy to deinitialize an MQL file because it usually does not take up any system resources. It’s rarely used for anything important. The only uses that I ever have for deinit() are to close an open file handle or to make some sort of closing note. This is often done either on the chart directly through the Comment() function or more often by writing directly into the log file.

The start() function is the real meat of an MQL expert advisor or indicator. Whenever MetaTrader detects an incoming tick, it alerts any MQL programs. Those programs then call the start function so that it can do whatever needs doing. All trading operations, calculations, account monitoring, etc, occur within this section.

All of the other custom functions within the program appear below start(). I usually prefer to rank them in order of their importance or the frequency with which I call them throughout the program. The order of placement of functions does not affect performance in any way at all. It’s strictly a cosmetic practice that makes programming code more legible.

MQL Data Types

by 2 Comments

MQL4 supports seven data types within the program. Each type is associated with different trading tasks that programmers need to perform. The goal of this article is to provide a brief reference for when to use each data type.

The double data type is probably the most common type found in MQL programs. This is because it is the type responsible for calculating floating point numbers. Say for example, that an expert advisor needs to determine when to adjust a trailing stop. The expert advisor looks at the current price and subtracts it from the current stop loss to maintain the appropriate distance (1.3230-1.3209=0.0021). The distance requires a decimal point. When the expert advisor saves the distance to memory, it needs to save the information after the decimal point. That forces the programmer to choose a variable of type double.

Integers, or int, is the simpler version of double. Double values require a decimal place to hold the number’s value accurately. An integer, or whole number, does not have a decimal place. Integers are appropriately used when the MQL programmer knows for a certain fact that the number will never contain a decimal. An example would be if you wanted to implement a max trades feature. If the number of open trades in the account exceeds a certain number, then prevent trades. We know in advance that there is no such thing at 4.76 trades being open. There can only be 4 trades open or 5 trades open. This clearly indicates the need to use an integer.

Datetime values are just what they sound like. They represent both the date and time. More specifically, a datetime variable represents the number of seconds that have elapsed since January 1, 1970. This is where it gets a little tricky. The number of seconds that have elapsed is actually an integer. Datetimes store integer values but then associate them with a date and time.

A value of 0 would indicate that the time is 00:00 on January 1, 1970. A value of 60 stands for one minute later at 00:01 1/1/1970, and so on. One benefit of knowing that the datetime type stores information as integers is that you can easily determine the amount of time that happens between an event. If the event starts at 15:35 and ends at 18:12, you can simply subtract 18:12 – 15:35 and wind up with the number of seconds between those values. That information can then be used to determine the number minutes/hours/days between the two events.

The color data type, not surprisingly, holds color information such as black, yellow, red and so on. Much like datetime types, color also uses integers to store the information. The difference, though, is that extracting the color information from the integer is not at all obvious. Increasing a color type from 32768 by one will not necessarily make it more or less green. Colors use the integer information to retrieve the red, green and blue components of the color in hexadecimal format. Explaining hexadecimals is well beyond the scope of this article. It’s unlikely to come up in your MQL programming. I’ve been doing this for over five years and only came across one project that required manipulating a color in way more complicated than alternating between two set colors.

A string is anything that resembles a word or sentence. It always uses quotes to contain the information. My favorite use of strings is to gather information to display on the chart or in a log file whenever I need to debug an expert advisor.

Char is the final data type. It’s so closely related to a string that I wasn’t even aware this type existed until I looked up information for this article. If we study the word “trade”, then we will find that it is composed of the five characters t, r, a, d and e.

A final note on data types. There are two ways that types are held in memory. An extern variable is one that shows up in the inputs screen whenever an expert advisor or indicator loads. Static variables are the opposite. They remain within the MQL program and never visible outside of it.

Problems Converting Between MetaTrader 4 and MT5

by 4 Comments

I recently completed a project that converted code I wrote from MetaTrader 4 to MT5. This post should provide some insight into the difficulties of moving back and forth between the two languages.

1) Learning about MT5 objects; their methods and members

MT5 is a quite a bit different from MT4. Compared to MT5, MT4 is a rather quaint non-object oriented ‘C’ like programming language with a limited set of statements. MT5 adds the object oriented language parts. In addition to objects, methods, and members, MT5 also adds various structures, enumerations and other constructs. Because of this I had to find the MT5 way to accomplish what was in the MT4 EA. This meant considerable time reading the language documentation, poring through newsletter examples and forum discussions when logic problems cropped up that were not related to syntax, coding, and debugging. One area in particular I had trouble with is the way historical data is returned within arrays.

2) MT5 combines multiple trades into one trade.

This is not so much a problem if the combined trades use the same stop loss and take profit. However, in this particular EA, the two trades placed on the same trade signal used different take profits. This required the storage of the first trade parameters and then monitoring when the bid or ask dictated a partial close of the now combined trade.

3) The Strategy tester had to work on an EA that accessed multiple currency pairs; and the creation of indicator handles.

The primary reason for converting this particular EA from MT4 to MT5 was to take advantage of the MT5 strategy tester, which permits backtests in multiple currency pairs. In its initial configuration the EA would test the various currency pairs to find the most favorable pair to work with every tick and then access and test numerous indicators (based on currency pair and chart periods) on the selected currency pair.

Through trial and error it was discovered that the removal of old indicator handles and the creation of new indicator handles every tick taxes memory and CPU resources to the point where the EA would exhaust those resources. After attempting various tweaks to reduce demands on these resources, I began to look for ideas from outside sources (on the Metatrader forum and from various blogs). After some forum and support ticket exchanges I decided on some approaches to try, including the use of retry loops and method calls to test for the existence of the necessary data. This worked only to a degree (with a reduced number of currency pairs).

I tried a few other approaches before finally deciding to preallocate all indicator handles in the initialization part of the EA. I had to allocate a different indicator handle for each currency pair and with each currency pair for each time period used. In the actual strategy test this greatly lengthens the EA’s initial load time but it is the only scalable solution.

4) Getting the historical data requires some manual setup.

One of the issues that delayed the discovery of the best solution in part 3 (above) is that the historical data for the various currency pairs was not present on the client terminal that the strategy tester was accessing. Many of the errors having to do with loading data for the indicators was due to this lack of data and had nothing to do with the other solutions being tried. While I feel there may be a more elegant way to programatically request this data, I have not come upon how this is done yet.

I ended up manually loading a chart of each currency pair, turned off chart autoscroll, then back spaced over the period of time being tested. Not only did I have to load a chart for each currency pair, I had to change the chart period to the 4 or 5 different periods that was to be used by the various indicators. I also had to add that indicator to the chart. In this way I was able to force the loading of historical data needed in the strategy test. Had I not been testing within a small window over a recent period of time this could prove to be a very burdensome task.

Function is not referenced and will be removed from exp-file

by Leave a Comment

The MT4 compiler likes to issue warnings whenever some components of the source code are not used. When you see this message, it means that MetaTrader decided to remove this unused portion of the code from your compiled file.

The fact that the MetaEditor labels it a warning is a misnomer. It does not impact your code in any negative way at all. It actually speeds up the file by removing unnecessary portions.

Color Changing Indicators in MetaTrader

by 5 Comments

Many custom indicators in MetaTrader use lines that change colors to indicate a change in trend or market condition. Those types of indicators are among the more common requests that we get for programming Expert Advisors. Unfortunately, these indicators often present problems. What you see on the chart is not necessarily what the indicator says.

Take a look at the image below or click the link to view it in full size. You'll notice that I included the data window for the indicator, which is SuperTrend. As we walk from left to right, the data window does not suddenly shift from Trend Up to Trend Down. Instead, it shifts with an in between period where the trend is both up and down.

MT4 Color Changing Indicator

While the visual effect is immediately obvious, the numbers do not clearly indicate the indicator's condition. In fact, it frequently happens where the indicator entirely misrepresents its true calculation. If the last bar was only green and the indicator says that the current bar is red-green, then we can safely assume that the indicator is switching from green to red.

When the indicator says it's red-green and the past bar was red-green, it gets more tricky. We are forced to keep looking back through time until a "clean" red or green value appears. This enables us to capture the indicator's real value.

It does not, however, make for a happy trader. Consider the case where the indicator plots red-green, red, red-green, red. Because of the way MetaTrader draws lines from point to point, the indicator actually appears as a solid red line – a long, beautiful sell signal, right?

In fact, its true calculated values are red, green, red, green. This glitch can make for some ugly surprises. This is especially so when traders expect to ride a down trend and the line appears as solid red, but the indicator (and thus EA) keeps flip flopping on the trade direction. You have to keep this in mind when building EAs around custom indicators.