Uncategorized

Strategy Decay

Ernie Chan wrote a great blog post today on the life and death of a strategy. He talks about some of my favorite points: KISS (Keep It Simple, Stupid!) and the importance of viewing real performance relative to negative performance.

If you like this blog, then you’ll love his. It’s very similar. The only difference is that Ernie tends to write with more assumptions about his readers’ math skills.

High Frequency Forex Seminar

One exciting opportunity popped up while I’m in Dublin next week. Best of all, it’s free and open to the public. If you’re in the neighborhood and would like to discuss trading in person, I’d love to meet you.

Trinity College Dublin invited me to present a graduate level seminar to MSc students in Finance and Alternative Investments on Wednesday, February 8, at 6 pm. The seminar will be hosted in the MBA room, which is on the second floor of the business school. The topic will be high frequency market making in forex.

Topics for the high frequency forex trading seminar (about 10 minutes per subject):

Market making versus price taking
Comparing frequency to expectation. The more you trade, the more you make
Liquidity risks and self-feedback loops
Technical approaches and limitations

High Frequency NinjaTrader Strategy

I’ve been working on a high frequency trading system for NinjaTrader on behalf of a long term client. My live account is with MB Trading. Rather than placing market orders and paying a commission, I changed the order types to limit orders. We want to receive a small commission for the market making strategy rather than paying a commission to accept the displayed prices.

MetaTrader suffers two major disadvantages that make NinjaTrader a superior option for high-frequency trading. MT4 does not offer charts lower than the M1 time frame and the trade context is busy error prevents multiple charts from running simultaneously. NinjaTrader is complex enough to where I can control most details, but simple enough that I don’t need to invest hundreds of hours to test an idea. After extensively testing the strategy on M1 charts as a price taker, I feel very confident that the strategy is sound. The only issue now is determining whether or not whether taking a passive (i.e., market making) approach will result in enough fills to make the strategy worthwhile.

The first issue that I came across wasn’t with NinjaTrader; it was with MB Trading’s API. The strategy worked fine on the simulation account, which only routes orders to NinjaTrader (NT). NT then makes guesses when fills would occur. The goal of that phase was not to test the strategy. I only wanted to test the programming to make sure that it worked properly.

100 trades went off without a hitch in the Sim account. The strategy only made it through 2-3 microlot trades on the live account before the pending orders hung. NinjaTrader pending orders pass through 3 states before they actually hit the market. For the programmers out there, these are the OrderState properties of IOrder objects.

Pending submit – the strategy sent the order to the broker and is waiting to hear back
Accepted – the broker acknowledges receipt of the order, but is still placing the order into the market
Working – the order is available for others to trade

The strategy updated orders on every tick. What often happened was that the pace would go far too quickly, creating a major communications backlog during fast markets. NinjaTrader never threw an exception. The only evidence of a problem was that I would see a hanging order with the PendingChange property. The inconvenient solution was to exit NinjaTrader and reload everything.

I figured that perhaps that the managed order state caused the issue. I changed my approach to unmanaged orders, but that did not make a difference. I eventually came to the realization that the MB Trading API cannot handle more than one order every few seconds.

The strategy found the sweet spot after changing from tick to second charts. Updates of 6 seconds or longer seem to give the MB Trading API enough time to update wihle still preserving something of a high frequency approach. Any trades that need to run faster than that threshold at MB Trading need to use the FIX protocol.

The other element that drove me crazy is that NinjaTrader limit orders automatically delete themselves once per bar. I nearly tore my hair out, and I don’t have all that much hair, for several hours trying to figure out why orders deleted themselves automatically. Many people identify with the school of hard knocks approach to learning. I’m as thick headed as most. I figured out the cause when I revisitied NinjaTrader’s online documentation and discovered a limit entry method that allows good till cancelled (GTC) orders.

The speed problem also manifested with overfills. An overfill is when a strategy requests to cancel a pending order, but the broker fills the order before the cancellation takes effect. The biggest concern with overfills is that NinjaTrader automatically disables a strategy and exits positions at market when an overfill occurs. The only way to programmatically prevent this is to change the entry methods to an unmanaged approach.

The easiest way to develop for a high frequency strategy in NinjaTrader (but not ultra-high frequency) is to use managed orders. Whenever an exit is needed, place the limit entry in the opposite direction. NinjaTrader takes care of placing the exit order for the open market positoin. Limit the updates to every handful of seconds. It allows the broker API to catch up and helps avoid the problem of overfills.

Develop Trading Strategy

One common element unites 99% of the trading strategies that we’ve developed. The final product does not match the initial design.

All programming projects proceed in one of two ways. You can either develop a project from the top-down, or you can build from the bottom up. The former option represents the old way of thinking, especially within the university system. When professors assign a project, the requirements never change. The assignment remains static.

Real life rarely cooperates like that. Instead, human beings change their minds. They see the work in progress and decide that a minor tweak might dramatically alter the outcome. The expert advisor programmer is forced to constantly adjust to the moving goal posts.

I always tell clients that they will change their minds about some of the rules during the process. Most insist that they will not. They do, after all, expect them to work.

Then, the rules inevitably change for most of them. Theory goes out the window. The traders discover the real issues that expert advisors face in the market.

Programmers developed a new methodology to accommodate the constantly shifting rules. It’s called agile development. This problem doesn’t only occur with forex traders. It also happens with businesses making web sites, database programmers storing information, and really any project that involves human beings. People are prone to changing their minds. It’s our nature. Agile development is the programmer’s acknowledgement that getting upset at people for changing their minds is not entirely reasonable.

The agile approach breaks a project into time periods with a ranked list of objectives. The team sorts out which objectives are most important and do-able within the allotted time. The period starts and everyone rushes to accomplish as much programming as possible. Everyone regroups after 2-6 weeks to evaluate the progress and to deliver the updated version to the client.

Most expert advisor projects only last a few weeks, so a precise copy of the agile approach would not fit very well with developing a trading strategy. We flip the strategy development approach on its head by assigning ranked priorities with a deadline of 1-2 business days. The goal is to facilitate communication, which is inevitably the most challenging obstacle in this field. We also make the process less formal to speed things along.

The quick turnarounds lead to a series of yes/no answers. The client either gets excited that it works properly, or more likely, feels that we did not understand him correctly. The honest goal of this approach is to get the user away from explaining the idea for the 1,000th time in the same, identical way. The last explanation did not help the project move along. Client feedback from working software leads the programmer into understanding the concept on his own without making the client feel like he’s repeating himself.

I like to think of it as the client steering the driver’s wheel while the programmer controls the gas pedal. The strategy development process only works when the steering and power move in synch.

Overcoming expert advisor programming challenges

When clients feel like an expert advisor has not made the desired progress, the first step is to assign goals one at a time. Too often, we receive reports with 20 different bugs. The emphasis on priority falls away as the desire to “fix everything and fix it now” drowns out the details of each request. I always like to take a step back and reaffirm our commitment to developing the strategy, but we can only handle so many issues at a time.

I have a very patient client in New Jersey (I never though I’d say “patient” and “New Jersey” in the same sentence”) who ordered a lot of custom elements in a basic expert advisor. Most of the complications arose from the fact that I had to remotely access his computer with LogMeIn to do the programming. His indicator only has one license. Purchasing a second for the EA programming was not economical.

The requirements to remotely access the computer and the amount of custom code turned a simple project into a complicated one. The way I addressed the issue was to remove almost all of the custom components and to replace them with code from our usual template. Now that the client sees that the “strategy” stuff works, he found it much easier to write a checklist of bugs to fix. More importantly, seeing critical components of the code working reinforced his confidence in the company’s programming ability.

Although he remained patient throughout the debugging process, I could hear the flagging confidence in his voice before we switched gears. Changing the direction of the project showed him that the EA actually does work; it’s the little pieces and how they fit together that are the problem. Of equal importance was the fact that he felt confident in my ability to deliver the results that he wants.

The client gets full credit for releasing control of the debugging process, even when he wasn’t entirely comfortable doing so. The idea of papering over the custom steps with pre-programmed template code struck him as a step backwards. He nonetheless followed the programmer’s lead, letting him drive the process with himself providing feedback where it was needed.

Both parties agree that we’re once again heading in the right direction. The client sees obvious progress in his expert advisor. The programmer is able to manage the debugging process in a methodical, organized manner. Everyone is happy.

Realistic strategy development

The most realistic way to develop a strategy is to acknowledge that you don’t exactly know where you’re going. You know it’s likely to involve a certain indicator and that it either trends or ranges. The best way to prepare for the journey is to acknowledge in advance that it is indeed a journey. You more than likely will make changes, most of which you cannot expect.

It’s 100% certain that something is going to go wrong when you program your first project. It’s your strategy, but it’s really a two-man team that builds it. Make sure to pick someone that knows what do when problems come up.

Before you choose your programming partner, make sure it’s someone that you actually trust to get the job done. You ideally want to find someone that offers strong communication skills and appreciates the importance of a timely reply. Organizing the project and overcoming unexpected challenges helps everyone get back to their true passion of trading.

Save MetaTrader Backtest

Saving a backtest in MT4 is very straight forward. Look for the “Report” tab at the bottom of the tester area. Right click anywhere on the report itself. MetaTrader only allows for saving reports as .htm files.

How to save an MT4 backtest report

Stop & Limit Order

Stop and limit orders are direct opposites. A limit order is jargon for “better price”, while a stop order means “worse price”. Many find it especially confusing that a stop entry order uses the same terminology as a stop loss. Technically, they are the same thing. A stop entry and stop loss are both prices worse than what you would get if you accepted the current market price.

Stop entry versus limit entry

Stop entries are used with momentum or breakout strategies. The theory is that if the price moves up, then it will be more likely to continue moving up. The trader loses out on the difference between the price at the time he decided to trade and the actual entry price. What he hopes to gain is the extra information that the price has moved, which might imply a higher probability trade.

A price that is worse the current market price for buying in the future is up. So, buy stops go above the current market price. Sell trades receive a worse price in the future if they wait and the price goes down.

Limit entries take the opposite approach. Market often tend to wander. They very rarely shoot off in a single direction without wiggling a little bit up and down. The idea behind a buy limit order is that you think that the price will increase in the future, but that you might be able to pick up the fx pair at a better price than what you’re seeing now. A buy limit goes below the current market price. A price improvement for a sell entry requires a price increase, so sell limit entries always go above the current market price.

Stop loss exit and limit exit

Most people find this part a lot simpler. Here, a stop loss means you lose money. The buy trade that you opened is going down. If it hits your stop loss, then the loss is realized. Stop losses go below the entry price for a buy trade and above the entry price for a sell trade.

Limits are the opposite. An open long trade is a bet that the price will increase. If it hits your limit exit, it means that you’re satisfied with the amount of profit on the table. Long trades place the limit above the entry price. Short trades place the limit exit below the entry price.

Max spread and slippage

Many novice traders mix up the distinction between the slippage and max spread. The spread refers to the trading cost. Designating a maximum spread forbids an expert advisor from entering orders whenever the cost of doing so exceeds a certain threshold.

Max spread

Forex spreads often widen around news events. It’s frequently a great deal of chaos where the end result is not much different from where it all started. Many traders find it preferable to sit out these events. It’s better to miss a trading opportunity than to pay an arm and a leg for it.

Max slippage

Slippage controls the execution of the order. MetaTrader offers a unique feature in the OrderSend() command called slippage. Most market orders are treated as pure market orders. It’s treated as a command to the broker to execute the order without regard to the price paid. The maximum slippage pulls back the reigns a little bit.

Say that the market price is 50 and an MQL program sets the maximum slippage to 2. The MetaTrader broker knows that it may only execute the price within a range of 2 pips from the requested entry price. Either the price 50, 51, or 52 will do.

The difference between maximum spread and maximum slippage

The easiest way to distinguish the two items is to remember the following two questions.

Does it look like I’m about to pay too much to enter this trade? If so, I should use the maximum spread to prevent expensive trades.

Am I worried about the broker abusing my market order request after I send the order? If so, I should use the maximum slippage setting.

OneStepRemoved.com uses a hidden maximum slippage variable in our expert advisor programming template. We usually set it at 2 micro pips. You can ask us to make it an external variable upon request.

Forex spread

Spreads are the true cost of trading forex. Opening a position results in an immediate loss. Profits only occur whenever the price moves beyond the gap, known as the spread, between the buy and sell prices.

Bid and Ask

The screen shows two prices, the bid and ask. Novice traders understand these terms more intuitively by thinking of them as the “sell” and “buy” prices, respectively.

You can think of electronic forex trading as a form of sohpisticated haggling. Everyone may agree but that the price is generally X, but everyone also wants to try to shave a few pips off of the price. Traders want a discount.

Disagreement in the marketplace causes the bid and ask to exist. If someone wants to buy euros, someone else wants to sell them and they both agree on a price, a trade occurs. When a trade does not occur, which is most of the time, the spread reflects disagreement among the group.

A trade entering the market generally causes the price to move. Consider an example where someone buys dollars. The dollar should rise in value to reflect the latest transaction. The buy trade removes the previous ask from the market depth. The next ask becomes the best available price for purchase.

The other side of the market, the bid, sees the price adjust. Traders typically responds by adjusting the bid in the same direction. The traders offering the bid expect the price to rise. That is, after all, the reason that you’re able to sell. You’re selling what they’re buying and you’re buying what they’re selling.

The movement of the price causes a reaction. The people behind the bid still want to buy at a price better than the market using a limit order. They also realize that the price is moving away from them. A common reaction is for the bid to adjust approximately the same distance as the movement in the ask. The adjustment in the spread is what keeps it relatively consistent.

Spread mark up

You only pay the spread once. When you enter a buy trade, you enter on the ask. The only way to exit that trade is by selling at the bid, a problem that occurs immediately upon entry. The movement of the ask price from that point forward is completely irrelevant. The ask could drop or explode upwards. It doesn’t matter to your trade. Your trade can only exit on the bid. It’s the only price that counts when you’re already long.

One unique aspect of forex trading is that the brokers bury their commissions in the spread. Say, for example, that the wholesale price of USDJPY is 76.480 on the bid and 76.485 on the ask. Brokers read this type of quote 76.480 at 76.485, with the bid coming first and the ask quoted last.

The broker then takes the prices and mark them up like any middle man. The mark up varies between brokers. Most charge an additional 1 to 2 pips per transaction.

The broker faces three options when marking up the spread – he can add it to the bid, the ask or he can split the difference between the two. Current market conditions dictate which option is the most suitable.

Although it seems like a good idea to tack half of the mark up on both sides of the bid ask spread to eliminate guessing incorrectly, this doesn’t usually work out well for the broker. Order flow comes in very unevenly and usually with a strong bias in one direction.

I posted in the past about forex trader sentiment and how lopsided the trades of retail traders can be. If the forex broker sees that 75% of the GBPUSD orders coming in are to buy, then it makes sense to heavily weight the spread cost to where all of the business is.

Consider an example where the next 100 lots traded will be 75% buys and 25% sells with a 1 pip markup. If the 1 pip mark up on the spread applies to only the bid, then there would only be 25 lots where the broker earns his pip. That’s the dumbest thing he can do in that situation. He could evenly split the markup with 0.5 pips added to the bid and 0.5 pips added to the ask. He would make 0.5 pips on 25 lots and 0.5 pips on 75 lots for a total of 50 pips. The best scenario for the broker is to let the traders going short trade for free and to charge all of the traders going long the full pip. He doesn’t make anything on the short flow, but he makes 75 pips on the long traders. It’s clearly in his best interest to full mark up the spread in one direction whenever the order flow is uneven.

Traders are on the losing end of spread markups. It completely eliminates pricing transparnecy. Most brokerages allow you to review within their statements the break down of profit and loss into categories. Commissions and trading costs are most definitely found on every trading statement from stocks and futures brokers. Not so in forex. You can guess trading costs and that’s it.

Spread mark ups also eliminates trading opportunities. A client that I visited in Dubai in 2009 absolutely loved FAP Turbo. He swore up and down that it was the greatest EA that he ever bought. His only complaint was that the EA’s performance depended entirely on the broker. He often complained the most about FXCM because they charged the largest mark ups.

FAP Turbo worked by waiting for unlikely, fleeting moments of volatility in ranging pairs like the EURGBP. When the price corrected, the EA sought to take the smallest of profits – something on the order of 3-5 pips. The problem stemmed from the fact that the mark up would cause the take profit not to hit. The wholesale price would reach the limit, but because of the spread mark up, the price he was paying often missed his limit. My client preferred to pay a direct commission. Even though the cost was identical, the increased probability of a successful trade dramatically affected the net outcomes.

Spread reflects risk and liquidity

Each currency pair usually has its own spread. The spread of a currency is a function of its risk and liquidity. When traders look at an exotic currency pair like the ZAR/JPY or USD/TKY, the first thing that they notice are the spreads that seem incredibly wide when compared to the pairs that they’re used to trading. That’s because hardly anyone trades those pairs.

The lack of interest means that anyone that makes a market in an FX pair needs to receive sufficient compensation to make it worth their while. Sufficient compensation here means charging you a fat spread. The low trading volume, which is called liquidity, creates a problem with risk. The largest financial instruments in the world like currencies and stock indeces usually only move a few tenths of a percent per day. Instruments with less trading activity almost always exhibit higher volatility. Prices are more suspeptible to violent moves when fewer people participate in a market. The people making a market in those currencies charge a higher premium as a consequence.

Trading is a marketplace

It’s important to remember that the numbers on the screen are not computer generated. They are real prices that real traders offer in the market place.

The word trade accurately describes the transaction. Someone must agree to a trade before it can occur. Clicking a button to buy does not assure that you get to buy an unlimited quantity at the price displayed.

When does an Expert Advisor need a dll?

Most Expert Advisors do not need a DLL. Although programming in MQL is fairly limiting, it suffices for 99% of expert advisors. DLL programming for MetaTrader is only appropriate whenever MQL does not offer a capability that the EA needs.

What is a DLL?

DLL stands for dynamic link library, which is not the least bit helpful in understanding what it does. Conceptualize a toolbox full of tools. The DLL itself is the toolbox. The “stuff” in the DLL is the tools that the expert advisor or indicator uses to make decisions. The MQL simply decides which tool to use at which time.

DLLs only work in MetaTrader if

MetaTrader allows Expert Advisors the ability to control whether or not MQL may use a DLL. This option is the first one appearing in the screenshot above. The indented option to “confirm DLL function calls” is a subset of this option. Whenever MT4 wants to use a DLL, it will prompt the user with a yes/no message box every single time that the DLL is used. It’s very annoying. MetaTrader would be better off without this option as it confuses most users.

When to program a MetaTrader DLL

Speed

I often encounter the need for a DLL whenever I plan to do mathematically intensive programming. MQL runs extraordinarily slowly for a programming language. Whenever an EA wants to process several million calcualtions, it tends to freeze. MetaTrader suspects that it fell into an infinite loop and chokes.

Moving those calculations into a DLL passes that responsibility for calculating the numbers to a more powerful language. MetaTrader only likes C/C++ or Delphi, which are referred to as higher level. The MQL programmer can control the physical hardware in order to optimize performance if it’s required. That would ususally be overkill. A generic dll program written C will run many, many times faster than the same code in MQL. We only program DLLs at OneStepRemoved in C or C++ (read as C plus plus).

Sell products in MetaTrader

MQL is a programming language designed entirely for trading. Anything that extends outside the sphere of trading, such as connecting to a database or reading information from a server, is not supported.

A business that wants to sell an Expert Advisor absolutely needs a DLL. The higher level languages easily support web operations that can authenticate who is and is not allowed to use a file.

More importantly, high level languages offer dramatically stronger encryption options. Decompiling an MQL file, which means breaking it open to steal the contents, only costs about $10-15 if you do enough Google searches. Cracking a DLL file takes much more work. Even an unencrypted file might take a programmer a few work full works days to open. Cracking an encrypted DLL could take months if the crack to the encryption is not known. Although anything that can be encrypted can theoretically be deencrypted, it becomes a matter of scale and resources. Would cracking your file warrant someone’s full time employment for several months? The answer is no for all but the most sensitive of material.

Bridge multiple instances of MetaTrader

Occasionally, traders design Expert Advisors that need to receive information from multiple brokers at the same time. Broker arbitrage and trade copiers fall firmly into this category. In the case of the trade copier, the MT4 account that wants to copy another account needs to know what is happening. The DLL stands in the middle between them as a path to pass information back and forth.

Many other problems are most appropriately solved with a DLL. If you aren’t sure if it makes sense for your situation, then please contact me.

Euro collapse

I’m writing this over the Atlantic on my way home from Dublin. I’m absolutely convinced that the euro is at the start of its death spiral.

Many of you know that I travel to Ireland about once a month to work with Traders Now and their q algorithm. I love the math, the programming and all the strategizing that goes into developing a fully automated trading system. One of my favorite parts, however, is the Friday night before I leave. Kieran and I play a long game of poker with his buddies.

The first few hands did not treat me kindly. I was busy chewing my sour grapes when Kieran mentioned one tiny, little thing about the euro. The place exploded with rumors and opinions. The poker game stopped entirely for fifteen minutes.

Mind you, this is not a poker table full of financial professionals. It actually seems to be a good cross section of Irish society. We had everyone from a plumber and retired taxi driver to a multimillionaire real estate developer sitting at the table.

The first thing that came up were the rumors of what would happen if the euro did in fact collapse. Two out of the nine players had independently heard rumors of the government printing Irish pounds and stockpiling them. They also mentioned a rumor of plans to completely lock down the country for three days in the event of a collapse. Banks close, businesses close, the airport and ferries close. Life goes into suspended animation.

These are wild rumors, mind you. I’d argue that it’s really not important whether they’re hard facts or bold faced lies. Perception is reality. It’s becoming obvious to me that the foundations of the fiat euro are growing shaky.

Modern currencies depend entirely on faith in the government in order to continue. They are, after all, simply scraps of paper and electronic entries in a bank’s database. The value of the paper itself is negligible. When ordinary Joes are beginning to question the status quo and whether or not the system is viable, the jig is up (Irish pun intended).

As more people question the system, a small percentage of them begin to take real action. The guys at the poker table seemed to like the idea of Swiss francs in spite of the Swiss National Bank’s peg to the euro. They also mentioned the US dollar as a second best option. I don’t necessarily agree with that in the long run, although I’m sure that will be the short term affect.

The small increase in euro selling begets more people worrying about their savings and assets. They see the price moving adversely, which spurns them into taking action, too. The effect is that the rumors and fear cascade into an avalanche of collapse. It takes on exponential growth as anyone with anything of worth tries to offload it in an attempt to protect themselves financially. Actually, it looks just like the bank runs that Latvia recently experienced.

PS: I lost badly at the poker game. 25 hands that never came out on top, combined with players who will never be bet out of the game makes for a bad night. I walked away with EUR 50 loss. Too bad that’s still worth $65.