optimize Archives - Algorithmic and Mechanical Forex Strategies

How to Use Machine Learning in Your Trading

April 5, 2015 by Shaun Overton 2 Comments

Machine learning presents many unique and compelling advantages for traders looking for an edge in the market. Just in the last year we have seen a huge amount of resources from the world’s top hedge funds, like Bridgewater Associates, dedicated to exploring these techniques.

While using machine learning or artificial intelligence seems incredibly complex and difficult to implement, there are still ways to leverage their capabilities without needing a PhD in math or science.

In this post, we’ll go through 3 different ways that you can use techniques from machine learning to improve your own trading.

Indicator Selection

One of the most important decisions is deciding on which indicators to use to trade. Whether you are a technical or fundamental trader, or you just use price action to trade, your success is going to be largely dependent on the indicators that you use and how you interpret them.

Luckily, there are many different methods for selecting your indicators and this is known as “feature selection” in the machine-learning world.

Using a Decision Tree to Select Your Indicators

Decision trees are very versatile algorithms that have the benefit of being easily interpretable. Given a large dataset of indicators and the price movement of the asset, a decision tree will find the indicators, and indicator values, that best split the data between price increases and price decreases. Indicators closer to the top of the tree are seen as better predictors than those closer to the bottom of the tree, and following a particular branch will allow you to easily find interdependencies and relationships between the indicators.

The decision tree will also give you a set of rules that you can use to trade based on those indicators, but you must be sure to properly prune the tree and test for overfitting.

The decision tree is a powerful, visual tool that can help you decide which combinations of indicators to trade and at what values to trade them. You can find a tutorial on how to build a strategy with a decision tree here or for a more general guide, in R here is a good resource.

Optimization

Once you have the basis for your strategy, the next step is optimization, or choosing the correct parameter values to maximize your chance of success. Many strategies have a wide variety of parameters, such as indicator settings, the entry and exit conditions, stop loss and take profit levels, and position sizing, that make “brute force” methods of trying every single combination extremely difficult and time consuming, if at all even possible.

Solving these types of problems is another area where machine learning excels.

Optimizing a Strategy Using Genetic Algorithms

Genetic algorithms mimic the process of natural selection by creating a unique set of “child” strategies that contains a mixture of the best “parent’” strategies, with a chance of random mutation.

The process begins by encoding your strategy into an array. For example it could read as something like:

Indicator 1 Period

Indicator 2 Period

Buy Condition

Sell Condition

Risk:Reward Ratio

Where a 50 – 200 period moving average cross, with a 50 pip to 100 pip risk-to-reward ratio would be:

50 – period Moving Average

100 – period Moving Average

Indicator 1 crosses above Indicator 2

Indicator 1 crosses below Indicator 2

50:100

You would then generate a large population of strategies with random variations of these parameters. These strategies all have different combinations of moving average periods, entry and exit conditions, and risk-to-reward ratios.

Next, you would test this population by running each strategy over a test set and selecting the top strategies based on a performance metric of your choice.

Finally, you randomly combine the traits of the top strategies, with a small chance of “mutating” some of the parameters, to create a new generation of “child” strategies. You then repeat the evaluation procedure and once again mate the top performing strategies from this new generation. This leads to a survival of fittest scenario where only the top strategies “survive” to pass along their genes to the next generation

Repeat this process a large number of times or until a certain performance criteria is reached and you are left with a very robust strategy built from generations of the best performing strategies!

You do have to make sure that you select an appropriate performance metric (such as risk-adjusted return) and always test the final strategy over data that wasn’t used to build the strategy to ensure that you aren’t overfitting to a particular data set.

This is a very powerful and robust method that has been successful in a wide variety of applications, including the world of trading. You can find a more detailed description here and a tutorial on how to implement it in R here.

Live Trading

One of the more attractive aspects of machine learning is having an algorithm that is able to learn and adapt to changing market conditions. However, this creates a “blackbox” strategy that, if you do not completely understand how the algorithms work and thoroughly tested it yourself, is very difficult to trust on a live account. Not knowing when or why a strategy is entering a trade can be a scary proposition.

However, there are ways to get the benefits of an intelligent, algorithmic approach while still maintaining transparency and understanding in your strategy.

Association Rule Learning

Association Rule Learning is the process of deriving a set of clear, understandable rules from the patterns uncovered by a machine-learning algorithm.

Algorithms, like the Apriori algorithm, search a dataset of indicators, indicator values, and the resulting price movement to produce a set of conditions; basically “if-then” statements, that lead to the highest-performing results. However, it’s still difficult to know exactly where these rules are coming from, the Apriori algorithm requires a fairly large number of parameters to be tuned and this process does not lend itself well to changing market conditions.

With TRAIDE, we took the process one step further and allow you to see the patterns found by an ensemble of machine-learning algorithms, from which you can create your own trading rules. These rules are then easy to implement and adjusted to changing market conditions, all without requiring any programming or mathematical experience. You are able to get the benefits of using machine-learning algorithms to trade while still maintaining complete transparency, an understanding of your strategy, and including your own domain expertise in your trading.

Using machine learning and artificial intelligence to find an edge in the market does not need to be exclusively owned by only the largest financial institutions. As this technology becomes more accessible and these techniques more common, you too can use machine learning to improve your trading.

Tad Slaff

CEO/Co-founder

Inovance

Automated Trading

December 28, 2012 by Shaun Overton 2 Comments

Nathan Orange contacted me in early 2012 looking for advice about automating a grey box strategy. Through the course of our conversation, it turned out that he was a profitable trader with a multiyear track record. Nathan has gone on to found his own forex signal service at Global Trend Capital.

Nathan conducted this interview with the intention of informing his readers about automated trading. You’ll have to pardon the vanity of publishing his interview of me, but I believe it’s useful for my own readers.

(Nathan):
Shaun, good to talk to you again and I appreciate you taking the time to discuss what I consider a very important topic. Before we jump into the specific questions, let’s fill everyone in on your background.

(Shaun):
I led the sales effort for the Sentiment Fund at FXCM, which was a fully automated strategy based on the market positioning of retail clients. I needed to understand how it worked in order to answer client questions. That interaction with the systems desk gave me access to one of the tiny handful of people in the forex industry that really knew anything about systems trading and analysis.

I tried trading manually during work hours, but as a broker, it was really difficult to manage trading accounts and to squeeze in 100+ attempted phone calls per day. I also suffered from the usual sob story that every trader endures. Account #1 blew up in 3 months. Account #2 blew up in 6 months. That was the first $5,000 thrown down the pit.

Technical analysis with its trend lines and other tools are hocus pocus pseudo-science. I traded like that for nearly a year, but I never felt confident or comfortable with the idea that subjectively drawing lines on the chart leads to any useful information.

The idea of quantitatively defining a strategy allows for testing and analyzing an idea to determine whether or not it really held any merit. The first non-technical analysis idea I had was to look at unusually big bars with the idea of fading those moves. Access with the FXCM Systems desk helped shape my idea from a subjective idea like “big bar” into a mathematical parameter like “standard deviation”. They also explained trading platforms to consider and recommended a few programmers to help develop the idea.

My experience working with programmers was uniformly terrible. I tend to dive into projects, so rather than depending on the clown-car brigade to half-develop my ideas, I wanted ultimate control over the development process. That eventually led to 20+ hours per week programming and analyzing strategies at home after working all day. The system design bug bit hard and never let go.

(Nathan):
One of the most common concerns when discussing back-testing is over-optimization. From your perspective, what are some of the common mistakes that most system developers make? I have my own list, but we can discuss those further when we turn the tables.

(Shaun):
The basic kernel of the idea either has merit or it does not. There is no secret set of magical inputs that turns a bad strategy into a good one. Bad inputs, however, can turn a good strategy into a bad one.

Optimization fails to differentiate between “profitable” and “good”. I flog this dead horse constantly, but the most confusing thing about trading is that you can trade by flipping a coin and setting a 50 pip stop, 50 pip take profit and actually come out a winner – sometimes. Most of the winners will show small profits. A tiny handful of them would show gigantic profits purely as the result of luck. What’s worse is that most of the profitable traders will actually believe that they are the reason for their success when it’s really just dumb luck.

Optimization is usually the process of finding the luckiest accidental winner. It’s no wonder that optimized strategies almost universally fail going forward. The real task is to distinguish between ideas that are inherently non-random versus strategies or expert advisors that coincidentally make money from a random process.

(Nathan)
Based on your experience and knowledge, if someone sends you a system to code can you quickly determine potential issues with their logic, or even over-optimization red flags? For example, you might a get a system a trader or hedge fund wants coded that has so many specific variables that you know immediately it won’t be robust. I can usually spot these issues from my own system development experience, but from your perspective as a coder is it fairly easy to recognize?

What do you do in those cases? Are most clients bull headed, avoiding any feedback or are they more open minded to listen?

(Shaun):
We see our primary role as that of a construction worker. If you want to build an ugly house, that’s your affair. On the flip side, if you solicit my opinion, I won’t hold back telling you it’s the ugliest house I’ve ever seen.

People frequently ask, “Do you think this will work?” I almost always answer no, and then they hire us to build it anyway.

Interestingly, strategy development is very similar to trading in that people get emotionally attached to ideas. Even in the face of strong warnings, they charge ahead. A dear friend of mine opined on the subject, saying, “A handful of people don’t try. An even smaller handful listen to good advice. The rest of us learn the hard way.” Most people require the experience of falling flat on their face before they learn the lesson behind the advice.

If you’re motivated enough to ask a programmer to build a strategy for you, it’s because you already know that it is something that you really want to try. I could bluntly say, “This is going to wind up in tears.” 95% of people go ahead with the project, anyway.

Despite my knowledge of markets and systems, I’m not an oracle, either. I’ve told people that I thought their ideas were bad, only to have them come back a year later and tell me they’re making money.

…….Stay tuned for Part II when we discuss HFT, more back-testing issues (including those unique to Metatrader) and if there are common themes to successful systems.

Curve Fitting

September 24, 2012 by Shaun Overton Leave a Comment

I generally avoid optimizing for philosophical reasons. Traders always care about the bottom line. The question people inevitably ask is “how much does the EA make?” Although understandable, I believe that such a black and white concern does not model real life.

In reality, people care very much not just about how much they make. They also care a great deal about how they make it. The difference is that they know the former and tend to ignore the latter. It’s the primary reason why traders cannot stick to successful systems. They tend to walk away whenever their personal pain threshold is exceeded.

Optimizing for balance is extremely risky because it ignores one critical factor: it’s entirely possible that the optimal strategy for a given time period is a random outcome. Short of nose diving to zero, it is very difficult to conclude a strategy’s merit solely on the basis of return.

More importantly, the process of parameter optimization implies that a magical combination of settings exists that’s just begging to get discovered. It’s plain fallacy to believe that using a 50 period moving average really makes a critical difference compared to a 51 period moving average. Yet I see traders all the time exhausting every possible outcome looking for historical sweet spots.

One test that I ran several years ago, and I wish that I had kept the screenshots, was to write a strategy that entered and exited the market purely at random. The only rule that I retained control over was the frequency.

The random process blew my mind. It was ultimately in a good way, but I realized that many of the strategies and expert advisors that I reviewed previously and deemed “not bad” were indeed truly worthless. 20 independent trials wound up with 20 different outcomes. Excluding commissions, most of the strategies neither made nor lost money. Most of the random trades exhibited wild round trips. Profits might run up 10%, decline to -10%, then settle near a 0% return. The profitable strategies stood out most in my mind. One or two of them resembled ATM machines that churned out consistent profits on most trades.

The easiest way to limit the risk of using random parameters is to look at the underlying indicators used in the decisions. Price crossing above or below a moving average, for example, exhibits a genuine entry and exit efficiency that NinjaTrader can evaluate (70% if used as a range bound exit strategy. Entries are random). Moving averages crossing one another, however, show an entry and exit efficiency of 50%. Using that metric, I confidently chuck that type of rule out the window.

If you’ve performed an analysis on the entry and exit efficiency and found something greater than 55%, then at that point optimizing is at least a less bad pursuit. I confess to using it in the past to look for hotspots on potential parameter settings. If you decide to go down that road, it’s important to consider them as suggestions to evaluate rather than a holy grail.