degrees of freedom Archives - アルゴリズムと機械の外国為替戦略

I was really excited about my Pilum strategy two months ago. The research looked great and everything was ready to rock and roll. Demo testing began and then… not much happened.

The Quantilator is (mostly) finished, which finally gave me time to circle back and review what happened with Pilum.

Live demo trading of Pilum. 12月 9, 2016 to Feb 7, 2017

The expected outcome was that I would win 75% 時間の. Trades were infrequent, so I thought maybe I’m just having bad luck. But then my win rate remained stuck around 50%. Simple statistical tests told me this was unlikely to be bad luck.

I used the research time to pour over my research code and to compare it with live trades. What I found was that a single line of code (AHHHHHHHHHHHHHHH!) was incorrectly calculating my entry price, dramatically overstating the profits.

、 flawed code produced this equity curve from a single combination of settings:

When the actual, correct result looks like this with those same settings:

The accurate backtest of Pilum

I’ll be honest… I like the flawed backtest a lot more!

The new, single-setting backtest isn’t as good, but it’s still trade-worthy. There are some characteristics that I dislike and features that I love. Let’s dig into those.

What I dislike

The frequency of trades is very low. Out of 19 months there were a total of 43 取引. 43 trades to comprise a backtest on 40+ instruments is a very small number.

If it weren’t for the statistical pattern backing up the frequency, I would not consider the test. しかし, there are 20,000 bars each on the 44 instruments. あります。 880,000 total bars used to analyze whether my Pilum pattern offers any predictive value.

The most valuable predictions, しかし, are also exceptionally rare. That’s why I’m not able to get the trading frequency higher, which would potentially smooth the returns.

What I love

My previous systems like QB Pro and Dominari traded actively for relatively small wins. Trading costs exercised a massive impact on the overall performance.

The accurate backtest of Pilum

Now look again at the correct equity curve (the image to the right). Do you see the final profit of roughly 0.14? That’s a 14% unleveraged return over a 19 ヶ月の期間.

Allocating 2:1 または 3:1 leverage on this strategy could average annual returns of 15-25%.

Detecting hidden risk

A key measure of risk is skewness. You may not use that term yourself, but it’s something most of you already understand. The biggest complaint about people trading Dominari was that the average winner relative to the average loser was heavily skewed towards the losers.

Dominari wins on most months, but when it lost in December it was devastating. I implemented what I thought was a portfolio stop after the December 9th aftermath. Then I had a smaller, but still very painful, loss in January. The portfolio level stop loss of 3% should prevent future blowouts now that I know what goes wrong.

I still believe in Dominari. しかし, I obviously lost the work of most of the year due to those events.

Knowing that skewness is a good measure of blowout risk (even if you’ve never seen it in a backtest, like happened with Dominari), Pilum looks extremely encouraging.

This is a histogram of profit and loss by days. You should notice a few things.

The tallest bar is to the right of 0. That means that the most frequent outcome is winning.

The biggest winning day is dramatically better than the worst losing day. The worst outcome was a loss of 2%. The best outcome is gains near 10% in a single day (unleveraged!).

This is the statistical profile of an idea that’s much more likely to grab an avalanche of profits than it is to get blown out.

It gets even better

Would you say that the blue and red equity curves are highly or loosely correlated? Look closely.

Writing this blog post made me think carefully about the Pilum strategy. I decided that maybe I should see if all of the profits are coming from different settings at the same time. There’s very little risk of overfitting the data as my strategy only has 1 degree of freedom.

The blue bars are the equity curve of Setting 1.

The red bars are for Setting 2.

Do you think these are tightly or loosely correlated?

If you said loosely correlated, then you are correct. Notice how each equity curve shows large jumps of profit. Did you notice how those profit jumps occur on different days?

The blue setting skyrockets on a single day in November 2016. It leaves the red equity curve choking in its dust.

But then, look what happens as I advance into December. The red curve dramatically catches up to the blue curve and even overtakes it.

The correlation between the 2 strategies is only 57%.

Combine multiple settings into 1 portfolio

This is a much nicer equity curve!

Loose correlations are a GIFT. Combining two bumpy equity curves into a single strategy makes the performance much, much smoother.

The percentages of days that are profitable also increases. Setting 1 is profitable on 58.0% of days. Setting 2 is profitable on 53.5% of days.

しかし… combining them makes Pilum profitable on 68.2% of days. 素晴らしい!

That also provides more data, which puts me in a stronger position to analyze the strategy’s skewness. Look at the frequency histograms below. They’re the same type of histograms that I showed you in the first section of this blog post. As you’ll notice, they look a lot different.

Pilum most probable daily profit and loss

The most probable outcome for any given day is a small winner

The tall green bar is the most probable trading outcome for any given day with filled orders. The average day is a positive return of 0-1%.

The small red bar is the worst trading day of the combined strategy.

The small green bars are the best trading days of the combined strategy.

Look how far to the right the green bars go. The largest winner is more than 3x the biggest loss. と, there are so many more large winners compared to losers.

Giant winners are far more likely than comparable losses.

The Plan

I immediately pushed Pilum into live trading this combination of two strategies. I expect that adding a second degree of freedom and running about 30 different versions of the strategy – all with different settings – will add to the performance and smooth the returns even further.

Dominari hasn’t been working on my FXCM account, which is very difficult to accept because the lacking performance seems to be a buried execution issue. Pilum, しかし, trades very infrequently. It’s unlikely that execution quality will make a dramatic difference in the long term outcomes.

だから, I’m going to convert the FXCM account to trading Pilum exclusively. That will be offered as a strategy on Collective2 within the next few weeks, a company with whom I’ve been working closely. Their users are more investor rather than trading oriented – they’re far more likely to view low trading frequency as a good thing. I suspect that most people here have a different opinion and want to see a lot of market action.

I’ll write an update on Dominari shortly.

One of the biggest appeals of mechanical trading strategies is that there are an endless number of parameters and indicators that a developer can add to any system. The first instinct of every new system trader is to attempt to improve on a basic system by adding another component to it. This usually results in a system that looks better in backtesting, but fails to perform well moving forward.

Almost all system traders begin by designing over-complicated strategies, only to find that simpler is usually better.

As system traders continue to experiment with different strategies, many come to appreciate that over-complicated systems are more prone to curve-fitting bias. Simpler strategies might look less impressive on backtests, but they are generally more robust options to trade moving forward.

Comparing System Trading To Cooking

The authors of GESTALTU published a post where they compared trading system development to cooking. The similarity that they identified in both fields is that more ingredients does not necessarily equate to a better overall result.

The article explains that adding more ingredients to a recipe will likely detract from the way the original ingredients worked together. 同様に, adding more indicators to a strategy will likely influence how that strategy performs in certain market environments.

The author describes a strategy that he developed early in his career that contained 37 different parameters. With that many different inputs, finding an optimized version of the strategy become nearly impossible. さらに, any backtesting results using that many variables are likely to be exposed to curve-fitting bias.

Limiting Degrees of Freedom

In yesterday’s post we touched on the different ways that price action and technical indicators can be exposed to curve-fitting. In each case, the key was to limit the degrees of freedom in the system.

The fact that price action generally used less parameters than technical indicators makes price action less likely to be exposed to curve-fitting. Applying that logic more broadly, the more parameters a strategy has, the more likely it is to contain some degree of curve-fitting.

Backtesting with Multiple Parameters

While backtesting strategies with large numbers of parameters is certainly possible, it is exponentially more difficult. Increasing the number of parameters will require a much larger sample size in order to produce backtesting results that are worthwhile.

Because backtesting results from systems with more degrees of freedom are more prone to curve-fitting, we can reasonably assume that results from systems with less parameters are more sound. したがって, we can have more confidence that lower parameter systems will continue to produce similar results moving forward.

One line of code makes all the difference