The Metatrader 5 Series : When Huge Backtesting Differences Appear

If you look at my last few weeks of posts I have been very excited about the new qualities of the Metatrader 5 trading platform and the benefits it brings to system development regarding faster execution, faster optimization and added flexibility. Using my prototype implementation of Watukushay FE I analyzed trading over several different instruments and I finally came up with the "starting point" of intra-currency trading for this well-known freely available trading machine. This week I decided to do a small experiment and compare my results of Watukushay FE for the AUD/USD on Metatrader 5 with those found out with Metatrader 4 and the results found were extremely surprising. On today's post I want to talk about my findings, the possible causes of the issues found and what I will be doing to investigate the nature of this problem and what solutions can be implemented to deal with it.

During the first post comparing Metatrader 4 with 5 and the backtesting results of Watukushay FE we already saw a small yet noticeable difference between the testing results obtained on both trading platforms. I talked about the possibility of these errors being feed-related and the fact that the Metatrader 5 history feed might be more reliable since it has been "remastered and fixed" for this new platform. However the difference was small and therefore there was no substantial issue besides an anecdotical note pointing out this curious fact.

However when I decided to run the initial AUD/USD tests on Metatrader 4 to compare the results I obtained with Metatrader 5 the difference changed from "noticeable" to "abismal". The pictures below show you the results for MT5 and MT4 using the exact same settings on the AUD/USD currency pair backtest from 2000 to 2010. The overall equity curve is very different and the results do point out that something is substantially very changed between the historical feeds of MT4 and MT5. I first thought that the difference would be due to the presence of Sunday candles but this turned out to be false since the MT5 feed doesn't have any of them, so regarding this aspect it is the same as MT4. I then thought about the possibility that the whole difference is caused by important changes in data prior to 2006 (before metatrader 4 was launched) and the fact is that data differences are NOT limited to pre 2006 periods, the whole historical feed is different between both trading stations and meaningful differences are present. If you analyze the results you'll notice that almost all candles have different - if only very slightly - high/low/open/close values pointing out that RSI and ATR values will be very different. The change in one minute interpolation mechanisms is also not likely a factor here as Watukushay FE strictly controls bar opening on both its MQL4 and MQL5 implementations.
--
What causes such a dramatic change in profitability ? To get to the bottom of this problem I decided to strip down the logic to its simplest form and eliminate the closing logic of the EA, leaving only the entry rules. This shows us that there is still some difference between backtesting results (shown below). This means that differences in results are caused by differences in the RSI and ATR indicator calculations which are dependent on each backtest's particular historical feed. Stripping down the logic does reveal that most dependency is located before 2002 with results beyond this date being in better agreement. However there is still some dependency which is caused by differences in data between both historical sets beyond this period.
--
Since we simply cannot know for sure which of the two historical data selections is better - and they are probably both valid within normal broker differences (with the 2000-2002 data being very different probably due to differences between feeds in this period) - it becomes a wise decision to run backtests on both and trust the less profitable results to calculate profit and draw down targets. In some cases like the EUR/USD backtests this proves to be trivial but on others like the one I showed you today doing this mixed analysis proves to be extremely important. I will email the people at metaquotes to get some information about the different nature of the feeds and I will let you know once I have more information about their origin. However up until now all backtests of Watukushay FE seem to be more profitable on MT5 (meaning that our MT4 simulations are in fact the worst case scenario). Investigating other issues which may be related with the closing mechanism of orders in MT5 is also something I am currenlty doing since I have seen that the differences when the closing logic is enabled seem to have other strong causes besides simple feed dependency (more on this on a later post !).

If you would like to learn more about automated trading, the evaluation of expert advisors and the programming of your own strategies please consider buying my ebook on automated trading or joining Asirikuy to receive all ebook purchase benefits, weekly updates, check the live accounts I am running with several expert advisors and get in the road towards long term success in the forex market using automated trading systems. I hope you enjoyed the article !