Unreliability of Backtesting Data... Why you should ONLY use the Alpari Downloaded Mt4 for Backtesting

Through all my time as an automated trading system developer one of the things which I have always dealt with is the problem of backtesting reliability. It quickly became apparent to me that the metatrader backtester had some serious flaws (as evident by some commercial experts which are able to turn dimes into millions in a few years) inherent to the way in which the program does its testing. Besides these flaws which include backtesting interpolation and similar problems I also found out that there were a lot of "data" related problems. Within this blog post I want to talk to you a little bit about these problems and why it is vital to address them in order to have reliable simulations of your trading systems. Particularly I am going to tell you the reasons why backtesting should ONLY be done with the metatrader downloaded from the Alpari website since backtests done on ANY other platform (any non 5 digit broker) are bound to be VERY unreliable, depending on your trading system.

So what are data-related errors ? These errors regarding a trading system's simulated performance are generated due to quality errors within the historical data used, that is, the data used to do the backtests has errors which do not allow us to do accurate historical simulations. Last week, with the release of the Teyacanani trading system, I received a couple of emails telling me that some Asirikuy members were unable to reproduce my backtesting results. Upon some asking I realized that these members were not using the Alpari downloaded metatrader platform but their OWN broker's platform to do the backtesting mainly because they believed that this would better reproduce their broker's "trading conditions". I was absolutely determined to look into the issue as it was vital for me to know why these results were changing so much.

What I found out was that the metatrader history downloaded on other broker platforms (I tried around 10 different broker platforms with the same results) gives you very inaccurate historical data. At first I thought that the problems were arising due to the appearence of Sunday candles within this historical data (which appear after 3 months of past data regardless of if your broker has or doesn't have these bars) but then I realized that not only was there an introduction of Sunday candles but there was also a substantial amount of differences with the data of the Alpari downloaded platform. These differences - which I examined from 2005 through 2007 - include the appearence of very low volume daily bars (non-Sunday) which are missing a massive amount of movement with much smaller high/low/open/close values than on the Alpari downloaded platform. Moreover, there is also the appearence of very odd holiday bars which are bars made up of 1 tick with the same high/low/open/close values. The below image shows examples of these errors as well as a comparison between the 5 period ATR on both accurate and inaccurate historical data.
--
My conclusion from this was basically that this data has a lot of intra-day information missing (which I confirmed) as well as massive amount of inaccurate daily candles, something which inevitably changes the profitability of many trading systems which rely on short indicator periods such as Teyacanani. I was previously aware of these problems with data downloaded through the history center from non-five digit broker downloaded mt4 platforms but the problem seems to be far greater than what I had previously imagined. However some systems are bound to show more or less changes between these two data sets depending on their sensibility.

Through this post I want to warn fellow traders and automated trading system developers that you should ONLY use the Alpari (or other five digit broker) downloaded metatrader 4 platform for backtesting and that using any other broker for backtesting does NOT make the backtest approach your "broker's conditions" , on the contrary, the downloaded data may introduce contradictions as Sunday candles on a non-Sunday candle broker and all the errors I mentioned before. For this reason- if you want to backtest your systems with reliable data- download the metatrader platform from Alpari and do your backtests there. I have compared this data with Oanda Tick data and it seems to be in fair agreement (while the other data is NOT) so again, ONLY use a five digit broker metatrader platform to perform backtests.

Previously I had written that the metaquotes platform should be downloaded but they recently changed from a five digit (in build 225) to a four digit (in build 226) broker by default making the platform download the bad data. Alpari US/UK (which was the previous broker used) does download reliable data.

This Sunday I will be releasing a video on Asirikuy about all these issues and the extent to which they affect automated trading, including changes with Teyacanani and other experts detailing all the possible effects these errors may have. If you would like to learn more about automated trading and backtest reliability please consider buying my ebook on automated trading or joining Asirikuy to receive all ebook purchase benefits, weekly updates, check the live accounts I am running with several expert advisors and get in the road towards long term success in the forex market using automated trading systems. I hope you enjoyed the article !