Saturday, September 11, 2010

Are You Backtesting Correctly ? : Six Common Technical Mistake That Will Make Your Simulations Useless

When simulating the performance of a trading strategy using historical data within the Metatrader 4 platform there are many things that you can do which will inevitably end in bad performance and unreliable results. Many of the reasons why people regard backtesting using Metatrader as unpredictable and difficult to reproduce are a consequence of one or several technical problems which can arise due to the lack of carefulness of the trader running the evaluation. Knowing about these problems and taking action so that they do not affect simulations is something necessary to arrive at reproducible and reliable results. I have proved - within several trading strategies - that simulations can be reliable and easily reproduced if special care is taken to avoid technical pitfalls.

Within the following paragraphs I will share with you the six main technical reasons why traders arrive at unreliable simulation results that greatly over or under estimate the profitability of their trading systems. These problems can be easily avoided through some simple measures that can ensure that simulations are as reliable and useful as possible. Falling into just one of these problems can cause back-testing results to be utterly meaningless so avoiding them is of primordial importance for anyone interested in the accurate evaluation of trading systems and expert advisors. These are the technical problems you might encounter :

1. Using a 4 digit broker to run simulations. Many people think that the most accurate simulation results for their systems are obtained when running backtests with their broker's Metatrader 4 platform. However they do not realize that backtesting data is ALWAYS downloaded from metaquotes servers and that the 4 digit broker data set downloaded from Metaquotes contains MANY errors which make simulations totally unreliable. There are major gaps in price in the lower time frames, many daily candles missing large segments of volume, introduction of Sunday daily candles on some years, etc. If you want your simulations to be reliable you need to use and ONLY use five digit brokers for backtesting which download the much more reliable five digit data set from metaquotes.

2. You are using the weekend spread. Another very common technical problem people come across is the running of simulations on the weekend when the spread is extremely high in some cases. When you run a simulation during the weekend using this spread values you will have much worse results than what you would have when using the regular spreads provided during the trading hours of the week. In the end you should always perform your backtests in trading hours or change the spread within the Metatrader 4 platform (we use a script in Asirikuy in order to achieve this).

3. Your strategy trades below the 1 hour chart. During the past few years I have tested and run live/back testing consistency analysis of several strategies that run within the 30 min, 15 min and 5 min time frames only to find out that their results are each and every time inconsistent with simulations. The reason why this is the case is because the lower the time frame the more prominent the effect of one minute interpolation errors when determining things such as indicator values becomes. The broker dependency also increases exponentially and when trading 5 minute charts it becomes so high that the simulations are utterly meaningless. The fact is that variability caused by broker dependency and interpolation errors within these time frames is SO high that you can have totally different results between your backtests and reality. The problem is less pronounced for the 15 minute chart and only a small effect occurs on the 30 minute chart but the problem is not almost completely eliminated until you move to at least 1 hour charts.

4. Your Take Profit and Stop Loss values are within 10 times the spread of the instrument you are trading. When you are running simulations of systems that use these type of trading obtaining reliable results is impossible, not only due to the problem with one minute interpolation errors (which for this case is huge) but because of execution variables (such as re-quotes and spread widening) which prove to be VITAL in the actual real-life profit of these strategies. If you want your simulations to mean something and provide you with some approximation to valid profit and draw down targets then your average Take Profit and Stop Loss must be above 10 times the spread.

5. You are not recalculating your data before each backtesting run. Something which is extremely important is the recalculation of data before starting each new simulation. When you load a chart or when your demo feed sends a tick to your platform there are sometimes history recalculations which corrupt your data and cause your simulations to become erratic and invalid. In order to correct this problem you must recalculate your data within the history center before running every back-test. This can be achieved by going to the one minute section of the instrument you want to recalculate within the history center and clicking the download button until it prompts you to recalculate data. Doing this ensures that your data will not suffer from corruption from your demo feed.

6. You are running a backtest over the last 1-3 months. Your historical data is composed of the data you download from Metaquotes servers and the data you obtain from your live/demo feed from your broker. The last 3 months of testing data are usually downloaded from your broker while the data before pertains to the history center. Usually if there is a time stamp mismatch between your platform's live feed and the Metaquotes data there will be massive generation of errors within the past 3 months of data as the program gets "confused" from these differences. If a chart of the instrument you want to trade shows massive gaps after you do a historical data recalculation then this is a problem. You can generally avoid this by only running backtesting that end three months before the current time.

Certainly the metatrader 4 platform has many limitations and the above restrictions limit us to the development of certain kinds of trading strategies. However this doesn't mean that simulations are unreliable but mainly that great care has to be taken in order to make the backtests reliable, reproducible and coherent with live trading results. By following all the above suggestions and avoiding this technical pitfalls you will be able to obtain reliable backtests of your trading strategies which will allow you to get a good picture of the possible long term performance of your trading strategies.

If you would like to gain a true education around automated trading systems and how you too can design strategies that achieve reliable simulations with accurate profit and draw down targets please consider joining, a website filled with educational videos, trading systems, development and a sound, honest and transparent approach automated trading in general . I hope you enjoyed this article ! :o)


townjet said...

Thanks for the post Daniel.
I needed this post. I knew about most of the 6 items you mentioned by trial and error, but I did not know about how incorrect the 4 digit can be and that the last 3 months can have gaps (Though I suspected something was up when I was seeing those gaps :))



Tcxmon said...


Thanks for the post.

I've had great success testing the God's Gift EA's since you re-released the new versions earlier this week. I was able to reproduce your back tests and get excellent results and smooth-as-a-babys-bottom equity curves on the 10-year backtests for the God's Gift EA's.

Also, you are 100% correct about Alpari having the best back-testing data. I was able to run 10-year back test with zero chart mismatch errors!

On Saturday morning, I had a lot of issues backtesting with Alpari due to the weekend spread. I tried to adjust the spread via the symbols.sel file (over a few hours) with no success. In each case, I clicked the "Symbol Properties" button to check the spread to find my changes not taking.

I went back and looked at the other symbols.sel files on the system and found the dates on those files set some time in the past, much further back than the last date used by the platform.

So I tried to delete all the symbols.sel files from the MT4 folder and still found MT4 not to be writing a new symbols.sel and instead getting the spread from some other source. So I have yet to fully understand from where MT4 is getting the spread info.

In any case, i'm sure we will figure it out. I do appreciate all the time and effort you spend helping us to figure out the complexities of MT4 and Forex.

I've e-mailed some EA vendors just to get a 2-line, unhelpful response.

So thanks again for all you do and i'll share any further insights I find on the source of the spread info.



Daniel said...

Hello Chris and JT,

Thank you very much again for your comments :o) I am very glad that you are such faithful readers !

@JT : Yes, I was also surprised to find about all the problems with the 4 digit data. Actually it only became clear after I did a very thorough analysis after developing Watukushay No.2. The issues with the last 3 months of data became evident after I started doing back/live testing consistency tests last year. I am glad I can share this knowledge with you so that you can improve your backtesting skills :o).

@Chris : I am glad that this new EA version is now working better for you ! :o) This new version has a lot of improvements with a massive reprogramming of the code (without changing the logic), a whole new user interface and adequate error handling capabilities. I took special care in making sure that backtesting was reliable and reproducible so the problems with simulations for this EA should be entirely solved :o).

Regarding the spread modifications, I did receive a few emails from other Asirikuy members facing similar problems reason why tomorrow I will be releasing a re-make of the spread changing video tutorial to avoid the problems you are currently having. I have tested the new procedure on several computers so you should be able to solve your problems using it. The video should be released tomorrow before 12 am EST along with the other videos and the weekly text newsletter. Of course if you have some problems with changing the spreads after you follow this new procedure feel free to share your experience within the Asirikuy community forum so that me and other members can help you solve it.

Thank you very much again guys for your comments and visits :o)

Best Regards,


Gabor said...

Hello guys,

I follow the follwing steps when I run backtests:
- delete all the *.hst files in the /history folder before I start the terminal
- start the terminal, log in and download the M1 data from metaquotes in history center
- log off the terminal (I mistype the password and login), this way the terminal gets offline and can't update the feed and the symbol.sel (which contains the spread and other trading environment variables), in addition you don't need to recalculate data after every backtest run (point 5). This however doesn't solve the problem in point 6, so you have to run backtesting that end three months before the current time
- close terminal, copy the hacked symbol.sel into /history folder and then re-open terminal
- running backtests :)

Daniel Fernandez's Expert Author Email Alerts
Sign up to receive email alerts of Daniel Fernandez’s latest articles from!

Email Address: