One of the most prominent mistakes I have noticed when people try to develop new trading systems or test commercial or free trading systems is their eagerness to jump to conclusions. I believe that the way in which many people evaluate systems and make conclusions about their profitability/unprofitability is actually faulted due to the fact that they don't take into account the significance of their results when evaluating a strategy. On today's post I will write about one -if not the most- important aspect when evaluating any trading system or strategy : Statistical Significance. Through the following paragraphs I will explain to you what this concept is and what its consequences are for expert advisor evaluation and trading system design.
When people design or evaluate a trading strategy they are usually very eager to say something about the performance of a system within a few weeks or a few months. Phrases like - if it works well on demo for a few weeks I'll put it on live - or - it didn't make a profit for 6 weeks - are very common and some of the first things new traders actually learn. The problem is mainly that this short-lived analysis of a trading system usually leads to conclusions which are only representative of short term performance and do not have any validity regarding the long term aspects of a given trading strategy.
I believe that this problem is mainly a consequence of the lack of formation people have regarding statistics - a field which is very important when developping long term performing systems. This science tells us that in order to draw valid conclusions for a given question we must have a big enough sample size. So saying that a system is unprofitable or profitable from a few months of testing is simply not valid because a few months are simply statistically not representative of long term performance. When you evaluate a trading system's performance in the short term you are simply evaluating it under current market conditions and possibly under a temporary draw down cycle which does not appear evident to you due to the short scope of the analysis being done. This is specially important for many systems that have predictably long cycles of draw down which then lead to very profitable trading periods, systems which would most of the time be discarded by people who simply don't evaluate systems in a rigurous manner.
Another problem of this "short term syndrome" is the fact that people quickly jump to conclusions about how to modify a system to make it "perform better". These modifications which are almost NEVER based on an analysis of a statistically significant number of trades may lead to an improvement of performance under current market conditions with a fatal blow to long term trading performance. On the other hand, there will also be a "praise" of systems which perform very well in the short term, often leading to the heavy use of systems that use unsound trading techniques and which put accounts at a great risk of facing a complete wipe out.
So what is a statistically significant sample size ? This question is not very easy to answer since there is no mathematical criteria to put aside a market condition from another. I have discussed this question a few time with a friend of mine who has a major in statistics and we arrived at the conclusion that - according to volatility measures - at least 5 years of analysis are necessary to draw valid conclusions about a trading system. Therefore, when going through system development and the proposal of modifications it becomes necessary to evaluate a modification or performance through the course of a five year period in order to draw valid conclusions.
In the case of systems which ARE back/live testing consistent, five years of live trading are necessary in order to conclude that the adaptability and performance seen in backtesting can effectively be reproduced under future market conditions. If the system is not back/live testing consistent the problem becomes harder as at least an initial 5 years of live trading with no modifications are necessary to make an analysis to propose changes to the system to increase its profitability.
I know that it is actually hard to go through hundreds of trades and evaluate one by one the effect of a certain modification. For most people it is much easier to just evaluate systems for a few months and draw conclusions which are simply not statistically valid meaning that they are not representative of the system's long term performance. Some people even venture to modify systems based on just a few weeks or days of trading, making modification which have an unpredictable effect on long term performance.
It is easy to understand- with such a lack of rigurous analysis and evaluation - why there is an overvaluation of systems that give short-term result and a systematic discarding of some systems that have indeed potential for long term profitability. Most reviewers show little or no knowledge regarding this field and they will be very quick to jump to conclusions without having a clue about the validity of what they are actually saying from a "sample size" point of view. I encourage everyone interested in system design or in automated trading in general to get a basic formation in the field of statistics and particuarly in the field of "hypothesis testing" which evaluates the whole process of drawing valid conclusions from a given statistical sample.
If you would like to learn more about what I have learned in automated trading and how I focus my efforts in the teaching and development of systems which have a high like hood of being profitable in the future please consider buying my ebook on automated trading or joining Asirikuy to receive all ebook purchase benefits, weekly updates, check the live accounts I am running with several expert advisors and get in the road towards long term success in the forex market using automated trading systems. I hope you enjoyed the article !
When people design or evaluate a trading strategy they are usually very eager to say something about the performance of a system within a few weeks or a few months. Phrases like - if it works well on demo for a few weeks I'll put it on live - or - it didn't make a profit for 6 weeks - are very common and some of the first things new traders actually learn. The problem is mainly that this short-lived analysis of a trading system usually leads to conclusions which are only representative of short term performance and do not have any validity regarding the long term aspects of a given trading strategy.
I believe that this problem is mainly a consequence of the lack of formation people have regarding statistics - a field which is very important when developping long term performing systems. This science tells us that in order to draw valid conclusions for a given question we must have a big enough sample size. So saying that a system is unprofitable or profitable from a few months of testing is simply not valid because a few months are simply statistically not representative of long term performance. When you evaluate a trading system's performance in the short term you are simply evaluating it under current market conditions and possibly under a temporary draw down cycle which does not appear evident to you due to the short scope of the analysis being done. This is specially important for many systems that have predictably long cycles of draw down which then lead to very profitable trading periods, systems which would most of the time be discarded by people who simply don't evaluate systems in a rigurous manner.
Another problem of this "short term syndrome" is the fact that people quickly jump to conclusions about how to modify a system to make it "perform better". These modifications which are almost NEVER based on an analysis of a statistically significant number of trades may lead to an improvement of performance under current market conditions with a fatal blow to long term trading performance. On the other hand, there will also be a "praise" of systems which perform very well in the short term, often leading to the heavy use of systems that use unsound trading techniques and which put accounts at a great risk of facing a complete wipe out.
So what is a statistically significant sample size ? This question is not very easy to answer since there is no mathematical criteria to put aside a market condition from another. I have discussed this question a few time with a friend of mine who has a major in statistics and we arrived at the conclusion that - according to volatility measures - at least 5 years of analysis are necessary to draw valid conclusions about a trading system. Therefore, when going through system development and the proposal of modifications it becomes necessary to evaluate a modification or performance through the course of a five year period in order to draw valid conclusions.
In the case of systems which ARE back/live testing consistent, five years of live trading are necessary in order to conclude that the adaptability and performance seen in backtesting can effectively be reproduced under future market conditions. If the system is not back/live testing consistent the problem becomes harder as at least an initial 5 years of live trading with no modifications are necessary to make an analysis to propose changes to the system to increase its profitability.
I know that it is actually hard to go through hundreds of trades and evaluate one by one the effect of a certain modification. For most people it is much easier to just evaluate systems for a few months and draw conclusions which are simply not statistically valid meaning that they are not representative of the system's long term performance. Some people even venture to modify systems based on just a few weeks or days of trading, making modification which have an unpredictable effect on long term performance.
It is easy to understand- with such a lack of rigurous analysis and evaluation - why there is an overvaluation of systems that give short-term result and a systematic discarding of some systems that have indeed potential for long term profitability. Most reviewers show little or no knowledge regarding this field and they will be very quick to jump to conclusions without having a clue about the validity of what they are actually saying from a "sample size" point of view. I encourage everyone interested in system design or in automated trading in general to get a basic formation in the field of statistics and particuarly in the field of "hypothesis testing" which evaluates the whole process of drawing valid conclusions from a given statistical sample.
If you would like to learn more about what I have learned in automated trading and how I focus my efforts in the teaching and development of systems which have a high like hood of being profitable in the future please consider buying my ebook on automated trading or joining Asirikuy to receive all ebook purchase benefits, weekly updates, check the live accounts I am running with several expert advisors and get in the road towards long term success in the forex market using automated trading systems. I hope you enjoyed the article !
Posts
No comments:
Post a Comment