[WEBINAR] Principles of Backtesting - Yuval Taylor

Principles of Back Testing

In this webinar, the speaker discusses the principles of back testing and how it can be used to predict future results. The speaker explains that there are laws and theories that govern portfolio performance, and that back testing is based on the assumption that there is a relationship between past performance and future results.

Laws of Portfolio Performance

• The law of diversification states that the standard deviation of a portfolio's returns is more likely to decrease than to increase as more imperfectly correlated assets are added to it.

• Regression to the mean implies that as long as there is a meaningful average of values, an extreme value will be more likely to become less extreme over time than to continue to be extreme.

• Outliers have a significant impact on results, making them harder to replicate. A strategy with high returns due to one or two outperforming stocks or periods is less likely to have its return replicated out of sample than one whose return is relatively unaffected by such stocks or periods.

• Alpha and beta are also important laws in portfolio performance.

Theories of Portfolio Performance

• There are hundreds or thousands of theories about portfolio performance, but they are not nearly as reliable as laws.

• Some examples include stock momentum over a six-to-twelve-month period and companies with very low free cash flow yields being more likely to have higher free cash flow yields in the future.

Conclusion

Back testing can help investors predict future results by analyzing past performance. By understanding the laws and theories that govern portfolio performance, investors can make informed decisions about their investments.

Understanding Alpha and Beta

In this section, the speaker discusses the compound annual growth rate, meaningful averages, alpha and beta.

Compound Annual Growth Rate

• The outperformance of low ranked stocks would not have been apparent if robust measures like trimmed alpha were used.

• Meaningful average is hard to get for things like market cap where there's a geometric range.

• Averages are not meaningful when there's a logarithmic scale between the lowest and highest.

Alpha and Beta

• Alpha and beta are inversely correlated so long as the market return tends to be positive.

• Low beta portfolios are likely to have higher alpha than high beta portfolios.

• The inverse correlation is not very strong but it's a factor that can be looked at mathematically.

Theories on Stock Performance

In this section, the speaker shares some theories on stock performance.

Theories on Stock Performance

• Low price stocks tend to outperform high price stocks.

• Small stocks have greater volatility than large stocks.

• Companies with growing sales and shrinking inventory outperform companies with shrinking sales and growing inventory.

• Companies that pay dividends regularly will continue to do so.

Correlation Between Past and Future Results

In this section, the speaker talks about how past results correlate with future results in backtesting.

Backtesting Correlation Study

• To figure out how to backtest, you need to find under what conditions past results best correlate with future results.

• Running a correlation study takes two weeks but it's important in order to measure correlations between strategies and equity curves.

• There are several questions that need answering such as how many years should be backtested, what performance measures should be used, etc.

• The best way to backtest is to maximize the correlation between past and future results.

Developing and Testing Trading Strategies

In this section, the speaker discusses how to develop and test trading strategies.

Creating Strategies

• Develop strategies that make sense and are not random.

• Use flexible strategies that can adjust portfolio size.

Testing Strategies

• Determine how long you want your strategy to last and use that as your out-of-sample period.

• Measure correlation between total return of out-of-sample period for 50 or 100 strategies and the return of immediately preceding in-sample periods.

• Test over a long time period with different portfolio sizes.

• Look for good correlation of strategy ranks between in-sample and out-of-sample periods.

Ideal In-Sample Period

• Ten years is the most correlative in-sample period with a three-year out-of-sample period.

• Shorter periods have worse correlation, while longer periods may be misleading due to factors working great during certain time frames.

Varying Number of Positions

• After determining ideal in-sample period, look at results for varying number of positions in your in-sample period.

Backtesting and Optimization

In this section, the speaker discusses the steps involved in backtesting and optimization.

Portfolio Backtesting Tools

• The speaker recommends using portfolio 123's backtesting tools to determine which tool gives the highest correlation with the actual out-of-sample period.

• The simulation tool is found to be the most reliable by the speaker.

Performance Measures

• The speaker suggests looking at which performance measure correlates best, such as CAGR, alpha, sharp ratio, or gain-to-pain ratios.

• Trimming outliers and using a weekly or bi-weekly two-week alpha period for measuring alpha is recommended by the speaker.

Summary of Correlation Study

• In-sample period of 10 years with holdings of two to five times the number of stocks held in the out-of-sample period.

• Backtesting method simulation and performance measure alpha using data whose outliers have been trimmed.

Optimization Procedure

Optimizing a Universe

• Liquidity limits are not to be optimized.

• Expected excess return per position should be calculated based on transaction records.

Portfolio Management

• Optimal holding period formula weights and anything else involved in portfolio management should be optimized.

• Ranking system where you want to optimize individual factors in a ranking system you want to optimize which factors to include and which factors to exclude and you want to optimize factor weights.

Avoiding Overoptimization

• The speaker believes that optimizing is necessary but overoptimization should be avoided.

• Positive correlation between in-sample and out-of-sample results leads logically to optimizing.

Liquidity Limits and Portfolio Optimization

In this section, the speaker discusses how to calculate slippage on each stock and exclude stocks with low liquidity. They also talk about setting liquidity limits and avoiding backtesting when doing so.

Setting Liquidity Limits

• Avoid setting liquidity limits based on backtesting because low liquidity stocks tend to outperform in most ranking systems.

• Use hard limits instead by running a variety of backtests with different ranking systems, methods, and portfolio sizes to see if adding a limit based on certain measures adds or detracts from returns.

• Consider excluding certain industries or categories of stocks that may not apply to your ranking system's factors.

Portfolio Optimization

• Optimize portfolios using a ranking system created three to five years ago or longer, or use a recently created and optimized ranking system.

• Avoid backtesting for portfolio optimization using a ranking system that was backtested during the period you're testing for portfolio optimization.

• Treat portfolio parameters and ranking systems separately when optimizing them. Vary things like number of stocks held, buy/sell rules, formula weights, minimum holding period, sector/sub-sector/industry counts, maximum weight of single position.

Optimizing Factors

In this section, the speaker talks about how certain factors can benefit from optimization instead of simply higher or lower is better.

Size Factors

• Lower market cap isn't always better as some lower liquid elite and market cap level stocks behave erratically. Stocks with slightly higher market caps may be more responsive.

Other Factors

• No specific bullet points provided.

Factors and Optimization

In this section, the speaker discusses how to create factors that favor certain industries and ranks others low. They also talk about optimizing these factors by testing them on different subsets of the universe.

Creating Factors

• Use the eval command and arbix codes to give higher weights to certain sectors, sub-sectors, or industries.

• Growth factors may benefit from ranking companies with middling values highest.

Optimizing Factors

• Split your universe into compartments and test your ranking system that excludes those factors on each compartment.

• Test your ranking system on a small part of your universe using between f rank market cap ascending previous exclude n a 9100.

• Eliminate size vectors from your ranking system and repeat the test after changing 9100 to 80 90 and so on.

• Test responsiveness of sub-sectors to your ranking system by comparing overall return of the sub-sector with the return of the sub-sector in your ranking system.

Combining Questions

• Combine which factors to use in which to exclude and what weights you should assign into one test by allowing zero percent weights for your factors.

• Develop a long list of factors where most get zero percent weight but have a few get higher weights.

• Set up different universes that are subsets of your universe using mod stock id comma five equals zero mod stock id five equals one, etc.

Bootstrapping Technique

In this section, the speaker talks about bootstrapping technique and how it can be used to optimize ranking systems.

What is Bootstrapping?

• Bootstrapping is a machine learning technique that can be used to optimize ranking systems.

• The speaker got the idea from James O'Shaughnessy who writes about it in the fourth edition of "What Works on Wall Street".

How to Optimize Ranking Systems using Bootstrapping?

• Vary your portfolio weights iteratively until you get the ranking systems that perform best in each subset universe as well as ranking systems that perform well in all of them.

• Average the weights of those outperforming ranking systems.

• Use factor weights divisible by four percent or two and a half percent or two percent and then randomize. Four percent is a good way to do it.

• Expect to spend a lot of time on backtesting because you're testing all these different iterations of ranking systems on five different universes.

Tips and Warnings

• Don't vary factor weights by less than two percent; it's a complete waste of time.

• The amount of backtesting this method involves is extreme, so expect to spend a lot of time on it.

• Optimizing only on the entire universe may not give good results. Use bootstrapping instead.

• Optimizing using small number of positions may not give accurate results. Optimize using two to five times the number of positions that you actually expect to hold when you're bootstrapping.

• Unrealistic returns are common after optimization. A rough approximation is good enough.

Common Back Testing Mistakes

• Basing your optimized portfolio parameters on testing a period with a ranking system already optimized over that time period

• Basing your preferences simply on CAGR rather than trying out more robust performance measures

• Taking into account only overall performance rather than looking at performance during various discrete time periods is another back testing mistake.

Backtesting Strategies

In this section, the speaker discusses the importance of backtesting strategies and the potential problems with using an out-of-sample period.

Importance of Backtesting

• It is important to take into account changes that have happened in the stock market in the last three years when backtesting strategies.

• Stress testing your systems is vital to subject your systems to stress tests.

Problems with Out-of-Sample Period

• Using a little tiny out-of-sample result may not be very indicative of anything.

• Small caps did terribly during 2019, so if 2019 was used as an out-of-sample period, it would not have been possible to optimize anything.

Five Universes

• The speaker performs iterative backtests on all five universes and ranks strategies that perform best on each universe.

• The best performing strategies from each universe are averaged to get the final strategy.

Stress Testing Strategies

In this section, the speaker emphasizes the importance of stress testing strategies and provides resources for further learning.

Importance of Stress Testing

• Stress testing your systems is vital to subject your systems to stress tests.

• Backtesting for failure may be almost as important as backtesting for success.

Resources for Learning

• Ricardo Tamara has given a webinar on stress testing strategies that viewers can watch.

• The speaker has written a blog article on stress testing strategies called "Break Your Strategy," which viewers can access through the provided links in the slides.

Q&A

In this section, the speaker answers a question about how to use the five universes and invites further questions or feedback.

Using Five Universes

• The speaker performs iterative backtests on all five universes and ranks strategies that perform best on each universe.

• The best performing strategies from each universe are averaged to get the final strategy.

Feedback and Questions

• Viewers are invited to ask any further questions or provide feedback using the chat box.