Regime Switching Models with Machine Learning | Piotr Pomorski

Introduction to AI in Finance

Welcome and Guest Introduction

• Joy introduces the UCL Artificial Intelligence Society and welcomes Piotr, a special guest presenting his PhD research.

• Piotr expresses gratitude for the invitation and shares excitement about applying AI solutions in finance.

Background of Piotr

• Piotr is a part-time PhD candidate at UCL with a background in economics and finance, highlighting his hybrid expertise.

• He is a CFA charterholder since 2019 and currently works as a senior quant at Church Commissioners for England, managing around $10 billion in assets.

• His role involves complex quantitative analysis within the church's endowment fund, emphasizing responsibility towards financial management.

Machine Learning Applications in Finance

Mentorship and Personal Interests

• Piotr mentors students interested in financial machine learning, helping them bridge gaps between mathematics and practical applications.

• He shares personal interests including video games, astronomy, history, and football.

Focus of PhD Research

• His PhD research centers on detecting and predicting regime switches in finance, which are critical for asset management strategies.

Understanding Financial Regimes

Definition of Financial Regimes

• A financial regime refers to distinct periods characterized by variations in asset returns, volatility, and correlations within financial time series.

Importance of Regime Detection

• Recognizing different regimes helps asset managers adjust portfolio weights effectively to optimize returns during varying market conditions.

Characteristics of Low vs. High Variance Regimes

• In low variance regimes:

• Calm market conditions with rising asset values; lower correlation among assets.

• Investors generally experience higher returns.

• In high variance regimes:

• Market panic leads to decreased returns; increased volatility; higher correlation as investors sell off assets indiscriminately.

Implications for Asset Management

Strategic Adjustments Based on Regime Predictions

• Asset managers must shift investments based on predicted regime changes—moving towards riskier assets during growth phases while seeking safe havens during downturns (e.g., bonds or gold).

Historical Context: The 2008 Financial Crisis

Markov Switching Regression and Machine Learning in Finance

Overview of Markov Switching Regression

• Markov switching regressions are robust models from the 1970s that assess the probability of states (low or high variance) persisting over time, providing a form of nowcasting.

Predictive Capabilities Beyond Current Models

• The speaker aims to enhance predictive capabilities beyond what traditional Markov switching regression can offer, emphasizing the need for early warnings about market shifts rather than just current conditions.

Challenges in Prediction During Market Crises

• The COVID-19 pandemic exemplified difficulties in prediction; traditional models struggled to account for unprecedented events like a global health crisis, highlighting limitations in existing methodologies.

Enhancements Through Technical Analysis

• The research involves enhancing Markov switching regression with technical analysis—often dismissed by academics but valued in industry—to improve accuracy regarding market regime transitions.

Issues with Traditional Models and New Approaches

• Traditional two-state models often produce erratic signals (e.g., alternating red/green), which may not be useful for long-term strategies. Enhanced three-state models aim to capture medium variance periods more effectively.

Transition Regimes and Smoothing Techniques

• By incorporating Kaufman's Adaptive Moving Average, the speaker developed a four-state model that better identifies transition regimes, offering insights into when markets might stabilize or become volatile.

Future Directions and Machine Learning Integration

Machine Learning in Finance: Key Models and Challenges

Importance of Accurate Classifications

• The need for accurate classifications in financial models is emphasized, as manual methods are insufficient for determining the start and end of events.

• Helpers or models are necessary to assist in this process, although some may not perform as well as others.

Common Machine Learning Models in Finance

• Discussion on standard machine learning models used in finance, including supervised, unsupervised, and reinforcement learning.

• Supervised learning is categorized into complex supervised learning and common supervised learning; the latter includes algorithms like random forest, extreme gradient boosting (XGB), and support vector machines.

Model Selection Considerations

• The "no free lunch" theorem suggests that no single model excels across all scenarios; thus, practitioners must choose based on specific needs.

• Neural networks represent a more complex form of supervised learning often applied to asset portfolio optimization due to their ability to handle smaller datasets effectively.

Unsupervised Learning Applications

• Unsupervised learning techniques such as hierarchical clustering are also utilized for portfolio optimization within finance.

Reinforcement Learning Insights

• Reinforcement learning is noted as an underexplored area in finance but holds potential for developing AI agents that can autonomously trade by learning from experiences.

Data Challenges in Financial Machine Learning

• Data acquisition poses significant challenges compared to other fields like image recognition; financial data often requires purchasing from providers or utilizing alternative datasets.

• The necessity of cleaning data before application highlights the complexity involved in preparing financial datasets for modeling purposes.

Historical Data Limitations

• Limited historical data availability complicates model training; many models require extensive historical context which may not be accessible beyond certain timeframes (e.g., 1990).

Emerging Markets Considerations