Ask a quant what keeps them up at night and regime detection will be somewhere near the top of the list. Markets don't behave the same way all the time — anyone who traded through 2008, March 2020, or the 2022 rate shock knows that viscerally. The question of how to detect when the regime has shifted, programmatically, before your portfolio is already bleeding, is genuinely hard.

Hamilton's 1989 Hidden Markov Model framework gave us a mathematically elegant answer: assume markets switch between latent states — say, a "calm" regime and a "volatile" regime — and use the return series itself to infer which state you're probably in at any given time. The model estimates transition probabilities between states and the statistical properties of each. On paper, brilliant. On live ASX sector data, immediately humbling.

CONCEPTHMMs estimate the probability you're in a given market regime using only observed returns — the states themselves are never directly visible.
WARNINGRegime labels are assigned in-sample first — applying them out-of-sample on ASX data often produces regime switches that lag the actual market turn by weeks.
KEY IDEAThe number of regimes you specify (K) is a modelling assumption, not something the data tells you — and getting K wrong poisons everything downstream.

The first practical wall you hit is model selection. Do you specify two regimes or three? Materials sector returns on the ASX carry a commodity-cycle overlay that arguably warrants a third "transition" state sitting between bull and bear. But add states liberally and your expectation-maximisation algorithm starts finding regimes that exist in the maths but not in any recognisable market reality. It's like asking someone to sort a bag of mixed nuts into exactly five categories — eventually "large cashew" becomes its own regime.

Regime Probability: Calm vs Volatile StateJanMarMayJulSep1.00.50.0Calm stateVolatile state

The second wall is estimation instability. ASX sector return series — particularly Small Industrials or Healthcare — are short relative to what the EM algorithm really wants. Fit an HMM on five years of weekly Healthcare sector returns and the transition probability matrix can shift dramatically depending on whether you include the COVID crash window. The model is technically correct; it's just telling you something different each time you feed it a slightly different sample. Robustness checks aren't optional here — they're the whole job.

For traders navigating this seriously, the canonical starting point remains Hamilton's original paper, which is contextualised well on Wikipedia's Hidden Markov model page. The statistical mechanics of regime-switching returns are covered thoroughly on Investopedia's regime definition, and the broader family of statistical tools used in quantitative finance provides essential context for where HMMs sit in a practitioner's toolkit. The pragmatic implementation lesson is this: treat your regime assignments as probabilistic signals, not binary flags, and never trade a hard rule off a soft probability.

HMMs don't fail because the theory is wrong — they fail when traders forget the model sees the past clearly and the present only dimly.

This content is for educational purposes only and does not constitute financial product advice. Past performance is not indicative of future results. Profit Logic Ltd (ACN 688 669 936) accepts no responsibility for errors or omissions in this content or anywhere on this website. Always seek advice from a licensed financial adviser before making investment decisions.