Advertisement
Machine Learning

Regime Detection With Hidden Markov Models: Knowing When Your Strategy Should Sit Out

Mouad — EdgeQuant Trading · Oct 19, 2026 · 6 min read
Abstract financial chart showing shifting market data patterns

Every quant pitch deck has a version of the same slide: a two-color background showing "risk-on" and "risk-off" regimes, with a strategy equity curve that magically avoids every drawdown by switching sides right on cue. The model behind that slide is usually a Hidden Markov Model (HMM), and the slide is almost always lying by omission — not about the math, but about how the regime labels were assigned. Hamilton's 1989 regime-switching framework is real and has genuine uses in systematic trading. The gap between what gets published and what survives live trading is where most of the retail confusion sits.

What an HMM Actually Does

A Hidden Markov Model assumes your observed data (daily returns, realized volatility, whatever feature you feed it) is generated by one of a small number of unobserved "states," and that the market switches between those states according to a transition probability matrix. A standard 2-state Gaussian HMM on futures or equity returns typically settles into something like a low-volatility trending state and a high-volatility mean-reverting state. One widely cited applied study on futures markets found a persistent split of roughly 68% low-vol trending observations versus 32% high-vol mean-reverting observations, with distinct autocorrelation structures between the two states — the low-vol state shows positive serial correlation (trending), the high-vol state shows negative serial correlation (mean-reverting). Regime-conditional position sizing in that study improved the Sharpe ratio by about 0.18 and cut max drawdown by roughly 5.6 percentage points versus a naive constant-sizing baseline. That's a real, non-trivial risk-adjusted-return edge — but it's also a single in-sample study, and the number that matters more than the sizing improvement is how the parameters were estimated.

It's also worth being precise about what the regime split is actually capturing. A large part of why a 2-state model separates cleanly in the first place is volatility clustering — the well-documented tendency of large price moves to cluster together in time rather than appear independently. An HMM doesn't need to understand why volatility clusters; it just needs the clustering to be persistent enough that a transition matrix can pick it up as a sticky state. That's a feature of the data, not evidence the model has found anything causal about market structure.

The model is fit using the Baum-Welch algorithm (an EM variant), and the state sequence is inferred with the forward-backward algorithm. Critically, there are two different outputs here that get conflated constantly: smoothed probabilities (using the entire dataset, past and future, to estimate the state at time t) and filtered probabilities (using only data up to time t). Every backtest that looks great and then fails live is almost always using smoothed probabilities during development and filtered probabilities in production without adjusting for the difference in classification accuracy between the two.

Volatility graph on a trading screen illustrating regime shifts

The Three Failure Modes Nobody Talks About

A sober breakdown of regime-detection research identifies three specific ways these models get gamed, intentionally or not, before they ever reach a live account.

Post-hoc regime labeling. The most common error in published regime-switching results is assigning the regime label after the regime already ended, using outcome data that wouldn't have existed in real time. A strategy that "rotated into defensives entering the COVID drawdown" was almost certainly fit with the benefit of knowing COVID was coming. Run the same model honestly — classifying each date using only information available on that date — and performance drops dramatically, often to the point where the regime overlay adds nothing over a simple volatility filter.

Parameter explosion. A 2-state Gaussian HMM with a transition matrix already carries five to seven free parameters before you've specified anything about the asset universe. Add a third state, layer in extra features, and make transition dynamics asset-specific, and the parameter count quickly exceeds the number of genuinely independent regime episodes in your historical sample. A model with that many degrees of freedom relative to data can fit the past almost perfectly. Whether it has learned anything that generalizes is a separate, much harder question, and it's rarely tested with real discipline.

Hindsight in dataset selection. Researchers often pick the historical window to fit the model knowing which regimes occurred in that window. If you build and tune your HMM over 2007-2022, you've baked in the 2008 crash, the 2020 COVID crash, and the 2022 rate-hike bear market as "known" regime examples. The model looks robust because it was handed exactly the regime diversity it needed to be robust against.

Where It Actually Helps

None of this means regime detection is useless — it means the bar for calling it useful is higher than most backtests clear. Bayesian HMM approaches that output a time-varying probability of being in each regime (rather than a hard binary call), with explicit uncertainty bands, tend to be more honest tools in practice: on synthetic data with a known ground truth, one such Bayesian regime model correctly identified the true regime in 94% of months — a model-validation number, not a live-trading accuracy figure, and worth remembering as "this is how good the method can be when it's cheating," not a promise. The genuinely useful signal from most regime models isn't the hard regime call at all; it's the width of the uncertainty band, which tends to blow out right at regime turning points — precisely when position sizing should get more conservative, regardless of which regime the model thinks you're in next.

In practice, the more durable use of regime detection in a live systematic book is defensive, not predictive: using a filtered (not smoothed) volatility-regime signal to scale position size down when the model's confidence collapses, rather than trying to flip strategy direction on a state change. That's a smaller claim than the pitch-deck version, and it's also one that survives an honest forward test far more often.

What To Actually Check Before You Trust a Regime Model

If you're evaluating a regime-switching strategy — your own or someone else's — ask three questions before looking at the equity curve. First: were regime labels assigned using only data available at that point in time, or with the full sample? Second: how many free parameters does the model have relative to the number of distinct regime episodes in the backtest — if you count fewer than 10-15 genuinely independent regime transitions in your sample, a 3+ state model is almost certainly overfit. Third: was the fitting window chosen before or after the researcher knew which crises it contained? If the answers aren't clearly documented, treat the backtest the same way you'd treat a backtest with no transaction costs: directionally interesting, not something to size real risk against. This is fundamentally a model risk management question — the regime model itself is a source of risk, not just a tool for managing it, and the strategies that survive longest treat it that way.

hidden markov model regime detection regime switching quant trading volatility clustering