Every compliance officer in financial services eventually lands on their desk a glossy vendor pitch deck showing a backtest curve that goes up and to the right like a ski jump ramp. The question — "is this legitimate?" — sounds simple. It absolutely is not. Backtested performance sits at a uniquely uncomfortable intersection of statistics, regulation, and motivated reasoning.
The direct answer: most backtested performance claims from algorithmic strategy vendors require serious scrutiny before they mean anything at all. ASIC's Regulatory Guide 53 governs how financial performance is presented to retail and wholesale clients, and it draws a firm line between genuine information and misleading representations. A backtest, by its nature, is a simulation — and simulations can be tuned, accidentally or otherwise, to look spectacular.
Think of it like a restaurant showing you photos of every dish taken by a professional food stylist under ideal lighting. The meal might still be good — but you'd want to taste it before writing the review. Backtests are the food photography of trading. The real meal is live, audited performance, with all the cold chips and rushed plating that implies.
When reviewing a vendor submission, your checklist should cover six hard questions. First: does the backtest account for realistic slippage and brokerage costs? Second: was the strategy developed on in-sample data and tested on genuinely out-of-sample data? Third: how many parameter combinations were tested before this "optimal" version was presented? That last one exposes data mining bias, the silent killer of backtest credibility. Fourth: does the Sharpe ratio hold up when transaction costs are realistic? Fifth: is there any live, audited track record — even six months — that roughly resembles the backtest? Sixth: does the strategy's logic make economic sense, or is it a pattern with no underlying cause? Understanding overfitting in algorithmic models is essential here, because a curve-fitted strategy can look perfect on paper and collapse immediately in live markets. RG 53 requires that performance representations not be misleading, which means your sign-off carries real weight — the mechanics of backtesting need to be disclosed clearly to any client who sees these figures.
Document every question you asked and every answer you received. If a vendor can't explain their methodology in plain language, that's your answer right there.
The best compliance officers treat backtest reviews like a detective interview — polite, methodical, and deeply sceptical of a story that has no holes in it.
This content is for educational purposes only and does not constitute financial product advice. Past performance is not indicative of future results. Profit Logic Ltd (ACN 688 669 936) accepts no responsibility for errors or omissions in this content or anywhere on this website. Always seek advice from a licensed financial adviser before making investment decisions.