Limitations & Pitfalls

Lookahead bias, survivorship bias, overfitting, regime change, and other pitfalls that silently inflate backtest results.

Limitations & Pitfalls

A backtest can tell you exactly what would have happened in the past. What it cannot do is guarantee that those results mean anything for the future. The gap between backtest results and real-world performance comes from a handful of well-known pitfalls, each of which silently inflates your numbers if you are not aware of it.

This article walks through the most common ones. Understanding them is the difference between treating a backtest as a rough estimate and mistaking it for a crystal ball.

Lookahead Bias

Lookahead bias happens when a strategy uses information that would not have been available at the time of the trade. It is the single most dangerous bias in backtesting because it can make a worthless strategy look like a money printer.

Imagine you are testing a strategy that buys when tomorrow's close is higher than today's. In a backtest, tomorrow's close is already in the dataset -- the strategy can peek ahead and make a perfect decision every time. In real trading, that data does not exist yet. The result is a backtest that shows incredible returns from a strategy that is impossible to execute live.

Simula's engine processes candles in strict chronological order and never exposes future data to your conditions. At candle 50, the engine has no knowledge of candle 51. This eliminates lookahead bias at the engine level. But it is still possible to introduce lookahead bias through your strategy design -- for example, choosing a date range or asset because you already know what happened during that period. The strategy itself did not peek at future data, but the person who designed it did.

The engine prevents lookahead -- your setup might not

Even though the engine itself never looks ahead, your strategy design can still introduce subtle forms of lookahead. If you are testing on a specific date range because you already know what happened during that period, you are making decisions informed by future knowledge. See Overfitting & Curve Fitting for more on this.

Survivorship Bias

Survivorship bias occurs when you only test on assets that still exist today, ignoring the ones that failed. The assets that survived are, by definition, the ones that did not go to zero. Testing exclusively on survivors makes your strategy look better than it would have performed in real time, because the worst outcomes are missing from the data.

A classic example: testing a stock strategy on today's S&P 500 members. The current list only includes companies that are large and successful enough to remain in the index. Companies that went bankrupt, delisted, or shrank out of the index are not in your dataset -- but they would have been in the index when your strategy was supposedly trading them.

This bias is most relevant when testing on stocks. For forex and crypto, the effect is smaller -- major currency pairs do not delist, though crypto tokens can and do go to zero. If you are testing on a cryptocurrency that still exists and is still traded, keep in mind that you are testing on a survivor.

How do I avoid survivorship bias?

Be aware of what your dataset leaves out. Survivorship bias creeps in when you only test on assets that are still around today -- the delisted, bankrupt, or gone-to-zero names never make it into the sample, so the survivors make the market look healthier than it really was. When you read a stock backtest, ask whether the same test run on the full universe -- including the companies that failed partway through -- would look as good. Testing only on today's index members quietly omits everyone who dropped out.

Overfitting

Overfitting -- also called curve fitting -- is what happens when you adjust a strategy's parameters until it performs perfectly on the data you are testing. The strategy is not capturing a real pattern in the market. It is memorizing the specific price movements in your dataset.

Think of it like studying for an exam by memorizing the answer key instead of understanding the material. You will score 100% on that exact test, but you will fail any other version of it.

The more parameters your strategy has, the easier it is to overfit. A strategy with two conditions and fixed parameters has limited room to mold itself to the data. A strategy with six indicators, each with adjustable lookback periods and thresholds, has enough flexibility to match almost any historical curve -- and almost no chance of replicating that performance on new data.

The results look amazing -- should I be worried?

If a backtest shows extremely high returns with very low drawdowns, that is exactly the result overfitting produces. The stronger the results look relative to the market, the more important it is to test on data the strategy has never seen. Without that step, the numbers are not evidence of a good strategy -- they are evidence of a good fit to one specific dataset.

The most reliable defense against overfitting is to test your strategy on data it was not designed on. In-sample vs out-of-sample testing splits your data into a training set and a validation set. Walk-forward analysis goes further by repeating this split across multiple time windows, giving you a much clearer picture of whether the strategy generalizes or just memorized one period.

For a deeper look at why overfitting happens and how to detect it, see Overfitting & Curve Fitting.

Regime Change

Markets do not behave the same way all the time. Prices alternate between trending phases (sustained moves in one direction), ranging phases (sideways chop), and high-volatility phases (sharp, unpredictable swings). These broad behavioral patterns are called market regimes.

A strategy built during a trending market may rely on sustained momentum to generate returns. When the market shifts to a ranging phase, that same momentum simply is not there, and the strategy enters and exits repeatedly without the follow-through it needs. The reverse is equally true -- a mean-reversion strategy that thrives in a range will give back its gains when a strong trend develops.

The danger is that your backtest might cover a period dominated by a single regime. The results look strong because the strategy was well-matched to the conditions. But those conditions are not permanent. A two-year backtest during a bull market tells you how the strategy performs in a bull market -- it tells you nothing about a crash, a bear market, or a period of low volatility.

The first line of defense is testing over a longer date range that includes multiple market phases. If your backtest covers both trending and ranging periods, the results reflect a mix of conditions rather than a single favorable one.

Beyond that, Monte Carlo simulation stress-tests your results by reshuffling the sequence of trades. This reveals how sensitive your strategy's performance is to the specific order in which trades occurred. A strategy that only works when trades happen in a particular sequence is fragile -- and fragile strategies break when the market regime shifts.

Underestimating Trading Costs

Every real trade has costs: spreads, commissions, and slippage. If your backtest does not model these, or models them too low, the results are optimistic by definition. The gap between your backtest and reality grows with every trade.

This matters most for strategies that trade frequently. A strategy that opens and closes 500 positions over a year is paying trading costs 500 times. Even a small per-trade cost -- a fraction of a percent -- compounds into a meaningful drag on returns when multiplied across hundreds of trades. A strategy that looked profitable with zero costs can easily become a net loser once realistic costs are applied.

When configuring your backtest, set spread, commission, and slippage values that match or slightly exceed what your broker charges. It is better to overestimate costs in a backtest than to underestimate them. If the strategy is still profitable with conservative cost assumptions, the results are more likely to hold up in live trading.

A quick cost check

When costs are enabled, the Transaction Cost Analysis card in your results shows gross return (before costs) alongside net return and a per-cost breakdown. If the total cost drag is a large fraction of your gross return, the strategy's edge is thin and may not survive real-world execution.

Too Few Trades

A strategy that takes five trades and wins all five has a 100% win rate. That sounds impressive, but five trades is not enough data to draw any conclusion. Five coin flips can come up heads every time by pure chance.

The fewer trades a backtest produces, the more the results are influenced by randomness rather than by the strategy's logic. A high Sharpe ratio or a perfect win rate means very little when it is based on a handful of trades. You need a large enough sample for the numbers to separate signal from noise.

If your backtest is producing too few trades, there are two levers: extend the date range to cover more history, or use a shorter timeframe (more candles means more opportunities for the strategy to trigger). Both approaches give the strategy more data to interact with, which produces more trades and more statistically meaningful results.

For a deeper look at how many trades you need and why, see Statistical Significance & Trade Count.

The Open Position at the End

Historical data always stops on some candle, and your strategy may still be holding a position when it does. Simula closes that position at the final candle's price -- a Backtest End exit -- and counts its unrealized profit or loss like any other trade. This is the honest choice: the alternative, silently dropping the position, would hide real risk and make the trade table disagree with the return.

The pitfall is mistaking that final trade for a decision your rules made. A Backtest End exit only marks where the data ran out -- the price could have kept moving in your favor or against you. If that last position is sitting on a large loss, that loss is part of your result, even though your exit rules never fired.

Short windows feel this more

The shorter the test, the larger a single open-at-the-end position looms in the final numbers. This is especially true for walk-forward analysis, where each out-of-sample window is a hard date cut -- not a point your strategy chose -- so a window can end with a position still open. Do not read too much into a result that hinges on where the data happened to stop -- a longer date range gives the strategy room to actually close its trades on its own terms.

Putting It Together

None of these pitfalls are fatal on their own. The danger is in not knowing they exist. A backtest that shows strong results might be genuinely strong -- or it might be benefiting from one or more of these biases without you realizing it.

The checklist is short: make sure your strategy is not peeking at future data, that you are not only testing on survivors, that your parameters are not molded to one specific dataset, that you have tested across different market conditions, that your cost settings are realistic, and that you have enough trades for the results to be meaningful.

The tools that help you verify these things -- out-of-sample testing, walk-forward analysis, Monte Carlo simulation -- exist precisely because backtests are easy to get wrong without them. The backtest itself gives you a starting point. These tools tell you whether that starting point is worth trusting.

Where to Go Next

Overfitting & Curve Fitting -- a deep dive into why strategies memorize data and how to detect it.

In-Sample vs Out-of-Sample Testing -- how splitting your data protects against overfitting.

Spreads, Commissions & Slippage -- configuring realistic trading costs.

Statistical Significance & Trade Count -- how many trades you need for results to be meaningful.

Monte Carlo Simulation -- stress-testing your results against randomness and reordering.