Significance Testing: Is My Outperformance Real?

The statistical significance test in the Monte Carlo tab -- how it judges whether your edge over buy-and-hold could be luck, how to read the p-value, and why it cannot tell you whether a strategy is overfit.

Is My Outperformance Real?

When you run a backtest with Monte Carlo enabled, the results tab shows a Statistical Significance section with two numbers. They answer a question a single backtest cannot: did your strategy genuinely beat the market, or did it just get a lucky run of trades?

A backtest gives you one history -- one specific sequence of trades that happened to occur. A handful of fortunate trades can make an ordinary strategy look excellent. The significance test puts a number on how easily luck alone could have produced your result.

What the Test Does

The test rebuilds 1,000 alternate versions of your trading history. It does this by drawing from your actual per-candle results, with replacement, in short blocks that preserve winning and losing streaks. This resampling technique is called the bootstrap.

On each alternate history, it re-measures your edge over buying and holding the asset. The p-value is the fraction of those alternate histories where the edge vanished -- dropped to zero or turned negative.

If your edge survives almost every resampling, the p-value is low: the result is hard to explain as an accident of which trades landed where. If the edge disappears easily, the p-value is high: what you saw is well within what luck alone could produce.

Separate from the Monte Carlo resampling method

This test always resamples your per-candle returns in blocks. It is computed independently of the Reshuffle / Resample method you choose for the Monte Carlo simulation: that method resamples your completed trades to build the equity bands and drawdown distribution, while this test resamples the per-candle return series to judge significance. They are separate passes, so switching the Monte Carlo method does not change these p-values.

Measured on unleveraged returns

The test runs once, on the 1x (unleveraged) return stream, even if your backtest used higher leverage. Whether an edge exists is a property of the strategy itself. Leverage scales the size of the wins and losses, but it does not create or destroy an edge, so there is one significance verdict rather than one per leverage level.

The Two Numbers

Simula reports two p-values because there are two different ways to define your edge, and a strategy can look strong on one and weak on the other.

Beat Buy & Hold

This measures your outperformance over simply buying the asset and holding it for the same period. The gap between your strategy's return and the buy-and-hold return is your active return. The p-value asks how easily that active return could vanish under resampling.

Risk-Adjusted Alpha

This is a stricter test. Part of any strategy's return comes from simply being exposed to the market while it moves -- a quantity measured by beta. Alpha is the return that remains after the part explained by that market exposure is stripped out.

The Beat Buy & Hold number credits any outperformance, including outperformance that comes purely from market exposure. The Risk-Adjusted Alpha number only credits the return that exposure does not explain. A strategy that beats buy-and-hold mainly by staying heavily exposed during a market rise can score well on the first and poorly on the second.

Reading the P-Value

The p-value runs from 0 to 1, and lower is stronger evidence. Simula labels it for you so you do not have to memorize cutoffs.

Significant (below 0.05)

Your edge survived in more than 95% of the resampled histories. Strong evidence that the outperformance is not just an accident of trade sequence.

Marginal (0.05 to 0.10)

Borderline. There is some evidence of an edge, but a meaningful share of resampled histories still saw it disappear. Treat the result as suggestive, not settled.

Not significant (above 0.10)

The edge vanished easily under resampling. The outperformance you saw is well within the range that luck alone could produce, so it is not reliable evidence of a real edge.

A p-value is not your odds of being profitable

The most common mistake is reading a p-value of 0.04 as "a 96% chance my strategy is good" or "a 96% chance it will make money." It means neither. It says that if you resampled your own history, the edge held up in 96% of those alternate runs and vanished in 4%. It is a statement about how fragile your measured edge is -- not a guarantee about future profits.

The Caveat That Matters Most: Overfitting

This is the single most important thing to understand about the test. It looks at one strategy -- the exact one you ran. It has no idea how many strategies you tried before you arrived at this one.

Suppose you tested twenty indicator combinations and kept the one that looked best. That selection is itself a hidden source of luck the test cannot see. Try enough strategies and one of them will pass a significance test by chance alone -- and it will be the one you kept, precisely because it looked the strongest.

So read a low p-value as "this result is probably not a fluke of this single backtest." Never read it as "this strategy is good" or "this strategy is not overfit." Those are different claims that this test does not make.

Overfitting needs a different tool

The question "did I curve-fit this to past data?" is answered by walk-forward analysis, which checks whether your strategy holds up on data it was never tuned on. The significance test and walk-forward answer different questions -- run both for a complete picture.

When the Test Cannot Run

The test needs enough data to resample meaningfully. If your backtest produced too few candles -- a short date range on a long timeframe -- or if you ran it without 1x leverage selected, the significance section will say it could not run and give a short reason.

A longer date range or a shorter timeframe gives the test more to work with. The same data shortage that blocks the significance test also makes the rest of your results less trustworthy, which is covered in Statistical Significance and Trade Count.

Common Mistakes

Treating the p-value as a probability of future profit. It describes how fragile your measured edge is on past data, not what happens next.

Ignoring the overfitting caveat after optimizing. A significant p-value on a strategy you selected from many attempts is exactly the situation where the test is most likely to mislead you.

Trusting a significant result from very few trades. A handful of trades can look significant and still tell you almost nothing, because there was never much data to resample in the first place.

Where to Go Next

How It Works -- the resampling method behind Monte Carlo and this test.

Interpreting Results -- the confidence bands, ruin probability, and drawdown distributions shown alongside the significance numbers.

Walk-Forward Analysis -- the tool that addresses overfitting, the question this test cannot answer.

Statistical Significance and Trade Count -- why a small number of trades undermines any result.

Alpha and Beta -- the market-exposure-adjusted return behind the Risk-Adjusted Alpha number.