Comparing Strategies

Comparing Strategies

Sooner or later you have two candidates: a trend system and a reversion system, or two versions of the same strategy with different filters. Deciding which one deserves your capital sounds simple -- run both, keep the one with the bigger number. This article is about why that fails, which numbers actually separate two strategies, and the comparison traps that make the wrong strategy look like the winner.

Make It a Fair Race

A comparison only means something when the two backtests differ in exactly one way: the strategy logic. Everything else must match -- the same instrument, the same timeframe and date range, the same starting capital, the same spread, commission, and slippage, and the same leverage. Every mismatched setting is a second variable, and with two variables changed you cannot attribute the difference in results to either one. The classic self-deception is comparing your new idea on recent data against an old result you ran months ago on a different range -- the comparison feels rigorous and measures nothing.

Look Past Total Return

Total return is the number everyone looks at first, and it is the least informative number on the results page, because it says nothing about what the strategy put you through to get there.

The two curves above finish at exactly the same +30%. Strategy A never fell more than 7% below its own peak; Strategy B spent the middle of the test 28.6% below its peak -- below its starting capital -- before recovering. If those were live accounts, most traders would have abandoned B at the bottom, which means B's +30% was never really available to them. The comparison metrics below exist to expose exactly this difference.

Risk-adjusted return. The Sharpe ratio divides return by volatility -- return per unit of turbulence. The Sortino ratio counts only downside swings, and the Calmar ratio divides return by max drawdown. These are the headline numbers for comparing two strategies, because they reward the same return achieved with less pain.

Downside risk. Max drawdown is the worst peak-to-trough fall -- the number that decides whether you could psychologically survive the strategy. CVaR describes the tail: how bad the bad days are.

Trade quality. Profit factor and win rate with expectancy describe the engine room: how the wins and losses are distributed. Average holding period tells you whether the two strategies even operate on the same rhythm.

Efficiency. Exposure is the share of candles with a position open. A strategy that makes +35% while in the market 40% of the time is extracting more from each unit of market risk than one that needs to be invested constantly -- and the idle capital is not locked to the strategy.

Against doing nothing. Both strategies should also be compared against the benchmark of just buying and holding the instrument. Alpha and beta formalize that comparison. A strategy that underperforms buy-and-hold with more effort and more risk is not a strategy -- it is a hobby with fees.

A Worked Comparison

Here is the kind of table this article wants you to be able to read. Strategy A is a trend follower, Strategy B a mean-reverter, both run on the same instrument, range, and costs:

Neither strategy dominates. A made more money, but B made its money far more smoothly: half the drawdown and a much higher Sharpe. B's 190 trades also give its statistics far more evidential weight than A's 60 -- with few trades, a couple of lucky winners can carry the whole result. The win rates and profit factors tell the style story from the previous articles: A wins rarely but its implied average win is nearly three times its average loss; B wins constantly with wins smaller than its losses. Which one "wins" depends on the question. Maximum growth and a stomach for -18%? A. Smoother compounding you will actually stick with? B.

Cherry-Picking Traps

The Best-of-Twenty Trap

Test twenty variants of an idea and one of them will look great -- guaranteed, even if none has real edge. Somebody has to finish first, and with twenty entrants the winner owes a large share of its lead to luck. The polite name for this is selection bias; the practical consequence is that the winner of a variant tournament reliably disappoints on new data. This is the same mechanism that makes naive grid-search optimization dangerous, and it applies with full force to manual experimentation -- twenty backtests run by hand are a grid search with worse record-keeping.

Date-Range Shopping

Every strategy has windows where it shines. Compare a trend follower and a mean-reverter over a year that trended, and the trend follower wins by construction -- reverse the window, reverse the verdict. If a conclusion flips when the date range moves, the conclusion was about the window, not the strategies. Compare over a range long enough to contain both kinds of market, then check that the ranking holds on sub-periods.

The Tuned-vs-Untuned Trap

A strategy whose parameters were optimized on the test window carries an unfair advantage over one that was not -- it has effectively seen the answers. Comparing a freshly tuned variant against an untouched baseline on the tuning data measures the tuning, not the edge. The clean protocol is in-sample vs. out-of-sample: tune on one period, compare on data neither strategy has seen.

Robustness Is the Tie-Breaker

When two strategies are close on the headline metrics, the better question is not "which backtest is prettier" but "which result is more likely to be real." Three tests answer that. Monte Carlo reshuffles each strategy's trades to show how much of the equity curve's shape was sequence luck -- and whether the drawdown you saw was the typical case or a kind one. Walk-forward analysis re-tunes and re-tests each strategy across rolling windows, producing a series of out-of-sample results instead of one potentially lucky one. And parameter stability asks whether the strategy works across a neighborhood of settings or only at one magic combination. A strategy that survives all three with slightly worse headline numbers is the better bet over one that shines once and falls apart under inspection.

Comparing in Simula

The workflow: save both strategies so each is reproducible, run them with identical settings, and keep the runs in your backtest history so the results sit side by side instead of relying on memory. Reading Your Results walks through every tab the comparison draws on. When one strategy loses, resist resurrecting it with "just one more filter" -- that road leads back through the overfitting traps above.

Common Mistakes

One-number verdicts. Total return without drawdown, Sharpe, and trade count is a coin flip with confidence.

Unequal settings. Different date ranges, costs, or leverage make the comparison meaningless -- one variable at a time.

Ignoring sample size. A 60-trade result and a 190-trade result do not deserve equal trust, whatever the metrics say.

Calling noise a winner. If two strategies are within a few percent of each other, the honest verdict is a tie -- pick by drawdown, robustness, or the one you understand well enough to keep trading in a losing month.

Where to Go Next

Reading Your Results -- a tour of every tab these comparisons pull numbers from.

Sharpe Ratio and Max Drawdown -- the two metrics that separate the equal-return curves above.

Statistical Significance & Trade Count -- how much evidence a backtest actually contains.

Monte Carlo: Interpreting Results -- separating skill from sequence luck before declaring a winner.

Walk-Forward Analysis -- the strongest form of the fair-race idea: repeated out-of-sample comparisons.

Benchmarks & Relative Performance -- the comparison every strategy must win first: against doing nothing.