Grid Search & Overfitting

Grid Search & Overfitting

Grid search is the most straightforward approach to parameter optimization. It tests every combination in your defined range -- no shortcuts, no sampling, no guessing. This exhaustiveness is its strength and its risk.

How Grid Search Works

You define a range and step size for each parameter. For example, you might test an SMA period from 10 to 50 in steps of 5 (giving 9 values) and an RSI threshold from 20 to 40 in steps of 5 (giving 5 values). Grid search creates every possible combination of these values and runs a separate backtest for each one.

With 9 SMA values and 5 RSI values, that is 9 x 5 = 45 backtests. Each backtest uses a different parameter combination but the same strategy logic, date range, and market data. When all 45 finish, you can see exactly how performance varies across the entire parameter space.

The grid above shows a healthy pattern: a broad region of good performance in the center, with results declining gradually toward the edges. This means the strategy is not dependent on hitting one exact combination -- nearby values produce similar results. This is what a robust strategy looks like in parameter space.

The Combinatorial Explosion

Grid search is exhaustive, which means it scales multiplicatively. With one parameter and 10 values, you run 10 backtests. Add a second parameter with 10 values, and you run 100. A third brings it to 1,000. A fourth to 10,000.

This exponential growth is called the curse of dimensionality. It is not just a computational cost problem -- it is an overfitting problem. The more combinations you test, the more likely some will look good purely by chance.

Random Sampling: A Budget Instead of a Grid

Grid search's cost is set by the size of the grid, and the previous section showed how fast that grows. Random sampling breaks the link. Switch the Search Strategy on the optimization panel from "Exhaustive grid" to "Random sample", set a sample budget, and Simula draws that many combinations from your defined grid and runs only those. No combination is drawn twice, and the draw uses a technique called Latin hypercube sampling: rather than leaving coverage to chance, the sample is spread so that every parameter's range of values is covered evenly from end to end.

The compute cost is now set by the budget, not the grid. Sweep six parameters with 10 values each -- a 1,000,000-combination grid that could never run exhaustively -- and a budget of 2,000 runs exactly 2,000 backtests. The setup preview shows both numbers before you submit: the size of the defined grid, and how many combinations will actually be scanned, with the coverage percentage.

Why Partial Coverage Is Enough

You are not hunting for one perfect cell -- you are hunting for regions where performance is broadly good, like the plateau in the first diagram. Because the sample is spread evenly across the grid, the share of samples that lands in any region matches, on average, that region's share of the grid. A plateau covering 10% of the grid receives about 10% of your samples -- enough to show up clearly in the results.

The reverse works in your favor. A single lucky combination occupies one cell, so the chance it gets tested at all is your coverage percentage -- at 0.2% coverage, about one chance in 500. Random sampling naturally surfaces the broad regions worth trusting and skips over most of the isolated spikes you should not trust anyway.

Random sampling is also available for walk-forward analysis. Simula draws the sample once per run and tests the same set of combinations in every window, so when you compare which parameters won from window to window, you are comparing the same candidates throughout.

The results page adapts to sampled runs -- the heatmap and the robustness score both work differently when the grid is only partially covered. Reading Optimization Results covers the details.

The Overfitting Trap

Here is the core problem: when you test hundreds of parameter combinations on the same historical data, you are running hundreds of statistical experiments. Some will produce good results by random chance -- not because those parameters capture a real market pattern, but because they happened to align with the specific sequence of price movements in your data.

This is a form of data mining bias. If you flip a coin 100 times, you will find streaks of heads and tails. Those streaks are real in the data but meaningless as predictions. Similarly, if you test 1,000 parameter combinations, some will produce impressive Sharpe ratios and returns -- but the performance may not repeat on new data.

Degrees of Freedom

Every parameter you add gives the optimizer another dimension to exploit. With enough parameters, you can fit almost any dataset. A strategy with one tunable parameter can only bend so far to fit the data. A strategy with six tunable parameters can contort itself into shapes that perfectly match the historical prices but have no predictive value.

This is why simpler strategies are more trustworthy after optimization. A strategy with one or two parameters that produces good results across a broad range is much more likely to have a genuine edge than a complex strategy with many parameters that produces spectacular results at one specific combination.

The Sharp Peak Problem

When reviewing optimization results, the shape of the performance landscape matters more than the peak value. If the best combination sits on a broad plateau -- surrounded by other combinations that also perform well -- the strategy is capturing something real. The edge exists across a neighborhood of parameter values, not just one.

If the best combination is a sharp spike -- surrounded by mediocre or losing combinations -- the result is almost certainly overfit. The optimizer found one combination that happened to align with the historical data. Move one step in any direction and the performance vanishes. That is not a strategy with an edge. That is a strategy with a lucky accident.

Why Validation Is Non-Negotiable

Grid search tells you what worked in the past. It cannot tell you what will work in the future. The only way to distinguish real edges from data-mined artifacts is to test the optimized parameters on data the optimizer never saw.

A single in-sample / out-of-sample split is the minimum level of validation. But because a single split gives you only one data point, the gold standard is walk-forward analysis, which repeats the optimization-then-validate cycle across multiple time windows. If the optimized parameters hold up consistently on unseen data across multiple windows, you have much stronger evidence of a genuine edge.

Where to Go Next

Reading Optimization Results -- how to interpret heatmaps, identify plateaus vs spikes, and choose parameters.

Walk-Forward Analysis -- validating optimized parameters across rolling time windows.

Overfitting & Curve Fitting -- the broader problem that makes validation essential.

In-Sample vs Out-of-Sample Testing -- the simplest form of validation.