Overfitting is the adaptation of a model or decision rules to peculiarities of the available development sample at the expense of generalizing the relationship to further data. It also affects selection among candidates; quality cannot be judged solely by the result that simultaneously determined the winner.
QuantConnect describes Overfitting as fitting a limited training set too closely, capturing its details and noise at the expense of results on new data. Our distinction: the problem is not learning from history itself, but mistaking a property of a particular sample for a transferable regularity. In trading, even a manually assembled set of conditions can be overfit, rather than only a complex automated model. A handsome curve or good development-data score is therefore not yet independent confirmation of a procedure. [1]
The scikit-learn documentation shows the difference between an overly simple model unable to capture a relationship and a model sensitive to particular training samples. Our implication: parameter count alone is insufficient to diagnose overfitting. Simplifying or restricting a model’s flexibility can help, but can also remove a useful relationship. The decision should follow a predefined evaluation on suitable separate data. Even subsequently choosing the simplest variant based on an already inspected test does not restore that test’s independent role. [2]
Bailey and coauthors highlight false discoveries when strategies are repeatedly tried on the same data. Our methodological conclusion: record not only the winner’s parameters but also previous candidates, rule changes and the selection criterion. Even a simple final strategy can result from an extensive search. The selection pressure is not visible in its final description. The probability of overfitting cannot be determined from the number of variants alone; data, their dependencies and the search method also matter. [3]
The scikit-learn nested-validation example separates inner parameter selection from outer evaluation of the whole selection procedure. Using the same results both to choose and to present quality produces an overly optimistic score. Our requirement is to describe this boundary before the experiment and include preprocessing and metric choice in selection. If we further change the design based on the outer result, that information too has entered development. A nested structure does not permit repeated tuning against the same final test. [4]
The scikit-learn documentation warns that conventional splitting that assumes independent samples can distort generalization estimates for time-dependent data. Our trading-test check therefore preserves information order and examines whether every decision input and every target outcome used for training or rule selection was already available at the relevant time. A later period should not select a rule for an earlier moment. Walk-forward Analysis helps describe this sequence, but repeated adjustments to its protocol based on the same results are development again, rather than new independent evidence. [5]
scikit-learn requires learning transformations, such as normalization or feature selection, only on the training part before applying them to the test. Our distinction: leakage of unavailable information and overfitting are related but different problems. A strategy can fit noise without access to future rows; leakage can in turn produce an optimistic score even for a simple model. We therefore examine the whole procedure from data preparation to the final decision, rather than only the table split. [6]
Validation and learning curves in scikit-learn compare behavior on training and validation parts as settings or data quantity change. Our interpretation boundary: poorer subsequent performance alone does not prove a single cause. Markets, data quality or execution costs may change; chance can affect a short segment. We examine stability around neighboring settings and differences in conditions, but record every new search after viewing results. We do not present diagnosis as automatic statistical certification. [2]
NFA warns that hypothetical results are not actual trading and cannot fully capture liquidity, slippage or reactions to real losses. Our report therefore separates development and subsequent results and states all known trials, costs and data limitations. Any further assessment must have its rules recorded before it begins. Lower methodological-error risk is not the same as a demonstrated trading edge. Even previously unused data cannot guarantee a result under future market conditions. [7]
The selection winner need not win the subsequent test
For the clearest picture, read this entry together with Backtesting, Walk-forward Analysis, Look-ahead Bias, Paper Trading, Trading Journal. The reverse links also lead from Technical analysis, Technical indicator, Moving average crossover, Relative Strength Index.
01Can a simple strategy also be overfit?+
Yes. A simple final rule may have been selected from many attempts based on the same data. The entire search process matters, rather than just the number of parameters in the published version.
02Does every poor subsequent result prove overfitting?+
No. It may also relate to market changes, costs, faulty data or a short random segment. The methodology and conditions must be examined; a single performance difference determines neither the cause nor the probability of overfitting.