What Is Out-of-sample testing?

Out-of-sample testing evaluates a fully specified model or trading rule on observations that were not used to select or tune it. For time-ordered financial data, the evaluation period should be later than the development period so that the test reflects the information available when a live decision would have been made.

A valid out-of-sample test uses point-in-time data, realistic execution and cost assumptions, and no repeated tuning in response to its result. Walk-forward analysis repeats the train-then-test sequence through history and is often more informative than one holdout period. Reusing a test set until it guides model changes turns it into in-sample evidence.

See also

• In-sample testing

• Walk-forward analysis

• Backtest

Research and literature

• David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, and Qiji Jim Zhu, Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance, Notices of the American Mathematical Society 61(5), 2014.