We can think of backtesting as attempting to build a bridge from the past to the future. Bridges don’t always work.
Figure 2: Tacoma Narrows Bridge on 1940 November 7 (6 minutes).
The typical backtest will yield a plot of results like Figure 3.
Does the strategy work for the first quarter?
We get deceived into thinking that Figure 3 answers those questions. It doesn’t.
The effect of the strategy is confounded with other things, most notably the performance of the portfolio that we start with.
Figure 4: Strategy performance and 95% confidence interval for zero effect.
Figure 4 indicates the extent of paths that have no strategy at all, just random trading like the trading done with the strategy.
Figure 5: Strategy performance relative to mean random path.
Figure 5 gives the performance of the strategy that we are really looking for. It has substantial differences from Figure 3 (the impostor).
Actually, we want to do this whole process several times with different starting portfolios because the effect of the strategy may well depend on the starting portfolio. This is discussed in the backtesting talk.
Photo in Figure 1 from istockphoto.com.