Historical Simulation, or Backtesting

What is one key difference between the two simulations?

So he gets an idea: what if this strategy was employed in the markets five years ago. What sort of success would have been realized by someone doing the exact same thing in real markets? Howe's idea is nothing new: it's a __historical simulation__. This compares to Monte Carlo simulation in what way?

That's right! Actual reality is as realistic as you can get, but there's only one. If you want to test a strategy multiple times, you're out of luck. Another drawback of historical simulation is you have to assume the recent past is always the best indication of the future. This is not always the case. Data from a 5-year bull run in the markets is not going to give you very useful information about what happens in a downturn, for example.

No, the beauty of Monte Carlo simulations is that the risk estimation comes from a distribution, so many numbers can be generated at the same time for a complete simulation.

Suppose that Howe Kudabin is running his historical simulation, and doesn't really like what he sees. So he changes a rule here, fixes something there, and keeps "tweaking" his model until he ends up with the best outcome over the past five years. Satisfied, he takes his strategy to the marketplace to start making money. What might be the most realistic outcome given these late efforts?

Not necessarily; he has built a model which is best given what the market _has done_. But there might be very different market conditions in the next five years.

Yes, this unfortunately sounds very possible. There might be very different market conditions in the next five years.

To summarize this discussion: [[summary]]

No, just the opposite, actually.

No, there are some key differences. Think of the steps of Monte Carlo simulation, and what is no longer needed.

No, this is actually a bit backwards. Historical simulation, true to its name, looks to the past.

Sometimes, the best simulation of reality is actual reality. Howe Kudabin has developed a new investment strategy. It's fairly complicated, with several investing rules and action triggers. There's just no way that Howe can solve explicitly for anything like a mean return and variance measure with just the underlying assumptions that he's using.

In putting all of his faith in a historical simulation, Howe has calibrated his model to a specific set of conditions, and has not done any sort of robustness checks. This is the benefit of performing a simulation that has repeated trials with different assumptions. A historical simulation can be a useful tool, but its limitations are not to be ignored.

Exactly! A Monte Carlo simulation will employ estimates of risk measures like market returns, where a historical simulation will simply use the actual market returns realized over a given time period. This is also referred to as __backtesting__ a strategy.

Monte Carlo simulations can be run just once, but historical simulations can be run multiple times for robustness

Monte Carlo simulations can be run multiple times for robustness, but historical simulations can be run just once

To run Monte Carlo simulations over and over, you constantly have to enter new risk estimates

Same thing, different name

Monte Carlo simulation requires estimated distributions for risk measures, where historical simulation uses the actual measures

Monte Carlo simulation uses real numbers, where historical simulation uses estimates

His model probably won't perform well, since it's specifically designed for the past five years

He has finally built a model which can handle exactly what the market will do

Continue

Continue

Continue

Continue