tradefloorStart free

Guides

Backtests and simulated markets

A backtest answers one question very well. It tells you what your strategy would have earned in the single history that happened, on prices your orders could not move.

That is a useful question. Backtests catch bugs, check an idea against real data and keep everyone honest about fees. But traders tend to ask a backtest three other questions as well, and it answers those with a confident silence that is easy to mistake for a yes.

A simulated market can answer them, because it generates the prices instead of replaying them. Here are the three, asked of tradefloor in one script.

article-backtest-vs-simulation.py

"""Three questions a backtest can't answer, asked of a simulated market.

1. Would my own orders have moved the price? Buy a large block with and
   without it in the market and price the difference.
2. Was that one result typical? Run the same strategy on 20 markets instead
   of one.
3. What if one thing had been different? Fork a market and raise rates in
   one copy.
"""
import struct

import tradefloor as tf

market = tf.Universe.random(20, seed=111)


# 1. A block of 20% of a day's volume in every name, priced against the same
#    market without it. A backtest fills this at the last price, for free.
class Block:
    def act(self, obs):
        if obs.day == 1 and obs.step_of_day == 0:
            return {t: round(0.20 * obs.avg_volume(t)) for t in obs.tickers}
        return None


block_cost_bps = round(tf.tca.analyse(Block(), seed=7, universe=market, days=2,
                                      cash=1e12, max_leverage=None).shortfall_bps(), 1)

# 2. One strategy, 20 markets: momentum against buy-and-hold, seed by seed.
SEEDS = list(range(20))
ranking = tf.rank(lambda: {k: v for k, v in tf.reference_agents(seed=3).items()
                           if k in ("momentum", "buy_and_hold")},
                  seeds=SEEDS, universe=market, days=20, workers=4)
gap_k = [round((m - b) / 1000, 1) for m, b in zip(ranking.records["momentum"].pnls,
                                                   ranking.records["buy_and_hold"].pnls)]
ahead = sum(g > 0 for g in gap_k)
best_k, worst_k = max(gap_k), min(gap_k)
worst_loss_k = -worst_k
per_seed = {str(s): g for s, g in zip(SEEDS, gap_k)}


# 3. One past, two futures: rates 2 points higher in one copy from day 60.
def index(engine, base):
    prices = struct.unpack(f"<{len(engine.tickers)}d", engine.prices())
    return round(100 * sum(p / b for p, b in zip(prices, base)) / len(base), 2)


engine = tf.Engine(seed=7, universe=market)
engine.run_days(60)
base = struct.unpack(f"<{len(engine.tickers)}d", engine.prices())
calm, hiked = tf.branch(engine, 2)
shock = tf.Scenario.load("rate_shock").starting_at(0)
paths = {"calm": [100.0], "hiked": [100.0]}
for day in range(40):
    shock.apply(hiked, day)
    calm.run_days(1)
    hiked.run_days(1)
    paths["calm"].append(index(calm, base))
    paths["hiked"].append(index(hiked, base))
rate_effect = round(paths["hiked"][-1] - paths["calm"][-1], 1)

print(f"1. a block of 20% of a day's volume cost {block_cost_bps} bp; a backtest says 0")
print(f"2. momentum against buy-and-hold on 20 markets: ahead on {ahead}, "
      f"best ${best_k:,.1f}k, worst ${worst_k:,.1f}k")
print(f"3. two points on rates, 40 days later: the index is {rate_effect:+.1f} points "
      "against the same market without it")

Output on tradefloor 0.10.2

1. a block of 20% of a day's volume cost 30.3 bp; a backtest says 0
2. momentum against buy-and-hold on 20 markets: ahead on 2, best $20.3k, worst $-231.8k
3. two points on rates, 40 days later: the index is -8.0 points against the same market without it

Your own orders and the price

The script buys 20% of a day's volume in every stock at once. In a backtest that fills at the last price, free of charge, every time. In tradefloor the order walks up the order book, and against the same market without the order it cost 30.3 basis points. That's small for one trade and large for a strategy that trades like this every day.

One result among many

A backtest is one market. It can't tell you whether its result was the usual outcome or the lucky one. The script runs a momentum rule against buy-and-hold on 20 generated markets instead.

Momentum came out ahead on 2 of 20. If your one backtest happened to be one of those, you saw a strategy that beat the market by up to $20.3 thousand. On the rest it lost, once by $231.8 thousand. Both stories come from the same rule, and only one of them belongs to the strategy.

One change and two futures

History can't be rerun with interest rates two points higher. A simulated market can. The script runs one market for 60 days, copies it, raises rates in one copy and runs both on for 40 days. The copies are identical up to the fork. Everything that differs afterwards is the rate rise.

Forty days later the index in the copy with higher rates sat -8.0 points below its twin. One caution: tradefloor's scenarios move prices by about half to four-fifths of what real ones did, in the right direction. Use a scenario to see how a strategy responds, and keep its size away from your forecasts.

Use both

None of this makes backtests worthless. A simulated market's prices are generated. However carefully they are checked against real markets, the record of what actually happened lives in the data your backtest replays. Backtest on real history to see whether an idea survived the actual past. Then use a simulation to ask the three questions a backtest can't, before the market asks them for you.

Running it yourself

Install the library with pip install tradefloor and run the script. It takes about half a minute. tradefloor compared sets it beside the main backtesters and simulators, and The Model shows how its markets are checked against real ones.