Skill or luck in a trading strategy

A strategy that wins once has proven one thing beyond doubt. It had a good day, and so did the person who wrote it.

A backtest is a single market. However long the history, it's one draw of everything that could have happened, and a strategy that looks brilliant on that draw might look ordinary on the next one. The honest way to find out is to run it on many markets and count. The counting is cheap. tradefloor will generate as many markets as you like from seeds.

Two rules on forty markets

Two classic rules go head to head: momentum buys what went up, and mean reversion buys what went down. They disagree about everything, so surely one of them is right. Each plays the same 40 markets of 20 days, generated from 40 seeds, and buy-and-hold comes along for scale.

"""Skill or luck: how many markets it takes to tell two strategies apart.

Momentum, mean reversion and buy-and-hold run on the same 40 markets of 20
days each, so every comparison is paired. Two questions. Can the paired
test tell momentum from mean reversion at all? And what does a real
difference look like next to that?
"""
import tradefloor as tf

SEEDS = list(range(100, 140))
DAYS = 20
market = tf.Universe.random(30, seed=11)


def entrants():
    agents = tf.reference_agents(seed=3)
    return {name: agents[name] for name in ("momentum", "mean_reversion", "buy_and_hold")}


ranking = tf.rank(entrants, seeds=SEEDS, universe=market, days=DAYS, workers=4)
pnl = {name: list(rec.pnls) for name, rec in ranking.records.items()}

mr_wins = sum(a > b for a, b in zip(pnl["mean_reversion"], pnl["momentum"]))
mom_wins = len(SEEDS) - mr_wins
test = ranking.separation("mean_reversion", "momentum")

hold_wins = sum(h > m for h, m in zip(pnl["buy_and_hold"], pnl["momentum"]))
hold_test = ranking.separation("buy_and_hold", "momentum")

# The running score: after each market, the share of markets so far that
# mean reversion won. A real edge would pull away from 50 and stay there.
running = []
for n in range(1, len(SEEDS) + 1):
    won = sum(a > b for a, b in zip(pnl["mean_reversion"][:n], pnl["momentum"][:n]))
    running.append(round(100 * won / n, 1))
score = {"mean_reversion": running, "even": [50.0] * len(SEEDS)}
early_high = max(running[:10])
early_low = min(running[:10])

best_single = max(range(len(SEEDS)), key=lambda i: pnl["momentum"][i] - pnl["mean_reversion"][i])
best_margin = round(pnl["momentum"][best_single] - pnl["mean_reversion"][best_single])
per_market = [round((a - b) / 1000, 1) for a, b in zip(pnl["mean_reversion"], pnl["momentum"])]

print(f"{len(SEEDS)} markets, {DAYS} days each")
print(f"mean reversion beat momentum on {mr_wins}, momentum won {mom_wins}")
print(f"paired sign test: p = {test['p_value']:.2g}")
print(f"momentum's best market: ahead by ${best_margin:,}")
print(f"buy-and-hold beat momentum on {hold_wins} of {len(SEEDS)}, p = {hold_test['p_value']:.2g}")
print(f"mean reversion's share of wins in the first 10 markets ran from "
      f"{early_low:.0f}% to {early_high:.0f}%, and ended at {running[-1]:.0f}%")

40 markets, 20 days each
mean reversion beat momentum on 18, momentum won 22
paired sign test: p = 0.64
momentum's best market: ahead by $102,353
buy-and-hold beat momentum on 39 of 40, p = 7.5e-11
mean reversion's share of wins in the first 10 markets ran from 33% to 100%, and ended at 45%

The running score

Here is mean reversion's share of wins against momentum, recounted after every market.

After one market, mean reversion was undefeated. A backtest that happened to be that market would have produced a confident blog post and possibly a fund. By the sixth market it had won a third of them. By the end it had won 18 of 40, and the paired sign test gives p = 0.64. In plain words, these two rules can't be told apart on this evidence. Momentum's best single market put it ahead by $102,353, which is a lot of money to make from a coin toss.

A real difference for comparison

Buy-and-hold beat momentum on 39 of 40 markets. That is what an actual edge looks like when you count properly. It shows up on almost every market, and no amount of squinting is needed. Momentum here trades every day and pays the spread each time to chase moves, and buy-and-hold pays it once.

The number of markets to run

More than you'd like, and fewer than you'd fear. A real edge like buy-and-hold's shows within a handful of markets. Two strategies that are secretly the same can look different for a long time, which is the whole problem, and a few habits help with it.

• Run 30 or more seeds, the same ones for every strategy, and compare them seed by seed.
• Report the paired count and its sign test beside any average profit. An average can be carried by one lucky market. A count of 25 out of 30 can't.
• Commit to the seeds before you run. Then nobody can suspect you of picking the flattering ones. That includes you, six months from now.

Running it yourself

Install the library with pip install tradefloor and run the script. It takes about a minute and a half. Compare strategies explains tf.rank and the sign test in detail, and the video Skill or luck shows one unchanged bot scoring anywhere from 33 to 51 on the hosted app's sealed benchmarks, which draw fresh hidden markets for every run.