Skill or luck?
About this video
A momentum bot holds the 5 strongest stocks of the last 10 days. Claude runs it with tfrun on tradefloor's tf-quick benchmark, 8 fixed public markets of 20 days, and tries three changes one at a time, running the benchmark after each.
The score is the share of 10,000 random portfolios the bot beat on each market, averaged across the markets. A score of 50 means luck. Each run opens on tradefloor's Benchmarks page, which compares it with the run before on the same markets. None of the changes beats run 1, and Claude puts the original code back.
That code then runs three times on tf-quick sealed, which draws 8 fresh hidden markets of the same kind for every run and reveals them when the run ends.
Chapters
- The bot
- Tune
- Sealed
- Result
Figures on screen
Every figure here is on screen in the video and was checked against the product's own data.
- On tf-quick, run 1 with the original code scored 39, “No sign of skill”. Run 2 with
LOOKBACK = 5scored 28, run 3 with equal weights until 10 days of prices exist scored 37, and run 4 withTOP = 3scored 30. - On tf-quick sealed, the original code unchanged scored 51, 33 and 41, each “No sign of skill”, a spread of 18 points.
- On tf-quick the bot beat buy-and-hold on 5 of 8 markets, p = 0.36. On each sealed run it beat it on 4 of 8, p = 0.64.
The guide
Skill or luck in a trading strategy runs the same kind of test in Python on 40 generated markets, and shows how a single market can make either strategy look like a genius.
More demos
2:15Would my trading bot survive a crash?Branch a market into a rate shock and a liquidity crisis, run the same bot on every branch, ask Claude why it lost, and replay the crisis with Claude's one-line fix.
2:07Can Claude out-trade a bot?Claude, a momentum bot and a buy-and-hold bot trade one shared market for 25 days, straight through a liquidity crisis.
2:20Does my trading bot pass CI?A pull request raises a bot's leverage and tradefloor's stress suite fails it in CI. Claude fixes the bot from the CI log without loosening the check, and CI passes.
3:18A trading classA teacher sets a credit crisis from a template. One student's AI agent trades it, another trades by hand and hands in, and the teacher watches live and marks.
3:47No installA new account writes a trading bot in the browser, tests it against three crises in the same tab and asks Claude which one hurt, with nothing installed.