Market regime detection: testing Jev on markets with a known answer
Every trading floor has someone who can tell you what kind of market it is. Far fewer can tell you how often they've been right.
Real markets make that hard to check. Economists still argue about when some recessions started, years after they ended. A simulated market doesn't have that problem. You chose the scenario and you know the day it began, which means any call about it can be marked, and a classifier that gets marked can be fixed.
Jev, from TypeSafe, is a decision engine. It doesn't write text. You give it a state and a typed question, such as "pick one of these six regimes", and it returns a probability for every option and a confidence from 0 to 1. We asked it to name the market regime on 1,440 trading days, found the regime it couldn't see, rewrote the options, and marked the rewrite on 48 markets it had never been shown.
What we found
- It never named a recession. In round one, across 163 recession days, Jev said "recession" on none of them. It called every one a funding squeeze, sure of itself all the way.
- The fix was in the options. Rewritten in the words the market description uses, the options took Jev from 68% to 89% on fresh markets, and from 25% of recessions to 97%.
- Its confidence started to mean something. Before the rewrite, answers above a 0.6 gate were right 69% of the time and answers below it 71%. After it, 90% against 68%.
- Words beat figures. Given the market in words, Jev named the regime on 69% of days. Given the raw numbers, 51%.
The test
Six regimes, eight markets each, 30 days a market. Five regimes come from tradefloor's scenarios: a rate shock, a liquidity crisis, a recession, an oil price spike and a geopolitical conflict. The sixth is a market where nothing happens. Each scenario lands somewhere between days 8 and 12, and before that the right answer is "calm".
Each day Jev gets the market described two ways. One gives the raw figures: the VIX today and on day 0, interest rates, spreads. The other does the arithmetic first and hands over words, such as "far higher" or "up sharply", which is what TypeSafe's notes on Jev 1.13's weak spots recommend. A hand-written rule reads the same words. Its author had read the scenario files first, which no one trading a real market gets to do. Treat the rule as a ceiling more than a rival.
That's round one. Round two takes what round one taught us, rewrites the options, and runs on 48 new markets: the same six regimes on eight seeds the rewrite never saw. Without fresh markets, a better score would only show that we had tuned the question to the test.
article-decision-engine.py
"""Test a market regime classifier on markets where the right answer is known.
TypeSafe's Jev answers typed questions with a probability for every option
instead of writing text. Here it names the market's regime each day, and a
playbook in code sets how much of the account to hold in stocks for it.
The script chose each market's scenario, so it knows the regime on every
day. Each answer can be marked right or wrong, and its confidence set
against how often it was right. Round one finds what Jev gets wrong. Round
two rewrites the options and marks them on markets round one never saw.
Jev was asked once and its answers are saved beside this file. Later runs
read them: no API key, the same numbers.
"""
import datetime
import hashlib
import json
import os
import pathlib
import tradefloor as tf
from tradefloor.integrations.callable import callable_agent
MODEL = "jev-1.13.0" # pinned: a confidence gate is tuned to one version
GATE = 0.6 # act on an answer at least this confident
DAYS = 30
FIRST_SEEDS = range(8) # round one: find the mistakes
FRESH_SEEDS = range(8, 16) # round two: markets the rewrite was never shown
market = tf.Universe.random(10, seed=4242)
# The scenario that makes each regime happen. Calm is last, because Jev
# leans towards the first option when it is unsure.
SCENARIOS = {"rates_up": "rate_shock", "funding_squeeze": "liquidity_crisis",
"recession": "recession", "inflation_shock": "oil_price_spike",
"risk_off": "geopolitical_conflict", "calm": None}
REGIMES = list(SCENARIOS)
# Round one's options, described the way an analyst would.
FIRST = {
"rates_up": "The central bank has raised interest rates sharply.",
"funding_squeeze": "Money is hard to borrow: trading dries up and volatility jumps.",
"recession": "The economy is shrinking: earnings fall, credit gets much dearer "
"and volatility jumps.",
"inflation_shock": "Prices across the economy are rising fast, as after an oil shock.",
"risk_off": "Investors fear a conflict: volatility rises and credit gets dearer.",
"calm": "Nothing unusual is happening.",
}
# Round two's: the same regimes, in the words the description uses, written
# after looking at round one's markets and nothing else.
REWORDED = {
"rates_up": "The policy interest rate is up sharply.",
"funding_squeeze": "Volatility is far higher and bid-ask spreads much wider, "
"but corporate borrowing costs are up only a little.",
"recession": "Corporate borrowing costs are up sharply and volatility is far higher, "
"with the policy rate unchanged or cut.",
"inflation_shock": "Inflation is up, a little or sharply, while volatility and "
"spreads are much as usual.",
"risk_off": "Corporate borrowing costs are up sharply and volatility is higher but "
"not far higher. The policy rate hasn't risen.",
"calm": "Volatility, rates, borrowing costs and inflation are about where they started.",
}
# The playbook: the share of net worth to hold in stocks in each regime.
PLAYBOOK = {"calm": 1.0, "inflation_shock": 0.8, "rates_up": 0.7, "risk_off": 0.6,
"recession": 0.4, "funding_squeeze": 0.3}
def start_day(seed):
return 8 + seed % 5 # each market turns between days 8 and 12
def truth(regime, seed, day):
return regime if SCENARIOS[regime] and day >= start_day(seed) else "calm"
def scenario(regime, seed):
name = SCENARIOS[regime]
return {"scenario": tf.Scenario.load(name).starting_at(start_day(seed))} if name else {}
# 1. Two descriptions of the same day. Jev reads text well and arithmetic
# less well, so the second does the arithmetic in code and hands over words.
def figures(payload, first):
m, f, a = payload["macro"], first["macro"], payload["assets"]
return {"VIX": {"today": round(m["vix"], 1), "on day 0": round(f["vix"], 1)},
"policy rate %": {"today": round(m["federal_funds_rate"] * 100, 2),
"on day 0": round(f["federal_funds_rate"] * 100, 2)},
"corporate bond yield %": {"today": round(m["corporate_bond_yield"] * 100, 2),
"on day 0": round(f["corporate_bond_yield"] * 100, 2)},
"inflation %": {"today": round(m["inflation_rate"] * 100, 2),
"on day 0": round(f["inflation_rate"] * 100, 2)},
"average bid-ask spread, basis points": {"today": round(spread(a), 1),
"on day 0": round(spread(first["assets"]), 1)},
"stocks down over 2% in five days": f"{falling(a)} of {len(a)}",
"business cycle": m["cycle"]}
def words(payload, first):
m, f, a = payload["macro"], first["macro"], payload["assets"]
def points(key): # a change in percentage points, in words
move = (m[key] - f[key]) * 100
size = "unchanged" if abs(move) < 0.4 else "a little" if abs(move) < 1 else "sharply"
return size if size == "unchanged" else ("up " if move > 0 else "down ") + size
def times(now, then, small, large, same, more, most):
ratio = now / then
return same if ratio < small else more if ratio < large else most
return {"volatility (VIX)": times(m["vix"], f["vix"], 1.4, 2.5,
"about the same", "higher", "far higher"),
"policy interest rate": points("federal_funds_rate"),
"corporate borrowing cost": points("corporate_bond_yield"),
"inflation": points("inflation_rate"),
"bid-ask spreads": times(spread(a), spread(first["assets"]), 1.2, 1.45,
"as usual", "wider", "much wider"),
"stocks falling": ["few", "some", "most"][min(2, 3 * falling(a) // len(a))],
"business cycle": m["cycle"]}
def spread(assets):
quoted = [(x["best_ask"] - x["best_bid"]) / x["price"] * 1e4
for x in assets if x["best_ask"] and x["best_bid"]]
return sum(quoted) / len(quoted)
def falling(assets):
return sum((x["return_5d"] or 0) < -0.02 for x in assets)
# 2. Ask Jev, or read the answer saved when it was asked. An answer is saved
# under the state and the options together: new wording is a new question.
SAVED = pathlib.Path(__file__).with_name("article-decision-engine.json")
saved = json.loads(SAVED.read_text()) if SAVED.exists() else {"answers": dict()}
asked = 0
def ask_jev(state, options):
"""Jev's probability for each regime, and its confidence."""
global asked
question = json.dumps([state, list(options.items())], sort_keys=True)
key = hashlib.sha256(question.encode()).hexdigest()[:16]
if key not in saved["answers"]:
if not os.environ.get("TYPESAFE_API_KEY"):
raise SystemExit("an answer isn't saved: set TYPESAFE_API_KEY to record it")
from typesafe_sdk import Choice, TypeSafeClient # needed only to record
reply = TypeSafeClient(model=MODEL).system_one(state=state, questions={
"regime": Choice(instructions="Which best describes this market today?",
criteria=options)})
answer = reply.answers["regime"]
saved["answers"][key] = {"state": state, "options": "first" if options is FIRST else "reworded",
"model": reply.model, "probabilities": answer.probabilities,
"confidence": answer.confidence}
asked += 1
answer = saved["answers"][key]
return answer["probabilities"], answer["confidence"]
def best(probabilities):
return max(probabilities, key=probabilities.get)
# The rule a person writes after reading the scenario files: more than a
# real market would ever tell them.
def rule(w):
if w["volatility (VIX)"] == "far higher":
return "recession" if w["corporate borrowing cost"] == "up sharply" else "funding_squeeze"
if w["policy interest rate"] == "up sharply":
return "rates_up"
if w["corporate borrowing cost"] == "up sharply":
return "risk_off"
return "inflation_shock" if w["inflation"].startswith("up") else "calm"
# 3. Watch each market without trading, and mark every day's answers. A
# reader is one way of naming the regime from the day's descriptions.
BANDS = {"0-60": 0.6, "60-80": 0.8, "80-95": 0.95, "95-100": 1.01} # top probability, %
def mark(seeds, readers, banded):
"""How often each reader named the regime, and how sure `banded` was."""
seen = dict.fromkeys(REGIMES, 0)
right = {name: dict.fromkeys(REGIMES, 0) for name in readers}
bands = {band: {"answers": 0, "said": 0.0, "right": 0} for band in BANDS}
sure = {"days": 0, "right": 0} # the days `banded` cleared the gate
for regime in REGIMES:
for seed in seeds:
days_seen = []
tf.evaluate({"watch": callable_agent(lambda p: days_seen.append(p) or {"actions": []})},
seed=seed, universe=market, days=DAYS, **scenario(regime, seed))
for payload in days_seen:
actual = truth(regime, seed, payload["day"])
seen[actual] += 1
for name, read in readers.items():
probabilities, confidence = read(payload, days_seen[0])
right[name][actual] += best(probabilities) == actual
if name != banded:
continue
top = max(probabilities.values())
band = bands[next(b for b, below in BANDS.items() if top < below)]
band["answers"] += 1
band["said"] += top
band["right"] += best(probabilities) == actual
if confidence >= GATE:
sure["days"] += 1
sure["right"] += best(probabilities) == actual
days = sum(seen.values())
unsure_right = sum(right[banded].values()) - sure["right"]
return {"seen": seen, "right": right,
"overall": {name: round(100 * sum(n.values()) / days) for name, n in right.items()},
"named_right": {name: [round(100 * n[r] / seen[r]) for r in REGIMES]
for name, n in right.items()},
"calibration": {b: {"said": round(100 * v["said"] / v["answers"]),
"right": round(100 * v["right"] / v["answers"])}
for b, v in bands.items() if v["answers"]},
"gated_days": round(100 * sure["days"] / days),
"gated_right": round(100 * sure["right"] / sure["days"]),
"ungated_right": round(100 * unsure_right / (days - sure["days"]))}
def as_given(how, options=FIRST):
def read(payload, first):
return ask_jev(how(payload, first), options)
return read
def by_rule(payload, first):
return {rule(words(payload, first)): 1.0}, 1.0
round_one = mark(FIRST_SEEDS, {"words": as_given(words), "figures": as_given(figures),
"rule": by_rule}, banded="words")
round_two = mark(FRESH_SEEDS, {"first": as_given(words), "reworded": as_given(words, REWORDED),
"rule": by_rule}, banded="reworded")
days = sum(round_one["seen"].values())
overall, named_right, calibration = round_one["overall"], round_one["named_right"], round_one["calibration"]
seen, right = round_one["seen"], round_one["right"]
gated_days, gated_right, ungated_right = round_one["gated_days"], round_one["gated_right"], round_one["ungated_right"]
# How often an answer in each band was right, before the rewrite and after.
right_by_band = {"before": [calibration[b]["right"] for b in BANDS],
"after": [round_two["calibration"][b]["right"] for b in BANDS]}
# 4. Trade the fresh markets on the answers: hold the playbook's share for
# the regime named, in an equal-weight book, and skip trades too small
# to matter.
def rebalance(payload, share):
worth, assets = payload["portfolio"]["net_worth"], payload["assets"]
orders = []
for a in assets:
want = share * worth / len(assets) / a["price"] - a["position"]
quantity = int(min(abs(want), a["max_order_shares"]))
if quantity and abs(want) * a["price"] > 0.02 * worth / len(assets):
orders.append({"symbol": a["symbol"], "side": "BUY" if want > 0 else "SELL",
"quantity": quantity})
return orders
def trader(read=None):
days_seen = []
def act(payload):
days_seen.append(payload)
share = PLAYBOOK[best(read(payload, days_seen[0])[0])] if read else 1.0
return {"actions": rebalance(payload, share)}
return callable_agent(act)
TRADERS = {"always_in": lambda: trader(), "rule": lambda: trader(by_rule),
"first": lambda: trader(as_given(words)),
"reworded": lambda: trader(as_given(words, REWORDED))}
drawdown = {} # mean worst fall from a peak, %, by regime
for regime in REGIMES:
cards = [tf.evaluate({name: make() for name, make in TRADERS.items()},
seed=seed, universe=market, days=DAYS, **scenario(regime, seed))
for seed in FRESH_SEEDS]
drawdown[regime] = {name: round(sum(c[name].max_drawdown_pct for c in cards) / len(cards), 1)
for name in TRADERS}
mean_drawdown = {name: round(sum(d[name] for d in drawdown.values()) / len(drawdown), 2)
for name in TRADERS}
if asked:
saved["meta"] = {"model": MODEL, "sdk": "typesafe-sdk", "tradefloor": tf.__version__,
"recorded_utc": datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds")}
SAVED.write_text(json.dumps(saved, sort_keys=True, separators=(",", ":")))
markets = len(REGIMES) * (len(FIRST_SEEDS) + len(FRESH_SEEDS))
questions = len(saved["answers"])
print(f"{markets} markets, {questions} questions answered by {MODEL}")
print(f"round one, {days} days: figures {overall['figures']}%, words {overall['words']}%, "
f"rule {overall['rule']}%")
print(f" recessions named {right['words']['recession']} of {seen['recession']}")
print(f" confidence {GATE} or more on {gated_days}% of days: right {gated_right}% of those, "
f"{ungated_right}% of the rest")
for band, row in calibration.items():
print(f" top probability {band}%: said {row['said']}%, right {row['right']}%")
print(f"round two, fresh markets: first wording {round_two['overall']['first']}%, "
f"reworded {round_two['overall']['reworded']}%, rule {round_two['overall']['rule']}%")
print(f" recessions named {round_two['right']['reworded']['recession']} of {round_two['seen']['recession']}")
print(f" confidence {GATE} or more on {round_two['gated_days']}% of days: right {round_two['gated_right']}% "
f"of those, {round_two['ungated_right']}% of the rest")
for band, row in round_two["calibration"].items():
print(f" top probability {band}%: said {row['said']}%, right {row['right']}%")
print(" mean worst fall, %: " + ", ".join(f"{k} {v}" for k, v in mean_drawdown.items()))Output on tradefloor 0.10.2
96 markets, 1311 questions answered by jev-1.13.0
round one, 1440 days: figures 51%, words 69%, rule 89%
recessions named 0 of 163
confidence 0.6 or more on 85% of days: right 69% of those, 71% of the rest
top probability 0-60%: said 51%, right 94%
top probability 60-80%: said 69%, right 52%
top probability 80-95%: said 86%, right 11%
top probability 95-100%: said 100%, right 89%
round two, fresh markets: first wording 68%, reworded 89%, rule 92%
recessions named 154 of 159
confidence 0.6 or more on 95% of days: right 90% of those, 68% of the rest
top probability 0-60%: said 52%, right 61%
top probability 60-80%: said 71%, right 69%
top probability 80-95%: said 89%, right 73%
top probability 95-100%: said 99%, right 95%
mean worst fall, %: always_in 11.45, rule 9.6, first 8.68, reworded 9.52Round one: words against figures
Given words, Jev named the regime on 69% of days. Given the figures behind those words, 51%. TypeSafe's advice holds. Keep the arithmetic in code and let the engine read the result.
Round one: days on which the regime was named right
Eight markets of 30 days for each regime. Calm includes the days before each scenario began.
Jev, reading wordsJev, reading figuresThe rule
The overall figure hides a swap. Reading figures, Jev caught a conflict 100% of the time and a funding squeeze on 1% of days. Reading words, it was the other way round. How you describe the market decides which regimes the engine can see, not only how many. The average won't tell you which regime you just lost, and the chart above will.
The missing recession
Out of 163 recession days, Jev named a recession on none of them. Look at the two options it was choosing between.
- Funding squeeze. "Money is hard to borrow: trading dries up and volatility jumps."
- Recession. "The economy is shrinking: earnings fall, credit gets much dearer and volatility jumps."
Both end in the same three words. The rest of the recession option talks about earnings and the economy, and nothing in the description mentions either. Jev was asked to match a market described in volatility, spreads and borrowing costs against options written in an economist's vocabulary, and it matched on the only words the two had in common.
It was confident about it, too. Answers given with 80 to 95% probability averaged 86% and were right 11% of the time. A gate at 0.6 let through the wrong answers as readily as the right ones. Confidence says how clearly one option beat the others you offered, and it can't know that the right option was described in words that pointed somewhere else.
Round two: the options rewritten
The description stayed the same. Only the options changed, each rewritten in the description's own vocabulary from what round one's markets showed. A recession had pushed corporate borrowing costs up sharply and a funding squeeze had barely moved them, though both sent volatility far higher.
- Funding squeeze. "Volatility is far higher and bid-ask spreads much wider, but corporate borrowing costs are up only a little."
- Recession. "Corporate borrowing costs are up sharply and volatility is far higher, with the policy rate unchanged or cut."
The other four options got the same treatment. Then Jev answered both versions on the fresh markets, the old options and the new, alongside the rule.
Round two: the regime named right on fresh markets
Eight new markets of 30 days for each regime. The same description each time, with the old options or the new.
Jev, old optionsJev, new optionsThe rule
With the old options Jev named the regime on 68% of the fresh days, about where round one left it. With the new ones, 89%. Recessions went from 25% to 97%, which beats the rule's 77%. On a recession's quieter days volatility is only higher, and the rule, which keys on volatility first, calls those days risk-off. Jev weighed the spreads and the borrowing costs as well.
The rewrite moved the rule's knowledge into the options, written as descriptions instead of if-statements. That is a fair thing to do. The options are where a decision engine expects you to put what you know about the problem, and it is the part of the setup most people write once and never test.
Confidence after the rewrite
The bigger change was in how far Jev's confidence could be trusted.
How often an answer was right, by how sure Jev said it was
Every answer on the word descriptions, grouped by its top probability. Round one with the old options, round two with the new.
Before the rewriteAfter the rewrite
Answers at 80 to 95% went from right 11% of the time to 73%, close to the 89% Jev claimed. The gate started doing its job. Answers that cleared 0.6 were right 90% of the time and the rest 68%, where in round one the answers below the gate had done slightly better.
Jev itself didn't change between the rounds. The options did, and its confidence measures the options as much as the market. A gate tuned on badly written options learns nothing about the engine, which is a good reason to fix the options before you tune the gate.
The two regimes left
Two regimes stayed hard. Jev named the inflation shock on 40% of its days and the rule 50%. An oil spike takes weeks to reach the inflation figures, and for most of those 30 days the description still reads as calm. No wording of the options can name what the description doesn't show, and this one needs a description that notices an oil spike sooner.
Risk-off, the geopolitical conflict, fell to 74% against the rule's 100%. A conflict also sends borrowing costs up sharply, and Jev called it a recession even on days when volatility was only higher. That was the one detail the new options used to tell the two apart, and Jev let it slide. Fixing one boundary moved some of the trouble to the next one along. A test across every regime caught it, where an average would have counted round two a clean win.
Trading on it
Four traders ran the fresh markets, each holding a playbook's share in stocks for the regime it named, from 100% when calm down to 30% in a funding squeeze. Their worst fall from a peak averaged 11.4% for staying fully in, 9.6% for the rule, 9.5% for Jev with the new options and 8.7% with the old. The old options fell least because their favourite mistake, calling everything a funding squeeze, holds the least in stocks. In a falling market that is the right way to be wrong. The gaps are a point or two over eight markets a regime, enough to show that a hit rate and a portfolio count mistakes differently and not enough to crown anyone.
Questions about market regime detection
What is market regime detection?
Naming the kind of market you're in, such as calm, a rate shock or a recession, from what the market shows today. A trading system uses the answer to change how much risk it takes. The hard part is checking the answers, because in real markets the regime is often only agreed on years later.
How do you test a market regime classifier?
Run it on markets where you know the regime. tradefloor generates markets and lets you set a scenario, such as a recession or a liquidity crisis, to start on a given day. The regime on every day is then known, so each answer can be marked and its confidence compared with how often it was right. Look at every regime, not just the average, and test any fix on fresh markets the fix was never shown.
Can you trust a decision engine's confidence score?
Only after you've measured it on your own questions. In this test Jev's answers at 80 to 95% probability were right 11% of the time with the first options and 73% after they were rewritten. Confidence says how clearly one option beat the others you offered, which makes it only as good as the options.
What is Jev?
Jev is a decision engine from TypeSafe. It answers typed questions about a state: a Choice picks one option from a set you define, a Score rates the state on levels you define, and a Noul gives the probability that a statement is true. It returns probabilities and a confidence instead of text, and nothing it says needs parsing. Input costs 42 dollars per billion tokens and output is free.
Is a decision engine better than an LLM for a trading agent?
They do different jobs. An LLM agent writes the decision, orders and reasons included. With a decision engine your code makes the decision and the engine answers narrow questions inside it, such as which regime this is. The playbook stays in code you can read and test, and every answer arrives with a probability.
Running it yourself
Install the library with pip install tradefloor and run the script. It reads Jev's answers from article-decision-engine.json, which sits beside it in the repository. It needs no API key and prints the numbers above. Jev doesn't always give the same answer to the same question, and a recording is what keeps this page's numbers fixed.
To ask Jev yourself, pip install typesafe-sdk, set TYPESAFE_API_KEY and change something: an option's wording, the words in a description, the regimes. Only questions it hasn't seen go to Jev, and the 1,311 behind this page cost a few cents at TypeSafe's price of 42 dollars per billion input tokens. The inflation shock is the obvious place to start. Mark any change on seeds from 16 up, which neither round has used.
Decision engines in the docs covers wiring an engine into an agent, gating on confidence, and recording answers so a run replays without a key.