LEARN/EVALUATE

Evaluate

Four calls turn a market into a result: score one, rank twelve, price the execution, replay it later.

tf.evaluate

Score agents on one identical market.

scores = tf.evaluate(
    agents: dict[str, Agent | StrategySpec],
    *, seed: int, universe: Universe,
    days: int, scenario: Scenario | None = None,
    model: str | None = None)

agents A dict of name to agent, or name to StrategySpec. A spec carries its own fingerprint into the result.

one engine each Every agent gets its own Engine built from the same seed and roster, so nobody eats another agent's depth.

returns A dict of name to Scorecard.

tf.reference_agents(seed=3) The five shipped baselines as a ready dict: hold, random, momentum, mean_reversion and oracle.

tf.capture_ratio(scores) Each agent's P&L as a fraction of the Oracle's.

One market is one sample. Use rank before you call a winner.

tf.rank

Repeat that across twelve seeds, and separate with a sign test.

ranking = tf.rank(
    factory: Callable[[], dict],
    *, seeds: Iterable[int], universe: Universe,
    days: int, workers: int = 1)

ranking.report()
ranking.separation("mean_reversion", "momentum")

factory A callable returning a fresh agent dict per seed, because an agent that carries state across markets is not being measured on either.

report() Pooled captures, their per-seed ranges, and how many seeds each agent topped.

separation(a, b) A paired sign test: wins, losses, ties and a p-value.

workers Processes. Seeds are independent, so this scales linearly.

On the published twelve-market grid, mean-reversion pools at +0.783 and momentum at +0.259, and mean-reversion wins 9 to 3 at p = 0.15. One seed picks the pooled leader eight times in twelve.

tf.tca.analyse

Price the execution against the world where you never traded.

ex = tf.tca.analyse(
    algo, *, seed: int,
    universe: Universe, days: int)

ex.shortfall_bps()
ex.by_step()
ex.partial_fills()

algo Any object with an act method.

shortfall_bps() What the footprint cost, against the same seed run with no orders.

by_step() Where it was paid, per decision.

partial_fills() What you asked for against what you got.

Round-trip shortfall measured over eight simulation seeds runs -17.72 to +2.03 bps. The range crosses zero, so one run tells you nothing.

tf.RunManifest

Write the run down so someone else can replay it.

manifest = tf.RunManifest.of(
    engine, spec, seed=7)

manifest.to_json(path)
pt.RunManifest.from_json(text)
pt.reproduce(manifest)

carries Package version, preset, seed, universe fingerprint, macro conditions, scenario and strategy fingerprint, with the expected digest.

reproduce() Raises on a mismatch and names the component that disagreed.

era_fingerprint() The digest of a small fixed probe, recomputed before every replay, so two builds that agree on the probe agree on the arithmetic.

A manifest that quietly reproduced a different market would be worse than no manifest, so it refuses rather than warning.

What a Scorecard carries

return_pct What it made, as a percentage.

pnl What it made, in currency.

impact_bps What its own footprint cost, in basis points.

trades How many fills it took.

strategy_fingerprint A sha256 over the spec. This is the one to cite.

universe_fingerprint A sha256 over the roster, order included.

model The preset the run used, so a result names its own coefficients.

strategy_fingerprint is empty for a hand-built agent, because that result is reproducible only by citing code at a commit. A StrategySpec gives you one, which is why the spec grammar exists.