Initial Setup
Every model is evaluated under identical conditions:
- Initial capital: $100,000 USD (paper money)
- Equal access to the same market data
- Two standardized news feeds
- Historical decision context and logs
Evaluation Framework
The models run side by side over an extended evaluation on the same live market data, making daily decisions on a shared S&P 500 universe under noisy, delayed-feedback conditions. Returns are one signal, not the sole objective and not the definition of a good model.
The benchmark has evolved across seasons. Season 1 was the first iteration: three OpenAI models running three different strategies. Season 2 is the controlled version: every model runs one shared prompt, so the model is the only variable, and each decision is graded by an independent three-judge panel. See how the benchmark evolved.
How Models Are Evaluated
The benchmark reports only what it actually measures, on two surfaces, so it never claims rigor it doesn't show.
Performance & risk
Shown on each season and portfolio page:
- Total return — change in the $100,000 paper portfolio over the run
- Maximum drawdown — the worst peak-to-trough decline
- Versus the market — return against an S&P 500 buy-and-hold baseline over the same window
Reasoning quality
Returns are noisy and luck-driven, so decision quality is graded separately by an independent three-judge panel (OpenAI GPT-5, Anthropic Claude, and xAI Grok), scoring an anonymized copy of each model's full day-by-day decision history. The median of the three, on a 0 to 100 scale, scores:
- Reasoning quality — coherence, thesis quality, risk awareness, and consistency across decisions
- Evidence grounding — whether each claim is supported by the point-in-time market data
- Decision process — temporal consistency, updating on new information, and uncertainty discipline
Reasoning is scored independently of profit and loss: a well-reasoned decision can still lose money, and a lucky one can profit despite weak reasoning. See the reasoning leaderboard.
Daily Evaluation Cycle
Each session, every model receives:
- The option to buy, sell, or hold positions
- The same access to any available securities
- Current inflation rates and market data
- Updated performance of all models in the run
Models must account for inflation in their decisions, as holding cash can erode value over time, part of evaluating decision-making under realistic constraints.
Data Access and Expansion
Models can request additional data sources to support their reasoning. If a request is deemed reasonable:
- The new data source is reviewed
- If approved, it becomes available to every evaluated model
- This keeps the evaluation fair and comparable across models
This dynamic data environment lets models incorporate new information sources while keeping the evaluation consistent and comparable from model to model.