Train and Evaluate AI on Real-World Financial Reasoning
A longitudinal dataset of frontier AI models making real financial decisions over time — complete with written rationales, portfolio state, independent judge evaluations, market context, and subsequent outcomes.
Built for model evaluation, agent research, fine-tuning, reward modeling, and financial-AI research.
Growing every trading day. Every Season 2 contestant is independently evaluated by a multi-model judge panel. Counts as of 2026-09-14.
This isn't stock-tip data. It's longitudinal AI decision-making data: how frontier models reason, decide, get evaluated, adapt, and fail under real-world uncertainty. Financial markets are the experimental environment.
Two ways to get the data
Designed for organizations, labs, universities, and commercial research teams. The historical dataset is the primary product; the API is for teams that need the benchmark's new observations as they are produced.
Complete Historical Dataset
$3,000one-time
The complete AIStockChallenge research dataset across Season 1 and Season 2, through the purchase date, with a perpetual license to the included AIStockChallenge-generated data.
Season 1 complete historical dataset
Season 2 complete historical dataset through purchase date
Model BUY / SELL / HOLD decisions with written rationales
Portfolio state, positions, and daily value series
Decision timestamps and latency (Season 2)
Model and provider metadata
Independent judge evaluations and scores
Judge-level scores and panel agreement (Season 2)
Eleven evaluation dimensions and claim-level support analysis
Evidence, process, outcome, and efficiency scores where available
Daily leaderboard snapshots
Normalized structured data for ML pipelines
Documentation and data dictionary
Delivery formats: JSON / JSONL / CSV(Parquet on request)
Broad commercial and research usage rights. Use the dataset in your research, models, internal systems, commercial products, publications, evaluations, and training pipelines. Raw redistribution as a standalone dataset is not permitted. Third-party source material that cannot legally be redistributed is excluded or represented through derived metadata.
Secure checkout via Stripe · Commercial license included · Delivered by email within one business day
Need an invoice, purchase order, or institutional purchase? .
Updated Every Trading Day
Live Research Data API
Custom Pricing
Continuously updated AI decision and evaluation data delivered as the benchmark runs. For AI labs, research teams, quantitative platforms, universities, and companies running ongoing model-evaluation pipelines.
New contestant decisions
Written rationales
Portfolio state changes
Judge evaluations and scores
Outcomes and performance
Experiment and model metadata
Historical dataset access
Machine-readable responses
Daily automated delivery
Delivery options: REST API · JSON · Webhook · Daily Data Drop
Pricing depends on usage, delivery requirements, and organization size.
Most AI Benchmarks Measure Answers. AIStockChallenge Captures Trajectories.
Traditional benchmarks evaluate a model on isolated prompts. AIStockChallenge observes models operating repeatedly in a changing environment, and records the whole trajectory.
Every model in the dataset operated under conditions where:
new information arrives every day
models make decisions repeatedly
prior actions affect future state
models maintain portfolio context
uncertainty cannot be removed
outcomes arrive after the decision
reasoning quality can be compared with later results
multiple models operate under comparable experimental conditions
Market Context
→
Model Stateholdings, cash, rank, own notes
→
Written Rationale
→
DecisionBUY / SELL / HOLD
→
Portfolio Change
→
Independent Judge Evaluationthree-model panel
→
Future Outcomedaily value series
A longitudinal record of AI reasoning → action → evaluation → outcome.
Reasoning Isn't Just Collected. It's Evaluated.
AIStockChallenge doesn't simply record what a model decided. Season 2 decisions are independently evaluated by multiple AI judges across dimensions including evidence quality, reasoning quality, process discipline, risk awareness, efficiency, and eventual outcomes. Season 1 carries monthly single-judge evaluations on the same rubric.
Each evaluation document in the dataset contains, where available:
Headline judge scores for reasoning quality, evidence grounding, outcome, and data reliability
Judge-level evaluations: each panel member's independent scores and badge (Season 2)
Panel agreement: min, median, max, range, and standard deviation across judges (Season 2)
Eleven evaluation dimensions, from thesis quality and claim grounding to portfolio discipline and temporal consistency
Claim-level support analysis: individual factual claims marked supported, partially supported, or unsupported, with the market data used
Evidence quality: evidence the model used versus evidence its declared strategy called for
Ignored risks: risks visible in the market data the model did not address, plus concentration and sizing risks
Decision and process quality: cash discipline, position sizing, thesis continuity, update quality
Efficiency and latency: median decision latency and a reasoning-efficiency score (Season 2)
Subsequent outcomes: the outcome score, and the daily portfolio value series every decision joins to
Lookahead check: whether the judge confirmed no post-decision data leaked into the evaluation
Strengths, weaknesses, and what would improve the score, in structured form
A real Season 2 record: Anthropic Claude Sonnet 4.6 on Adobe, as it appears in the free sample. This is what a static benchmark cannot give you.
Model claim
“Adobe trades at roughly a forward P/E ~10x and PEG ~0.55, exceptionally low for quality software.”
from the written rationale · claim_type: valuation
↓
Model action
BUY 98 ADBE at $202.73 · 2026-06-29
later: SELL 16 at $291.52 (trim) · 2026-08-29
↓
Independent judge panel · as of 2026-09-12
62/100Evidence grounding
58/100Risk awareness
Partially supportedClaim support
59 – 76Panel range, median 65
judge note: “Forward P/E is even lower (9.31), PEG a bit higher (0.621 vs 0.55). Directionally supported.” · ignored risk flagged: high sector concentration in technology
↓
Future outcome
Price at trim +43.8% vs. entry · judge outcome score 68/100
the portfolio value series keeps recording after the evaluation, every trading day
Connect what the model believed, what it did, how independent judges evaluated the reasoning, and what happened afterward.
Real record from the shipped sample (decisions.jsonl, judge_evaluations/); the claim, verdict, scores, and trades are reproduced as stored. Values shown are panel medians unless noted.
Two Seasons, Two Types of Research
SEASON 1 · COMPLETED
Season 1 — Different Strategies
The original longitudinal experiment: three AI-driven investment approaches (fundamental, news-driven, trend-following), each with its own strategy brief, run daily over an extended live-market period. Evaluated monthly by a single LLM judge.
A more controlled model-comparison experiment. Frontier models from OpenAI, Anthropic, xAI, and Google receive the same financial-reasoning prompt and the same provided market information, with the same starting capital, so the model is the variable under study. Every trading day, an anonymized three-judge panel scores each model's full decision record, and decision latency is timed.
Decisions, portfolio state, and judge evaluations are separate record types that join on model id and date. Below is one decision joined to its context, its evaluation, and what followed.
{
"date": "2026-06-29",
"season": 2,
"model_id": "s2-anthropic-claude",
"provider": "anthropic",
"symbol": "ADBE",
"type": "BUY", "shares": 98, "price": 202.73,
"model_context": { holdings, cash, leaderboard rank, contextual_memory the model wrote earlier },
"reason": "Adobe trades at roughly a forward P/E ~10x and PEG ~0.55, exceptionally low for quality software …",
"analysis": "Establish a core software value position",
"portfolio_before": { holdings, currentValue },
"portfolio_after": { holdings, currentValue, performanceHistory[] that follows },
"latency_seconds": 96.5,
"judge_evaluation": {
"as_of_date": "2026-09-12", "lookahead_check": { "passed": true },
"scores": { "reasoning_quality": 68, "evidence_grounding": 62, "outcome": 68, "reasoning_efficiency": 0 },
"dimension_scores": { "thesis_quality": 72, "risk_awareness": 58, "claim_grounding": 63, … 8 more },
"claim_ledger": [ { "claim": "Adobe trades at roughly a forward P/E ~10x …", "support_status": "partially_supported", "market_data_used": ["ADBE.forwardPE=9.31"] } ],
"panel": { "num_judges": 3, "judges": [ { "judge_id": "openai-gpt-5", "evidence_grounding": 62 }, … ], "agreement": { "composite_median": 65, "composite_range": 17 } }
},
"future_outcome": { subsequent decisions on the symbol, daily portfolio values after the decision }
}
Illustrative schema — see the data dictionary for actual available fields. The values are real (Season 2, Anthropic Claude Sonnet 4.6, from the shipped sample); the joined shape is one way to assemble the underlying decision, portfolio, and evaluation records.
Built for AI Research and Development
Study how frontier AI models behave when uncertainty is real and outcomes arrive later.
Model Evaluation
Compare new models against the recorded behavior of frontier models under identical conditions, not on isolated prompts.
Agent Research
Study long-horizon decision making where today's action changes tomorrow's context and outcomes arrive later.
Reward Models
Relate independent judge scores, claim-level support, actions, and eventual outcomes to build or test reward signals.
Fine-Tuning
Train on written rationales paired with decisions, evaluations, and what happened next.
Financial AI
Develop and evaluate financial-reasoning systems on real longitudinal examples with recorded outcomes.
AI Safety and Reliability
Analyze consistency, calibration, risk awareness, evidence use, and adaptation over time, model by model.
Who This Is For
Teams that study how models reason and act. It is not built for retail trading.
Foundation-model teamsAI labsModel-evaluation teamsAgent researchersFintech companiesFinancial AI startupsQuantitative researchersUniversitiesAcademic AI researchersReward-model researchersAI safety and reliability researchers
See the Data Before You Buy
Representative production records · JSONL + CSV · Real schema · No email required
The sample demonstrates the full longitudinal chain, model context → rationale → decision → judge evaluation → outcome, with real records: decisions and written rationales from every model in both seasons, a portfolio snapshot with its daily value and latency series, daily leaderboard snapshots, the latest complete judge-panel evaluation of every Season 2 model plus one Season 1 evaluation (all dimensions, claim ledgers, and panel agreement), an evaluation time series, and the full data dictionary.
Its purpose is to show quality and schema, not volume.
The historical dataset includes a perpetual license to use the included AIStockChallenge-generated data for research and commercial purposes, including:
internal research
AI evaluation and benchmarking
machine learning, model training, and fine-tuning
academic publications
commercial applications
analytics
derived features, embeddings, and models trained using the data
transformed internal datasets
No per-seat pricing
No API-call fees for the downloaded historical dataset
No annual renewal
No attribution requirement unless legally necessary
Create derived features, embeddings, models, analyses, and transformed datasets for use within your products, research, and internal systems.
Raw or substantially equivalent redistribution of the AIStockChallenge dataset as a standalone competing dataset is not permitted.
The license covers AIStockChallenge-generated content: model decisions and rationales, portfolio state and value series, leaderboard snapshots, and judge evaluations. The daily AI market analyses published on the site are not part of the dataset. Third-party data remains subject to its respective rights; the downloadable dataset contains only material AIStockChallenge is permitted to redistribute. Vendor market-data snapshots (quotes, fundamentals, news feeds) are not included; prices at decision time and the market facts a model cited survive in the generated records. AIStockChallenge is a paper-trading AI-evaluation benchmark; nothing in the dataset is investment advice.
Frequently Asked Questions
What do I receive for $3,000?
The complete downloadable AIStockChallenge historical dataset covering Season 1 and Season 2 through the date specified on the purchase page, plus documentation and the data dictionary.
Is $3,000 a subscription?
No. It is a one-time purchase.
How do I pay?
Card checkout is handled by Stripe; click any purchase button. Institutional buyers who need an invoice or purchase order use the contact form on this page and we issue an invoice instead. Either way the dataset is delivered by email, typically within one business day.
What is the refund policy?
The dataset is a digital product and is non-refundable once delivered. If you have not yet received the archive and change your mind, contact us and we will refund the purchase.
Can my company use the dataset commercially?
Yes. The included AIStockChallenge-generated dataset may be used for commercial and research purposes under the dataset license.
Can we use the data to train or fine-tune AI models?
Yes. Models, embeddings, features, and transformed datasets you build from it are yours to use in your products, research, and internal systems.
Can universities use it for published research?
Yes.
Can we redistribute the dataset?
No. Raw or substantially equivalent redistribution of the dataset as a standalone competing dataset is not permitted. Publishing analyses, excerpts, figures, and derived results is fine.
Does the historical purchase include future data?
No. The one-time dataset contains the historical archive through its stated cutoff date. For continuously updated data, use the Live Research Data API.
How often is the API updated?
Every trading day as new AIStockChallenge benchmark observations are produced.
How much does API access cost?
API pricing depends on usage, delivery requirements, and organization size. Contact us for API pricing.
Does this contain investment recommendations?
No. AIStockChallenge is an AI-model evaluation and research benchmark. The data describes experimental model behavior and paper-trading results and should not be treated as investment advice.
Build on the Dataset Behind AIStockChallenge
Two seasons of frontier-model decisions, written rationales, independent evaluations, portfolio actions, and outcomes — packaged for research and machine learning.