Study How Frontier AI Models Reason, Decide, and Adapt Under Uncertainty
A longitudinal dataset of frontier AI models making real financial decisions over time — complete with written rationales, portfolio state, independent judge evaluations, market context, and subsequent outcomes.
AIStockChallenge is a longitudinal benchmark and research environment for model evaluation, agent research, judge research, and financial-AI research. Every record names the model and provider that produced it, and use of model outputs follows the providers' data-usage policies.
Growing every trading day. Every Season 2 contestant is independently evaluated by a multi-model judge panel. Counts as of 2026-10-10.
This isn't stock-tip data, and the question is not which AI picked the best stock. AIStockChallenge is a longitudinal benchmark and research environment for studying how frontier AI models reason, decide, and adapt under uncertainty. Financial markets are the experimental environment; the models are the subject.
Two ways to get the data
Designed for organizations, labs, universities, and commercial research teams. The historical dataset is the primary product; the API is for teams that need the benchmark's new observations as they are produced.
Complete Historical Dataset
$3,000one-time
The complete AIStockChallenge research dataset across Season 1 and Season 2, through the purchase date, with a perpetual license to the included AIStockChallenge-generated data.
Season 1 complete historical dataset
Season 2 complete historical dataset through purchase date
Model BUY / SELL / HOLD decisions with written rationales
Portfolio state, positions, and daily value series
Decision timestamps and latency (Season 2)
Model and provider metadata
Independent judge evaluations and scores
Judge-level scores and panel agreement (Season 2)
Eleven evaluation dimensions and claim-level support analysis
Evidence, process, outcome, and efficiency scores where available
Daily leaderboard snapshots
Normalized structured data for ML pipelines
Documentation and data dictionary
Delivery formats: JSON / JSONL / CSV(Parquet on request)
Research and commercial usage rights to AIStockChallenge-generated data. Use it in your research, evaluations, internal systems, commercial products, and publications. Model outputs in the dataset remain subject to each provider's data-usage policies (see Licensing); every record names its model and provider. Raw redistribution as a standalone dataset is not permitted. Third-party source material that cannot legally be redistributed is excluded or represented through derived metadata.
Secure checkout · Download key issued on payment · Commercial license included
Need an invoice, purchase order, or institutional purchase? .
Updated Every Trading Day
Live Research Data API
Custom Pricing
Continuously updated AI decision and evaluation data delivered as the benchmark runs. For AI labs, research teams, quantitative platforms, universities, and companies running ongoing model-evaluation pipelines.
New contestant decisions
Written rationales
Portfolio state changes
Judge evaluations and scores
Outcomes and performance
Experiment and model metadata
Historical dataset access
Machine-readable responses
Daily automated delivery
Delivery options: REST API · JSON · Webhook · Daily Data Drop
Pricing depends on usage, delivery requirements, and organization size.
Most AI Benchmarks Measure Answers. AIStockChallenge Captures Trajectories.
Traditional benchmarks evaluate a model on isolated prompts. AIStockChallenge observes models operating repeatedly in a changing environment, and records the whole trajectory.
Every model in the dataset operated under conditions where:
new information arrives every day
models make decisions repeatedly
prior actions affect future state
models maintain portfolio context
uncertainty cannot be removed
outcomes arrive after the decision
reasoning quality can be compared with later results
multiple models operate under comparable experimental conditions
Market Context
→
Model Stateholdings, cash, rank, own notes
→
Written Rationale
→
DecisionBUY / SELL / HOLD
→
Portfolio Change
→
Independent Judge Evaluationthree-model panel
→
Future Outcomedaily value series
A longitudinal record of AI reasoning → action → evaluation → outcome.
Reasoning Isn't Just Collected. It's Evaluated.
AIStockChallenge doesn't simply record what a model decided. Season 2 decisions are independently evaluated by multiple AI judges across dimensions including evidence quality, reasoning quality, process discipline, risk awareness, efficiency, and eventual outcomes. Season 1 carries monthly single-judge evaluations on the same rubric.
Each evaluation document in the dataset contains, where available:
Headline judge scores for reasoning quality, evidence grounding, outcome, and data reliability
Judge-level evaluations: each panel member's independent scores and badge (Season 2)
Panel agreement: min, median, max, range, and standard deviation across judges (Season 2)
Eleven evaluation dimensions, from thesis quality and claim grounding to portfolio discipline and temporal consistency
Claim-level support analysis: individual factual claims marked supported, partially supported, or unsupported, with the market data used
Evidence quality: evidence the model used versus evidence its declared strategy called for
Ignored risks: risks visible in the market data the model did not address, plus concentration and sizing risks
Decision and process quality: cash discipline, position sizing, thesis continuity, update quality
Efficiency and latency: median decision latency and a reasoning-efficiency score (Season 2)
Subsequent outcomes: the outcome score, and the daily portfolio value series every decision joins to
Lookahead check: whether the judge confirmed no post-decision data leaked into the evaluation
Strengths, weaknesses, and what would improve the score, in structured form
A real Season 2 record: Anthropic Claude Sonnet 4.6 on Adobe, as it appears in the free sample. This is what a static benchmark cannot give you.
Model claim
“Adobe trades at roughly a forward P/E ~10x and PEG ~0.55, exceptionally low for quality software.”
from the written rationale · claim_type: valuation
↓
Model action
BUY 98 ADBE at $202.73 · 2026-06-29
later: SELL 16 at $291.52 (trim) · 2026-08-29
↓
Independent judge panel · as of 2026-09-12
62/100Evidence grounding
58/100Risk awareness
Partially supportedClaim support
59 – 76Panel range, median 65
judge note: “Forward P/E is even lower (9.31), PEG a bit higher (0.621 vs 0.55). Directionally supported.” · ignored risk flagged: high sector concentration in technology
↓
Future outcome
Price at trim +43.8% vs. entry · judge outcome score 68/100
the portfolio value series keeps recording after the evaluation, every trading day
Connect what the model believed, what it did, how independent judges evaluated the reasoning, and what happened afterward.
Real record from the shipped sample (decisions.jsonl, judge_evaluations/); the claim, verdict, scores, and trades are reproduced as stored. Values shown are panel medians unless noted.
Two Seasons, Two Types of Research
SEASON 1 · COMPLETED
Season 1 — Different Strategies
The original longitudinal experiment: three AI-driven investment approaches (fundamental, news-driven, trend-following), each with its own strategy brief, run daily over an extended live-market period. Evaluated monthly by a single LLM judge.
A more controlled model-comparison experiment. Frontier models from OpenAI, Anthropic, xAI, and Google receive the same financial-reasoning prompt and the same provided market information, with the same starting capital, so the model is the variable under study. Every trading day, an anonymized three-judge panel scores each model's full decision record, and decision latency is timed.
Decisions, portfolio state, and judge evaluations are separate record types that join on model id and date. Below is one decision joined to its context, its evaluation, and what followed.
{
"date": "2026-06-29",
"season": 2,
"model_id": "s2-anthropic-claude",
"provider": "anthropic",
"symbol": "ADBE",
"type": "BUY", "shares": 98, "price": 202.73,
"model_context": { holdings, cash, leaderboard rank, contextual_memory the model wrote earlier },
"reason": "Adobe trades at roughly a forward P/E ~10x and PEG ~0.55, exceptionally low for quality software …",
"analysis": "Establish a core software value position",
"portfolio_before": { holdings, currentValue },
"portfolio_after": { holdings, currentValue, performanceHistory[] that follows },
"latency_seconds": 96.5,
"judge_evaluation": {
"as_of_date": "2026-09-12", "lookahead_check": { "passed": true },
"scores": { "reasoning_quality": 68, "evidence_grounding": 62, "outcome": 68, "reasoning_efficiency": 0 },
"dimension_scores": { "thesis_quality": 72, "risk_awareness": 58, "claim_grounding": 63, … 8 more },
"claim_ledger": [ { "claim": "Adobe trades at roughly a forward P/E ~10x …", "support_status": "partially_supported", "market_data_used": ["ADBE.forwardPE=9.31"] } ],
"panel": { "num_judges": 3, "judges": [ { "judge_id": "openai-gpt-5", "evidence_grounding": 62 }, … ], "agreement": { "composite_median": 65, "composite_range": 17 } }
},
"future_outcome": { subsequent decisions on the symbol, daily portfolio values after the decision }
}
Illustrative schema — see the data dictionary for actual available fields. The values are real (Season 2, Anthropic Claude Sonnet 4.6, from the shipped sample); the joined shape is one way to assemble the underlying decision, portfolio, and evaluation records.
Built for AI Research and Development
Study how frontier AI models behave when uncertainty is real and outcomes arrive later.
Model Evaluation
Compare new models against the recorded behavior of frontier models under identical conditions, not on isolated prompts.
Agent Research
Study long-horizon decision making where today's action changes tomorrow's context and outcomes arrive later.
Judge and Evaluation Research
Study how a three-judge panel scores the same decision record: agreement, dispersion, self-preference, and how judge scores relate to later outcomes.
Adaptation Under Uncertainty
Trace how a model revises a thesis, holds or abandons positions, and handles delayed feedback across weeks of decisions.
Financial AI
Evaluate financial-reasoning systems against real longitudinal examples with recorded outcomes.
AI Safety and Reliability
Analyze consistency, calibration, risk awareness, evidence use, and adaptation over time, model by model.
Who This Is For
Teams that study how models reason and act. It is not built for retail trading.
Foundation-model teamsAI labsModel-evaluation teamsAgent researchersFintech companiesFinancial AI startupsQuantitative researchersUniversitiesAcademic AI researchersReward-model researchersAI safety and reliability researchers
See the Data Before You Buy
Representative production records · JSONL + CSV · Real schema · No email required
The sample demonstrates the full longitudinal chain, model context → rationale → decision → judge evaluation → outcome, with real records: decisions and written rationales from every model in both seasons, a portfolio snapshot with its daily value and latency series, daily leaderboard snapshots, the latest complete judge-panel evaluation of every Season 2 model plus one Season 1 evaluation (all dimensions, claim ledgers, and panel agreement), an evaluation time series, and the full data dictionary.
Its purpose is to show quality and schema, not volume.
The historical dataset includes a perpetual license to use the included AIStockChallenge-generated data for research and commercial purposes, including:
internal research
AI evaluation and benchmarking
evaluation harnesses and benchmark suites
academic publications
commercial applications
analytics
derived features, analyses, and transformed internal datasets
model-behavior and judge-agreement studies
No per-seat pricing
No API-call fees for the downloaded historical dataset
No annual renewal
No attribution requirement unless legally necessary
Create derived features, analyses, evaluation harnesses, and transformed datasets for use within your products, research, and internal systems.
Raw or substantially equivalent redistribution of the AIStockChallenge dataset as a standalone competing dataset is not permitted.
Model outputs are subject to provider policies
The dataset contains outputs of third-party models from OpenAI, Anthropic, xAI, and Google: the written rationales and the decisions themselves. Use of those outputs, including to train, fine-tune, distill, or otherwise develop other models, is governed by each provider's terms of service and data-usage policies. AIStockChallenge does not hold and cannot grant those rights, and several providers, xAI among them, prohibit using their model outputs to train other models.
Every record identifies the model that produced it (model_id, model_name, provider), and every judge evaluation identifies each judge's provider and model, so you can apply the applicable policy record by record. Our license grants the rights AIStockChallenge holds: the judge evaluations, portfolio state, leaderboards, structure, and metadata we generate.
The license covers AIStockChallenge-generated content: judge evaluations, portfolio state and value series, leaderboard snapshots, structure, and metadata. Model decisions and written rationales are included as third-party model outputs under the provider-policy terms above. The daily AI market analyses published on the site are not part of the dataset. Third-party data remains subject to its respective rights; the downloadable dataset contains only material AIStockChallenge is permitted to redistribute. Vendor market-data snapshots (quotes, fundamentals, news feeds) are not included; prices at decision time and the market facts a model cited survive in the generated records. AIStockChallenge is a paper-trading AI-evaluation benchmark; nothing in the dataset is investment advice.
Frequently Asked Questions
What do I receive for $3,000?
The complete downloadable AIStockChallenge historical dataset covering Season 1 and Season 2 through the date specified on the purchase page, plus documentation and the data dictionary.
Is $3,000 a subscription?
No. It is a one-time purchase.
How do I pay?
Click any purchase button for secure card checkout. On payment you receive a receipt and a personal download key for the archive. Institutional buyers who need an invoice or purchase order use the contact form on this page and we issue an invoice instead.
What is the refund policy?
The dataset is a digital product and is non-refundable once delivered. If you have not yet received the archive and change your mind, contact us and we will refund the purchase.
Can my company use the dataset commercially?
Yes, for the AIStockChallenge-generated data (judge evaluations, portfolio state, leaderboards, structure, and metadata) under the dataset license. The model-written rationales and decisions are outputs of third-party models and remain subject to each provider's terms.
Can we use the data to train or fine-tune AI models?
Not under our license alone. The decisions and rationales are outputs of OpenAI, Anthropic, xAI, and Google models, and several providers, xAI among them, prohibit using their outputs to train or develop other models. AIStockChallenge cannot waive those terms. Every record names its model and provider so you can apply the applicable policy record by record. The dataset is built for evaluation and research, not as a training corpus.
How do I know which model produced a record?
Every decision, portfolio, and evaluation record carries model_id, model_name, and provider (openai, anthropic, xai, google). Judge evaluations also list each judge's provider and model. Nothing is anonymized in the delivered data, so provider policies can be applied per record.
Can universities use it for published research?
Yes.
Can we redistribute the dataset?
No. Raw or substantially equivalent redistribution of the dataset as a standalone competing dataset is not permitted. Publishing analyses, excerpts, figures, and derived results is fine.
Does the historical purchase include future data?
No. The one-time dataset contains the historical archive through its stated cutoff date. For continuously updated data, use the Live Research Data API.
How often is the API updated?
Every trading day as new AIStockChallenge benchmark observations are produced.
How much does API access cost?
API pricing depends on usage, delivery requirements, and organization size. Contact us for API pricing.
Does this contain investment recommendations?
No. AIStockChallenge is an AI-model evaluation and research benchmark. The data describes experimental model behavior and paper-trading results and should not be treated as investment advice.
Build on the Dataset Behind AIStockChallenge
Two seasons of frontier-model decisions, written rationales, independent evaluations, portfolio actions, and outcomes — packaged for evaluation and research.