FinanceGym
Point-in-time financial deep research benchmark
FinanceGym gives an agent a finance research question and a cutoff date. The agent may only retrieve documents published on or before that date, then writes a cited report. Reports are graded against expert rubrics that separate what was knowable at the cutoff from what actually happened afterwards. How the questions and rubrics are generated and validated is explained in the paper.
Overview
1. Retrieve, point-in-time
Every retrieval call is bounded by the task's cutoff date, so no document published after it can enter the evidence set. Enforcing that at the source is the benchmark's one hard rule.
2. Write a cited report
Any agent architecture is allowed — any loop, any planner, any backbone model. The single hard rule is point-in-time compliance on retrieval.
3. Graded on two axes
Pre-cutoff scores antecedent items against evidence available at the cutoff. Post-cutoff scores consequent items against what happened after. Rubrics stay withheld to prevent rubric hacking.
Leaderboard
Normalized rubric means (%) over the 400-question set. Pre-cutoff = grounding in evidence available at the cutoff, Post-cutoff = reasoning about what followed, ±SE = standard error of the Overall mean. System link points at the paper or repository for the agent framework the row runs on. Rank always reflects the overall standing, whatever column you sort by.
| # |
|---|
Post-cutoff scores sit far below pre-cutoff for every system — anticipating consequences from pre-cutoff evidence alone is the open problem this benchmark exists to measure.
Cost–quality frontier
Overall rubric score against estimated cost per query. The x-axis runs high → low left to right, so up and to the right is better, and the staircase marks the efficient set — nothing cheaper scores higher. Hover or focus any point for its numbers.
Data
benchmark_400_public.jsonl holds 400 tasks, one JSON object per line. Cutoffs
span 2024-12-31 to 2025-11-10 across 31 distinct dates.
| Field | Type | Meaning |
|---|---|---|
task_id | string | Stable unique id for the question. Handy for your own bookkeeping; not used for matching. |
question | string | The research question. Used verbatim to match your submission to its withheld rubric — copy it exactly, with no truncation, re-wording, or added point-in-time framing. |
cutoff | string YYYY-MM-DD |
The point-in-time date. Only information available on or before this date may inform the report. |
Rubrics are not part of the public file — this prevents score hacking. The tab shows an example rubric. Grading runs against a private full-rubric set.
The first line of benchmark_400_public.jsonl, pretty-printed. Every line of
the public file carries exactly these three fields — this is the whole record.
{
"task_id": "69f904b728538874c086db16",
"question": "How does Air France-KLM’s accelerated transition toward a 60.5% majority stake in SAS impact its strategic bandwidth and capital allocation for the privatization of TAP Air Portugal, given the group's 2026 full-control target and ongoing European airline consolidation?",
"cutoff": "2025-08-05"
}Rubric example
{
"question": "Why did AAPL revenue decelerate in Q1 2025?",
"cutoff": "2025-03-31",
"topic": "earnings",
"sector": "technology",
"reasoning_type": "causal",
"rubric": [
{
"category": "pre_cutoff",
"criterion": "Identify the Q1 2025 revenue figure for AAPL and the YoY change."
},
{
"category": "pre_cutoff",
"criterion": "Mention macroeconomic headwinds (USD strength, China demand) cited by management."
},
{
"category": "post_cutoff",
"criterion": "Note the Q2 guidance issued by AAPL after Q1 results."
}
]
}Submission
Five steps to run the benchmark and submit to us for grading.
-
Download the questions
-
Bring a point-in-time retrieval setup
The retrieval environment used for the published results is not distributed due to FinanceGym's CC BY-NC 4.0 license.
How you replicate the PIT search environment is up to you. Two examples:
-
Build it from our open-source code.
FinanceGym/scripts↗ takes a web corpus you supply and runs the whole pipeline — extract, embed, build the index, serve. -
Use a public search tool — such as Serper.dev or
DuckDuckGo — and add your own post-check that drops
any result published after the cutoff. We have implemented both
backends for reference in
financeharness/tools/research↗.
Whichever route you take, one hard rule applies: retrieval must return only documents published on or before the task's cutoff.
-
Build it from our open-source code.
-
Run your agent — point-in-time, no leakage
Any agent architecture is allowed: any loop, any planner, any backbone model. There is exactly one hard rule, and it is about retrieval.
The hard rule:
max_dateis mandatoryEvery single
/searchcall must carrymax_date= that task'scutoff. The server then never returns a document published after the cutoff. Omitting it — even on one exploratory query — voids point-in-time compliance for the whole run.A note on the model's own memory
A backbone model trained after the cutoff may simply know what happened next, and we cannot audit that. It is good practice to require, in your system instruction, that the report should stand on retrieved, cited evidence — so the agent asserts only what its point-in-time sources support, and cites each claim.
-
Format your
answers.jsonlA single JSONL file: one object per line, one line per benchmark question — the exact shape our grader reads. 400 lines for a full run; partial runs are accepted but are marked as such on the leaderboard.
Fields our grader reads from each line of answers.jsonl.Field Type Required Description questionstring required The benchmark question verbatim. This is the key that matches your answer to its withheld rubric — any edit, truncation, or added framing breaks the match and the line is dropped. cutoffstring YYYY-MM-DDrequired The task's cutoff, copied from the question file. reportstring required Your agent's full research report, with citations. Markdown is fine; it is graded as text. Do not wrap it in extra JSON. num_docs_retrievedint optional Number of documents your agent fetched. elapsed_sfloat optional Wall-clock seconds for the task. Example line
{"question": "How does Air France-KLM's accelerated transition ...", "cutoff": "2025-08-05", "report": "**Equity Research Note**\n\n...", "num_docs_retrieved": 16, "elapsed_s": 62.2} -
Send it to us
Email your
answers.jsonlplus the metadata block to rujunh@google.com with the subject line:FinanceGym submission: <agent-name>Metadata to include with your submission
repois what we publish as your System link on the leaderboard — a public repository or a paper URL.agent: my-research-agent org: Example Lab repo: https://github.com/your-org/your-agent # public repo or paper URL base_model: your-model # the backbone your agent used date: 2026-08-05 contact: you@example.com notes: agent loop, 8 workers, PIT-compliant (max_date enforced on every search).
What happens after you submit
We validate the file shape, run the judge over your reports, and add your row to the leaderboard, linked to your system's paper or repository. If anything about the submission is malformed or looks non-compliant, we come back to you before publishing a score.
License
Both FinanceGym and FinanceHarness — the benchmark, its datasets, the retrieval tooling, and the harness code — are released under Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) ↗.
Non-commercial use only
Everything here is provided strictly for academic and non-commercial research and evaluation. Commercial use — including integration into commercial financial advisory services, or using generated outputs to provide commercial investment advice — is prohibited.