How the benchmark actually works
This page is the reference for the ClaudeFinanceLab finance AI benchmark: what is tested, how ground truth is computed, how pass and fail are decided, and what the results do and do not prove. It is written to be citable — if you are documenting an AI model for review, you can link this URL and the live results directly.
The three tracks
asyncio.wait_for() with a 30-second timeout. Pass/fail is deterministic.
tools/list within 5 s),
discovery (schema completeness, valid JSON Schema), execution (first call returns content within
15 s), and resilience (bad arguments return a structured error). This is the suite behind the
certification badge.
How ground truth is computed
Track 1 never checks one model's answer against another model's answer. Every reference value comes from a closed-form formula or a published table:
| Domain | Ground-truth basis |
|---|---|
| Options pricing | Black–Scholes closed form; Greeks by analytic derivative |
| Cost of capital / beta | Hamada equation for relevering; CAPM for cost of equity |
| DCF & valuation | Terminal value = FCF·(1+g)/(WACC−g), discounted analytically; verified to 4 significant figures |
| Depreciation | MACRS schedules from IRS tables |
| Lease accounting | ASC 842 present-value calculation |
| Portfolio risk | Parametric and historical VaR from the return series |
Tests that must call a live external API (EDGAR filings, market-data feeds) are explicitly labelled
live. Everything else runs on locally computed ground truth and reproduces
with no network access.
Pass, fail, and tolerance
A Track 1 test passes only if the tool output is within tolerance of the reference value. Tolerance is per-test and documented in the scorer: exact-match for integer or categorical outputs (share counts, schedule rows, sanctions hits); a relative band — typically 0.01% — for continuous financial quantities. Any deviation beyond the band is a fail, with the expected and actual values recorded. There is no partial credit and no manual override.
When the benchmark re-runs
The full Track 1 suite runs on every deployment; the current results page
always reflects the deployed code. Track 2 re-runs weekly against every directory server so a
third-party regression shows up within seven days. The framework lives at
evals/runner.py; datasets and scorer code are available on request at
evals@claudefinancelab.com.
What the results prove — and what they do not
A green Track 1 board means the tool functions compute the covered calculations correctly, within documented tolerance, reproducibly. It does not prove that a language model calling those tools will select the right tool, pass the right arguments, or interpret the result correctly — that is Track 3's job, and Track 3 is judged, not proven. It also does not cover calculations outside the 66 tests. For a written evaluation of your model or workflow, scoped and mapped to the regime you answer to, see Enterprise validation.
Citing this in a governance file
The published methodology — deterministic tests, analytically-computed ground truth, documented pass criteria, reproducible results — is model-validation evidence under the SR 11-7 framework and maps to the traceability expectations in the EU AI Act. Institutions can reference the public results and this methodology in an AI model governance file. For a report formatted specifically for SR 11-7, SR 26-2, or EU AI Act review, contact us.
How consistent is this in practice, run to run? See our reproducibility pilot study — real data from repeating the same prompts multiple times.