Why does determinism matter for financial AI specifically?

Because financial regulators are starting to ask for it directly. The EU AI Act's high-risk system requirements call out reproducibility as a compliance baseline, the European Banking Authority's machine learning guidelines require documented, reproducible model behavior, and UK FCA guidance emphasizes that algorithmic decisions must be reconstructible on challenge. A model that gives a different answer to the identical question on different days is a real audit and governance problem, independent of whether the answer is correct.

Does this mean Claude's financial calculations are always correct?

No — this study measured consistency (does the same prompt produce the same answer every time), not accuracy (is the answer correct) in isolation. In our test, 2 of 3 prompts matched the reference ground truth exactly; the third was internally consistent but landed on a slightly different, more precise value than the reference dataset's own rounded intermediate figure. Consistency and correctness are separate properties, and both need to be tested.

How was this study conducted?

Three prompts (DCF enterprise value, WACC, LBO equity returns) were drawn from this site's existing ground-truth eval dataset. Each was sent to Claude Sonnet 5 via the claude CLI three separate times, as three independent process invocations, not a batch or shared context. The final numeric answer from each run was extracted and compared byte-for-byte.

Is Claude Deterministic? A Financial-Calculation Reproducibility Study

Q: Is Claude deterministic?

Not by design — Anthropic, like other model providers, documents temperature and sampling controls as 'best effort' for reproducibility, not a guarantee. In our own small pilot test (3 financial calculations, 3 repeats each via the claude CLI), every repeat produced the identical final numeric answer to the digit. That is a real, measured result for this narrow test set — not a claim that Claude is deterministic in general, across all prompt types, temperatures, or at scale.

We ran 3 financial calculations (DCF, WACC, LBO) through Claude 3 times each and compared every digit. Real methodology, real data, and an honest look at where the numbers didn't perfectly match our own reference dataset.

Educational content, not professional advice — AI output and figures here can be wrong. Verify before you rely on it. Full disclaimer →

Anthropic doesn't market Claude as deterministic, and neither do we. But "how deterministic is it, actually, for the kind of structured financial calculation this site is built around" is a testable question — so we tested it, on a small scale, and are publishing exactly what we found, including the parts that didn't perfectly line up.

Why This Question Matters Right Now

Reproducibility has moved from a nice-to-have to a stated regulatory expectation for AI used in financial decisions. The EU AI Act's high-risk system requirements name reproducibility as a compliance baseline. The European Banking Authority's machine learning guidance requires documented, reproducible model behavior. UK FCA guidance emphasizes that algorithmic outputs need to be reconstructible on challenge — if an examiner asks "run this again," a firm needs a real answer, not "it depends." At the same time, model providers are honest that reproducibility controls (temperature, seed, system fingerprint) are documented as best-effort, not guaranteed — non-determinism has been observed even at temperature zero. That's the real, current tension this study sits inside.

Methodology

We took three prompts directly from this site's own ground-truth eval dataset (the same one behind our 462-eval benchmark) — each with a known, analytically-computed correct answer:

DCF enterprise value — 5-year FCF projection, terminal value, WACC discounting. Reference answer: EV = $166.09M.
WACC calculation — CAPM cost of equity, after-tax cost of debt, capital-structure weighting. Reference answer: WACC = 10.50%.
LBO equity returns — 8x entry multiple, 60% debt, 5-year hold, 7x exit. Reference answer: MoM = 1.435x, IRR ≈ 7.5%.

Each prompt was sent to Claude Sonnet 5 via the claude CLI three separate times — three independent process invocations, each starting fresh with no shared context or conversation history. We compared the final extracted numeric answer across the three runs for each prompt, digit for digit.

Results

DCF (VAL-001): All 3 runs returned 166.09 — identical to each other and to the reference value.

WACC (VAL-002): All 3 runs returned 10.50% — identical to each other and to the reference value.

LBO (VAL-003): All 3 runs returned MoM = 1.427x, IRR = 7.38% — identical to each other, but not to the reference dataset's 1.435x / 7.5%.

The LBO gap is the interesting part, and we ran it down rather than rounding it away. Claude's exit EBITDA calculation was $50M × 1.06⁵ = $66.91M, computed to full floating-point precision — carried through correctly to exit equity ($228.38M) and MoM (1.427x). The reference dataset's own exit EBITDA figure is a rounded $67.0M. Recomputing from Claude's own unrounded intermediate figure reproduces its exact reported answer: ((7 × 66.911 − 240) / 160)^(1/5) − 1 = 7.376%. In other words: the model wasn't wrong or inconsistent — it computed to more decimal places than the reference dataset's intermediate value carried, and that precision difference is the entire gap. Both figures are well inside this eval's own 5% tolerance band, so it would pass either way — but "passes tolerance" and "matches exactly" are different claims, and we'd rather show you the real number than round it into looking cleaner than it is.

What This Does and Doesn't Show

What it shows: for this narrow set of structured, well-specified financial calculations, sent as single-shot prompts via the CLI, Claude produced byte-identical final answers across every repeat — 9 for 9. That's a real, measured data point in favor of practical reproducibility for this exact use case.

What it doesn't show: this is n=3 prompts × 3 repeats — a pilot, not a comprehensive benchmark. It didn't test: different temperature settings, the Claude API directly (as opposed to the CLI), longer or more ambiguous prompts, multi-turn conversations, other Claude models, or other providers. A single small study finding perfect consistency doesn't override the documented reality that non-determinism exists in LLM inference generally — it says something specific about this narrow, well-structured task category, not about LLMs in general.

Where We're Taking This Next

This is the first entry in what we intend as an ongoing series, not a one-off. Planned next steps: scale up the repeat count per prompt for statistical confidence, extend across the full 66-test eval suite rather than 3 prompts, test at the API level with explicit temperature/seed controls (not just the CLI default), and compare across models where API access allows. Results will be published here as they're run — including if a future batch finds real inconsistency, which is exactly the kind of result worth publishing, not suppressing.

Methodology note: raw output from all 9 runs is available on request — see the eval methodology page for how to request the underlying data and scorer code for this and the full eval suite.

Get free access

Is Claude Deterministic? A Financial-Calculation Reproducibility Study

Why This Question Matters Right Now

Methodology

Results

What This Does and Doesn't Show

Where We're Taking This Next

Related