How LLMs fail at SEC 10-Q balance sheet extraction

Results at a glance

Each model extracted the period date, the current-asset and current-liability lines and four totals (total current assets, total assets, total current liabilities, total liabilities and equity). The contract was tuned on 60 companies, then frozen and run on 60 others. Rejected answers got one retry with every violation in the prompt.

Every rejection was a real mistake, read by hand, and a plain JSON Schema would have accepted all of them. Of 452 real answers with ground truth, a JSON Schema accepted 3 that were wrong against the filing (DRI, KDP, CTVA). The frozen contract accepted 1 (CTVA, held out); a check added afterwards rejects it.

On synthetic corruptions (50 accepted answers per run, one seeded defect each), naive parsing caught 4 of 50, a JSON Schema 20 of 50, and the full contract 50 of 50 in all 8 runs. Those defects were written knowing what the gates check, so they show that each gate fires on its target, not how often models make each error.

Retries that rewrite the totals (TSLA, ADP, EXC)

A common pattern for structured output is to validate, send the errors back, and ask again. When the error is "the lines don't add up to the total", the cheapest fix for the model is to change the total, not to find the missing line. That is what happened.

TSLA

qwen2.5:14b added two non-current items, printed in blocks below the table, to Tesla's current assets. Its lines summed to 78,902 while the page prints a total of 70,402. Told the sum didn't match, the retry left the lines alone and changed the total to 78,902. Schema and arithmetic now pass. The contract still rejected it:

[provenance] value 78902 is not a number its quote prints (it prints 70,402)

ADP

qwen left out "funds held for clients" (46,413.1) and quoted several figures from the prior-year column. On retry it changed the totals to match its own sums:

[provenance] value 7821.0 is not a number its quote prints (it prints 12,266.5)

EXC

For Exelon, qwen left out lines (they sum to 8,507; the page prints 9,797) and quoted a number that is not on the page. The retry changed both totals to match its own sums: total current assets 8,507 against 9,797 in XBRL, total current liabilities 5,524 against 9,008.

[provenance] source_span.text "1,827" not found in the source document (whitespace and currency signs ignored)
[provenance] value 8507 is not a number its quote prints (it prints 9,797)

Every amount multiplied by 1,000 (DRI)

Darden's statement prints figures in millions with one decimal, such as "13,226.6". qwen returned 13,226,600 for total assets, 953,100 for total current assets and 3,199,000 for total current liabilities, each with the printed text quoted correctly.

This passed the first contract. Every quote was on the page, and every sum still held because every amount was scaled by the same factor. A JSON Schema accepts it too: they are all valid numbers. It is the kind of error that flows straight into a database.

The fix was numbersMatchQuotes(): a numeric value must be a number its own quote prints. The scaled answer is now rejected. The retry scaled again and was rejected again:

[provenance] value 220500 is not a number its quote prints (it prints 220.5)
[provenance] value 118000 is not a number its quote prints (it prints 118.0)

This check was added after seeing DRI, so the in-sample set is not an independent test of it. The held-out set is: there, gemini-3.8-flash dropped a digit from KDP's total assets (8,761,900,000 for "87,619,000,000"), a plain schema accepted it, and the frozen contract rejected it.

A total from the wrong statement (CTVA)

Corteva's 10-Q prints two balance sheets one after the other: Corteva's, then its subsidiary EIDP's, with almost the same lines. qwen took Corteva's current assets and EIDP's current liabilities, giving total current liabilities of 10,795 where Corteva's is 10,789.

Every quote was on the page, every number matched its quote, and every sum held, so the frozen contract accepted it. This is the one wrong answer that got through on the held-out set. Grounding checks that a value is in the source, not that it comes from the right part of it.

The fix was ProvenanceGate.within(Region, fields...), which requires the date and the four totals to be quoted from the first statement on the page, from the top through its first "liabilities and equity" total. qwen's 10,795 is printed only in EIDP's statement, so CTVA is now rejected. Re-scoring every recorded answer of all four models in both sets changes exactly one verdict, this one, and every run still catches 50 of 50 corruptions. It was written after seeing CTVA, so it is a fix, not a held-out result.

The benchmark case viewer shows the CTVA answer beside the page, with each quote highlighted.

Other errors: left-out lines, subtotals, invented quotes

The pattern: models struggle most with layout, not reading. Lines printed outside the main table get dropped or wrongly added, and subtotals get listed beside the lines they sum. Arithmetic across fields catches those; quotes checked against the page catch the rest.

The contract that caught them

This is the balance-sheet contract from the repository's examples. Swap the field paths and it applies to any structured output with parts that must add up and values that must come from a source.

Contract contract = Contract.builder()
        .gate(TypeGate.of(BalanceSheet.class))
        .gate(MathGate.builder()
                .sumEqualsRounded("current_assets[*].amount", "total_current_assets")
                .sumEqualsRounded("current_liabilities[*].amount", "total_current_liabilities")
                .custom("total_assets = total_liabilities_and_equity", n ->
                        n.get("total_assets").compareTo(n.get("total_liabilities_and_equity")) == 0)
                .tolerance(1)
                .build())
        .gate(ProvenanceGate.builder()
                .require("period_end", "current_assets[*].amount", "total_current_assets", "total_assets",
                        "current_liabilities[*].amount", "total_current_liabilities",
                        "total_liabilities_and_equity")
                .groundedIgnoringWhitespace()
                .ignoringCurrencySigns()
                .numbersMatchQuotes()
                .within(FIRST_STATEMENT, "period_end", "total_current_assets", "total_assets",
                        "total_current_liabilities", "total_liabilities_and_equity")
                .build())
        .gate(RegexGate.builder().isoDate("period_end").build())
        .build();

Verdict verdict = contract.evaluate(output, EvalContext.empty().withSource(document));

An output either passes every gate or is rejected with every violation listed, so the same contract can fail a JUnit build, score a golden set, or decide at runtime whether to accept an answer, retry it, or send it to a person. Install from Maven Central as io.github.ravismash:outputgate-core:0.1.0; see the README.

What these numbers don't show

• Three gates (numbersMatchQuotes(), sumEqualsRounded(), ignoringCurrencySigns()) were added after the in-sample run exposed gaps, so the in-sample set is not an independent test of them. The held-out tables are the frozen contract's.
• Ground truth covers four totals per filing, and XBRL values were available for 58 of 60 in-sample and 55 of 60 held-out filings.
• The cost of a strict contract is the answers it turns away. Each rejection is a document a pipeline would route to a person.
• outputgate checks what can be written as an invariant: shape, arithmetic, provenance, patterns and budgets. It does not judge tone or relevance.