Rudimint post-trained Nemotron to score 42.1 on ExtractBench Pro, up from 31.2 and ahead of GPT-6 Luna’s 31.7.
In the large majority of ExtractBench Pro documents, most answers must be worked out from rules, tables and drawings. In our work, the model does this in one call per document, with no tools or agent loop.
Rudimint built 23 synthetic business worlds where code and SQL deterministically create each document and answer key. It trained on 664 worked solutions and on no ExtractBench Pro documents.
Rudimint post-trained Nemotron
42.1
GPT-6 Luna
31.7
Nemotron, before training
31.2
ExtractBench Pro value F1 across all 100 documents, using one model call per document.
ExtractBench Pro tests more than just parsing: its 100 documents ask models to calculate amounts, apply rules, follow diagrams, and reason across pages. Consider an insurance packet with forty claims, a policy, and a letter near the end that changes an earlier deductible. The model must match each claim to its policy, decide which version applies, calculate the payment, and return every row. Reading every word on the page correctly does not guarantee any of those steps, and one wrong interpretation can change dozens of answers.
This is difficult even for capable models. In our initial single-call evaluations, open-weight Nemotron and GPT-6 Luna scored almost the same. We wanted to improve that single-call setup, teaching the model to do the work a document requires without adding tools at inference time.
Of the 68.8 points Nemotron lost, misapplying rules and other wrong computed values cost about 35.4, giving up on values that had to be calculated at least 15.8, and stopping before every row was written 13.2.
We used ExtractBench Pro to understand where Nemotron failed, then built synthetic documents that exercised those skills. ExtractBench Pro documents and answers were not included in the training set.
We sorted Nemotron’s failures into a taxonomy of five types: giving up on computed values, misapplying rules, stopping before every row, failing to trace a drawing, and mapping another party’s terms to house categories. Worlds were chosen from a grid of those failures and the rule kinds ExtractBench Pro uses, and a coverage table shows which mechanics are thin. Many of the rule mechanics started from real contracts and policies: we mined patterns from commercial leases, credit agreements, health plans and labor rules, each tied to the failure it trains.
Each synthetic world is an invented business: a grain co-op, a ferry operator, a hospital payroll office, or a lease auditor. We build its records in DuckDB, an embedded SQL database. Code samples the people, transactions, dates, rates, and relationships; SQL queries apply the business rules and compute the answers. Agents write each world’s code and wording on a shared kit, and a fresh reader and an automatic audit check the result; code, not a model, supplies every value.
The same records and rules become the paperwork: tables, policy text, amendment notices, charts, and drawings. A rule may appear several pages after the table it changes. Superseded figures remain visible. Related records may be split across pages. The model sees the document and requested answer schema, not the database or SQL.
Each document comes from a seed, so it can be rebuilt, and every value lives in one place, so a table and a rule cannot disagree.
A synthetic group ownership chart: one sheet of several.A synthetic well log, from a section about 13,000 pixels wide.
Learning from real world documents
To make the documents look like real work, we studied real ones. Agents harvested 632 public PDFs on the web across families such as planning, energy, finance and construction, aimed at real paperwork with rules and pictures, not brochures or templates.
To sort them we used Clef, Cloudflare’s Jev-like decision model. Clef answered 27 questions about each page: what kind of page it is, whether it is filled in, legible and sensitive. SigLIP 2 page embeddings found near-copies of the same template. Of 889 documents, 642 stayed, after dropping blank forms, brochures and near-copies.
We use this pool for structure, and it shaped how documents open, where rules sit, which contents pages and signature blocks they carry, and 100 house styles, each modeled on a real document.
Scans, handwriting and pictures
Most ExtractBench Pro pages are not clean PDFs: 60 documents are scans, and 22 carry hidden text layers that disagree with the page. Ours follow a similar mix of digital, scanned and image-only pages. A scan gets faded toner, uneven paper, skew, blur, grain and JPEG artefacts. Tesseract then reads the damaged page to make the hidden text, so its OCR errors are real and the image stays the authority.
Some worlds add handwritten notes, corrected figures, strike-throughs and rubber stamps. Digits stay clear: the skill is noticing a mark and applying it. Charts and drawings are images with no text layer, and every value on them can be read from a label or gridline.
Before: a synthetic page as generated.After: the same page scanned.
On the page
In the hidden text, from OCR
$694.87
$694:87
$7,399.70
$7,399:70
Handwriting and a stamp on a synthetic payment application.
Computed answer keys
That gives us a document to read and a computed answer key from the same underlying case.
We also encode plausible mistakes: using an old rate, copying a decoy total, or joining tables by row position instead of identity. The generator checks whether these wrong paths change the answer and plants cases around thresholds. This makes the difficulty deliberate, rather than simply adding more pages.
Every printed rule is tested for whether it changes an answer. The key must score 100 on our grader, an empty answer about 0, and every planted wrong path less than 100. Documents must be hard for Nemotron but still solvable.
Code can still be wrong, and a correct query can disagree with a poorly worded rule. Read-throughs caught ambiguous rounding instructions, cramped drawings, and faulty joins. We corrected those in the generator so the fix applied to future documents too.
Kimi K3 was the teacher: it solved the synthetic documents and showed its working. The selected solutions matched their answer keys exactly. Near misses were reviewed against the document and corrected where appropriate; a matching final answer alone does not establish that every intermediate step is sound.
Solutions stayed balanced across each world’s difficulty, document kinds and answer shapes.
Some chart reads needed extra help, so the teacher received chart values at the precision readable from the page. The student received only the document, without that aid.
Nemotron was then trained with supervised fine-tuning on Tinker. Each example contained the page images, text layer, and schema, followed by the teacher’s working and JSON answer.
Across all 100 documents, Nemotron rose from 31.2 to 42.1, a gain of about 11 points. Value F1 compares each answer value with the key, balancing values missed against values wrongly given, and every document counts equally.
Across documents, the 95% interval on the gain is 7.9 to 14.4. GPT-6 Luna scored 31.7 in one call at about $0.013 per document; our trained run cost about $0.12 per document on Tinker, and training cost about $121. On 30 synthetic documents from worlds held out of training, the score rose from 66.1 to 73.8.
Document conventions (22)
48.2 → 71.4
House vocabulary (14)
48.3 → 62.8
Charts in documents (25)
26.7 → 35.5
Drawn relations (20)
25.3 → 28.7
House counting rules (12)
12.4 → 16.9
Long chart sheets (7)
8.7 → 14.4
Mean value F1 by category, with the number of documents in brackets. The grey bar is before training and the coloured extension is the gain. Every category rose, but charts, drawn relations and counting stay low.
Shipment
Contract rate
Before
After
Key
BW-25-0102
9.00%
9.0
7.0
7.0
BW-25-0107
17.00%
12.0
7.0
7.0
Clause A.5 from the last page of a scanned 49-page ExtractBench Pro document, and two of its 419 shipments. Insured interest rate correct: 90, 5 and 206 of 419 on three reads before, 416 and 380 on two reads after. The earlier reads often stopped short of the full list. These rows are from one read of each model.
Building the business data and rules first let us create difficult examples with checkable answers; worked solutions supplied the steps between the page and those answers.
Next, we will add reinforcement learning on top of this model, so it can improve directly against the answer keys. We will also extend the synthetic worlds toward what it still finds hardest: charts, drawn relationships, counting rules and very long documents. Then we will test whether the method transfers to customer workflows.
