Multimodal Document AI: Build an Accuracy Test Set

A document AI demo can look convincing while failing on the exact pages that matter to a business: rotated scans, faint totals, stamps over text, handwritten corrections, long tables, or a language missing from the demo. A useful evaluation begins with the documents the workflow actually receives and scores the fields or decisions that have real consequences.

Why this decision matters

Generic benchmarks rarely match a company’s forms, image quality, regional formats, or tolerance for error. Extracting a supplier name is not equivalent to extracting a bank account or tax total. The same character mistake can be harmless in a note and costly in a payment instruction. A local test set lets the team compare models, prompts, OCR stages, and human-review rules against one stable target. It also exposes whether improvements for clean PDFs make photographed receipts worse.

A practical workflow

1. Define the unit of success. List required fields, tables, classifications, and page-level decisions. Assign stricter thresholds to identifiers, amounts, dates, and account details than to descriptive text.
2. Sample the real distribution. Include digital PDFs, phone photos, fax-like scans, different templates, languages, multi-page files, small fonts, skew, shadows, handwriting, and deliberately blank fields.
3. Create a trusted answer key. Have trained reviewers label exact values and permitted alternatives. Resolve disagreements and record uncertainty instead of forcing a false ground truth.
4. Score structure and meaning separately. Measure exact match, normalized match, numeric tolerance, table row alignment, missing fields, hallucinated fields, and document-level acceptance.
5. Design the review threshold. Route uncertain or high-risk cases to a person. Test whether the confidence or validation rule actually separates correct from incorrect results.

Work through a realistic example

An accounts-payable team tests 300 invoices across its largest suppliers and a long tail of rare formats. For invoice number, supplier tax ID, bank details, currency, subtotal, tax, and total, it stores a verified answer. It also labels duplicate invoices and pages that are not invoices. A model receives no credit for a total that is numerically correct but assigned to the wrong currency. The team reports accuracy by supplier template and scan quality, then sends every changed bank account and any unbalanced total to manual review.

What to measure and record

Field precision and recall are more useful than a single document score. Report missing values separately from invented ones. For tables, measure row and column alignment, not only whether words appear somewhere in the output. Add straight-through-processing rate, manual-review rate, average correction time, and cost per accepted document. Track severe error rate for fields that could move money or change legal records. Keep an untouched holdout set so prompt tuning does not gradually turn the benchmark into a memorization exercise.

Common traps

• Only clean sample files: A test set built from vendor demos will overstate real performance.
• Treating blank as failure: The correct answer may be an absent field. Models should not invent a value to satisfy a schema.
• Hiding normalization: Date, decimal, and currency normalization must be explicit so a formatting conversion is not mistaken for extraction accuracy.
• No page trace: Reviewers need the source page and region for important values, especially when multiple totals appear.

Review questions

• Does the set represent rare but costly document types?
• Which fields can be auto-accepted and which always require review?
• How are handwritten changes and crossed-out text labeled?
• Can a reviewer trace each value to the source page?
• Has the final holdout remained untouched during tuning?

A 30-day implementation plan

Begin with one bounded case and an owner who can make a decision. The first milestone is define the unit of success. Write down the current state, the intended result, and the evidence that will count as complete. Keep the initial scope small enough to review in one working session, but realistic enough to expose operational friction.

During the second week, run the workflow with a colleague who did not design it. Ask them to answer: “Does the set represent rare but costly document types?” Record where they need undocumented knowledge, which data is unavailable, and which step depends on a person or system that has no backup. Fix those gaps before increasing volume or authority.

By the end of the month, repeat the process under a failure condition related to only clean sample files. Compare the observed result with the original acceptance criteria, assign unresolved actions, and set the next review date. Preserve the decision record beside the operational documentation. A modest control that is used, measured, and improved is more valuable than an ambitious design that exists only in a policy file.

Put the result into routine operations

Version the dataset, annotation rules, model, prompt, and preprocessing pipeline together. Add newly observed failure types after incidents, but keep a stable core for trend comparison. Review performance by subgroup and investigate regressions before changing the production route. Store documents with appropriate access controls and retention because test sets often contain real personal or financial data. When sharing examples with a vendor, use sanitized or synthetic fixtures that preserve the layout problem without exposing the original record.

Related Duck Cloud reading

Review incoming files with the Secure File Upload Validation Guide.

Conclusion

Document AI should earn automation through representative evidence. Build a trusted answer key, score fields by consequence, preserve difficult inputs, and combine validation with human review. That turns an impressive demo into a measurable workflow decision.