Structured OCR Schema

doc2mark’s OCR layer does not return a single free-form markdown blob. In the default structured mode, every image becomes a structured OCRPage, carried on OCRResult.document (None for structured=False answers and failed Tesseract images). The schema enforces a hard boundary between two concerns:

  • raw — what is literally on the page: a verbatim transcription, any tables, label/value fields, and additive verbatim indexes (headings, dates, metrics). No inference, no commentary.

  • interpretation — the model’s reading of the page: document type, a short summary, key findings, a knowledge graph, and confidence. This is the part that requires a language model to reason about the content.

This split is the most important idea in the schema. The raw half is always present and is the record of what the page shows; the prompts require it to be verbatim (one exception: when a structured answer came back empty and the free-form retry answered instead, raw.text holds that free-form answer). The interpretation half is None when the model was not asked to – or could not – reason about the page:

  • detail="raw" was requested (Vertex AI drops the interpretation; OpenAI asks the model to leave it out),

  • the provider is Tesseract, or

  • the structured-output parse failed and the layer fell back gracefully.

Every model on this page is a Pydantic BaseModel subclass, and every field is defaulted: the LLM providers emit them through LangChain’s with_structured_output(method="json_schema"), and OpenAI strict mode requires all properties to be present, so each field must be satisfiable without input. Optional fields serialize as anyOf: [T, null].

Note

Import these from doc2mark.ocr.schema. The convenience accessor to_markdown() collapses the structured page back into a single markdown string, which is what powers the back-compatible OCRResult.text.

The authoritative, always-current field list for each model is generated below directly from the source (defaults, types, and per-field descriptions). The prose around each block explains why the field exists and how the models fit together; the field documentation itself is the rendered docstring.

The verbatim / interpretation boundary

The single rule that holds the schema together: the interpretation layer never invents a printed value, and never moves one out of raw. Anything the model claims is on the page — a figure axis label, an entity name, a relation’s subject — is an additive copy of a string that already appears in raw.text. This keeps the sparse index intact (BM42 reads raw.text) while making the same facts queryable as typed structure.

router_invariants() checks that invariant: it returns a list of violations for a page (empty means OK), for CI and evaluation over recorded outputs. Its withholding part (withholding_violations) also runs on every structured result at run time and triggers a verbatim redo. It checks, among other things, that every verbatim figure string is a substring of raw.text, that every Section.heading came from raw.headings, that primary_date was selected from raw.dates rather than fabricated, that entity names and relation subjects/objects are substrings of raw.text, and that withheld (“illustrative”) values appear only on a high-confidence screenshot page.

When you read a page, treat raw as ground truth and interpretation as an additive overlay that is safe to ignore. Always check page.interpretation is not None before reading interpretive fields: it is None for Tesseract and for Vertex AI with detail="raw", and usually for OpenAI with detail="raw" (the model is asked to leave it out).

OCRPage — the top-level result

One image’s structured OCR result. It bundles the always-present raw extraction with the optional interpretation, and exposes one method.

Signature

class OCRPage(BaseModel):
    raw: RawExtraction = Field(default_factory=RawExtraction)
    interpretation: Optional[Interpretation] = None
to_markdown() -> str

Render a readable, escaped markdown view of the page, used as OCRResult.text: the page title (when raw.text does not start with it), raw.text, each table (html, else markdown, else a table built from headers / rows), a table of the non-illustrative metrics, the figures and, when sections carry summaries, a section outline. raw.fields, headings, dates, entities and the summary are not rendered. When the interpretation carries a page_markdown whole-page rendering that covers at least 85 % of the verbatim tokens, that rendering replaces raw.text (with a hidden raw-verbatim-tail comment keeping any uncovered tokens).

class doc2mark.ocr.schema.OCRPage(*, raw=<factory>, interpretation=None)[source]

Bases: BaseModel

One image’s structured OCR result, carried on OCRResult.document.

Parameters:
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_markdown()[source]

Render a readable markdown view of this page.

Used as the back-compat OCRResult.text and by pipelines that want a single string. Prefers structured tables/fields over the flat text dump.

This is the Markdown boundary for model output: the sanitized Table.html is the only live HTML emitted; every other field is escaped (see _escape_text_block()), and a table whose rows were withheld as illustrative is followed by a visible [N illustrative rows not transcribed] marker (fields, metrics and figures withheld likewise get [N illustrative fields/metrics/figures not transcribed]).

Return type:

str

RawExtraction — verbatim page content

A verbatim transcription of the page: no commentary, no inference. This is the raw field of OCRPage and the BM42 token source. Alongside the full text and structured tables / fields, it carries three additive verbatim indexes — headings, dates, and metrics — each entry of which is a copy of tokens already present in text, surfaced so an indexer can boost or filter without re-parsing. The metrics list holds Metric objects (typed numeric assertions); headings and dates are plain verbatim strings.

class doc2mark.ocr.schema.RawExtraction(*, text='', tables=<factory>, fields=<factory>, headings=<factory>, dates=<factory>, metrics=<factory>, detected_language=None, has_handwriting=False)[source]

Bases: BaseModel

Verbatim transcription. No commentary, no inference. BM42 token source.

Parameters:

  • text (str)

  • tables (List[Table])

  • fields (List[KeyValue])

  • headings (List[str])

  • dates (List[str])

  • metrics (List[Metric])

  • detected_language (str | None)

  • has_handwriting (bool)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Table — a transcribed table

A table transcribed verbatim, used inside RawExtraction.tables. The preferred representation is html: a clean <table> that can encode merged cells via colspan / rowspan (which the flat headers / rows grid and markdown cannot). The html value is sanitized at the model boundary by a field validator (see sanitize_table_html()) to a strict table-only allowlist, so the stored string is always safe to embed in rendered output. illustrative flags a demo/mockup table whose values are not real data, and row_count records how many sample rows a header-only illustrative table intentionally left untranscribed.

class doc2mark.ocr.schema.Table(*, caption='', headers=<factory>, rows=<factory>, html='', markdown='', illustrative=False, row_count=None)[source]

Bases: BaseModel

A table transcribed verbatim from the image.

html is the preferred representation: a clean <table> that can encode merged cells via colspan/rowspan (which headers/rows and markdown cannot). headers/rows remain a best-effort flat view for simple, machine-readable access.

Parameters:

  • caption (str)

  • headers (List[str])

  • rows (List[List[str]])

  • html (str)

  • markdown (str)

  • illustrative (bool)

  • row_count (int | None)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

KeyValue — a label/value pair

A single label/value pair, used both inside RawExtraction.fields (forms, receipts) and inside Interpretation.definitions (term/definition pairs). illustrative flags a demo/sample value from a screenshot or mockup region.

class doc2mark.ocr.schema.KeyValue(*, label='', value='', illustrative=False)[source]

Bases: BaseModel

A label/value pair, e.g. for forms and receipts.

Parameters:

  • label (str)

  • value (str)

  • illustrative (bool)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Metric — a typed numeric assertion

A single number printed on the page, captured as a flat, four-field typed fact (label, value, unit, illustrative). A Metric is an additive view of a number that is also present verbatim in raw.text — never a relocation of it. value is copied exactly as printed (e.g. "$4.2B", "98.5%", "< 100 ms") and never normalized or computed. Metrics live in RawExtraction.metrics and make a number queryable as a typed fact while the sparse index keeps reading raw.text.

class doc2mark.ocr.schema.Metric(*, label='', value='', unit='', illustrative=False)[source]

Bases: BaseModel

A single typed numeric assertion printed on the page.

Additive structured view of a number that is ALSO present verbatim in raw.text — never a relocation of it. Flat by design (4 fields) to stay fillable on weaker models; the BM42 sparse index reads raw.text while this makes the number queryable as a typed fact.

Parameters:

  • label (str)

  • value (str)

  • unit (str)

  • illustrative (bool)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

The nested interpretation models

The interpretation layer carries a small family of nested models that structure a page’s meaning — figures, hierarchy, and a knowledge graph. They are deliberately shallow (max nesting depth 4: OCRPage → interpretation → figures → data_points), with no recursion, no model-unions, and every field defaulted, so they stay fillable under with_structured_output(json_schema). They are additive: every verbatim string inside them should mirror text already in raw.text, an invariant router_invariants() checks.

  • Metric (above) — a typed number, in raw.metrics.

  • Figure — one chart / diagram / infographic panel, in interpretation.figures. Its kind drives which branch fills: quantitative kinds (bar/line/pie/…) fill data_points; structural kinds (flowchart/org_chart/network) fill nodes + edges; everything else falls back to labels + meaning. meaning (and trend for charts) is the always-attempt interpretive fallback when the typed data can’t be read.

    • DataPoint — one (label, value, series) chart reading, flattened to tidy-long form (the deepest leaf, depth 4). All three fields are verbatim copies of printed text; values are never pixel-estimated.

    • DiagramNode — one labelled box / shape / actor in a flowchart, org chart, or network. There is no synthetic id — edges reference nodes by their verbatim label.

    • DiagramEdge — a directed connection between two nodes, referenced by verbatim from_label / to_label (each must match a DiagramNode.label in the same figure).

  • Section — one heading-delimited region in reading order. Hierarchy is a flat list plus an integer level (never recursive children). Its heading is a verbatim raw.headings entry; summary and key_points are paraphrase.

  • Entity — a typed named entity (person / org / product / location / concept / other) with a salience ranking. Its name is verbatim in raw.text; dates, money, and KPIs are not entities — they live in raw.dates / raw.metrics.

  • Relation — a knowledge triple (subject / relation / object) for a claim explicitly stated on the page, with an evidence quote that makes it falsifiable. subject and object are substrings of raw.text; the predicate and evidence may paraphrase.

Finally, the interpretation can carry a page_markdown string: a clean, structured Markdown re-layout of a whole-page image render. It is filled only when explicitly instructed (slide / scan image-strategy pages) and, when filled, must cover every word, number, and CJK character from raw.text verbatim — it re-flows existing content into a readable document, it never adds or removes any.

class doc2mark.ocr.schema.Figure(*, kind='other', title='', x_axis='', y_axis='', data_points=<factory>, trend='', nodes=<factory>, edges=<factory>, labels=<factory>, meaning='', illustrative=False)[source]

Bases: BaseModel

Typed structured view of ONE chart/diagram/infographic panel, in interpretation.figures (flat list, never nested in each other). kind drives which branch fills: quantitative->data_points; structural->nodes+edges; else ->labels+meaning. Every verbatim string is an ADDITIVE copy of raw.text; meaning/trend are the always-attempt interpretive fallbacks.

Parameters:

  • kind (Literal['bar', 'line', 'pie', 'scatter', 'area', 'combo', 'table_visual', 'flowchart', 'org_chart', 'network', 'timeline_diagram', 'map', 'infographic_panel', 'other'])

  • title (str)

  • x_axis (str)

  • y_axis (str)

  • data_points (List[DataPoint])

  • trend (str)

  • nodes (List[DiagramNode])

  • edges (List[DiagramEdge])

  • labels (List[str])

  • meaning (str)

  • illustrative (bool)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class doc2mark.ocr.schema.DataPoint(*, label='', value='', series='')[source]

Bases: BaseModel

One (category, value, series) reading from a chart, flattened to tidy-long form (the deepest leaf, depth 4). label/value/series are VERBATIM copies of text also in raw.text; never pixel-estimated.

Parameters:

  • label (str)

  • value (str)

  • series (str)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class doc2mark.ocr.schema.DiagramNode(*, label='', kind='')[source]

Bases: BaseModel

One labelled box/shape/actor in a flowchart, org chart, or network. No synthetic id — edges reference nodes by verbatim label (already in raw.text).

Parameters:

  • label (str)

  • kind (str)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class doc2mark.ocr.schema.DiagramEdge(*, from_label='', to_label='', label='')[source]

Bases: BaseModel

A directed connection between two DiagramNodes, referenced by verbatim label (membership in nodes is mechanically checkable, like primary_date-in-raw.dates).

Parameters:

  • from_label (str)

  • to_label (str)

  • label (str)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class doc2mark.ocr.schema.Section(*, heading='', level=1, summary='', key_points=<factory>)[source]

Bases: BaseModel

One heading-delimited region, in reading order. Hierarchy is a FLAT list + int level (never recursive children). heading is a VERBATIM raw.headings entry; summary/key_points are paraphrase.

Parameters:

  • heading (str)

  • level (int)

  • summary (str)

  • key_points (List[str])

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class doc2mark.ocr.schema.Entity(*, name='', type='other', salience='mentioned', role='')[source]

Bases: BaseModel

A typed named entity (replaces the flat entities list). Additive view of a name already VERBATIM in raw.text. Dates/money/KPIs stay in raw.dates/raw.metrics.

Parameters:

  • name (str)

  • type (Literal['person', 'org', 'product', 'location', 'concept', 'other'])

  • salience (Literal['primary', 'secondary', 'mentioned'])

  • role (str)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class doc2mark.ocr.schema.Relation(*, subject='', relation='', object='', evidence='')[source]

Bases: BaseModel

A knowledge triple for a claim EXPLICITLY stated on the page (never inferred). Flat triple of strings; evidence makes it falsifiable. Highest confabulation risk — emit only when explicit.

Parameters:

  • subject (str)

  • relation (str)

  • object (str)

  • evidence (str)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Interpretation — the model’s reading

The model’s analysis of the page, never mixed into raw. This is the interpretation field of OCRPage, and is None whenever the model was not asked to (or could not) reason.

It classifies the page via document_type (one of 16 values: document, table, form, receipt, handwriting, code, chart, photo, screenshot, diagram, infographic, logo, stamp, mixed, blank, other) and carries the retrieval / comprehension anchors and overlays described above — page_title, primary_message, keywords, the nested figures / sections / typed_entities / relations, plus column_layout, page_role, primary_date, action_items, definitions, content_fidelity, self_confidence, legibility, and page_markdown. See the rendered field list for the exact defaults and meanings.

class doc2mark.ocr.schema.Interpretation(*, document_type='other', summary='', key_findings=<factory>, visual_notes='', page_title=None, primary_message=None, keywords=<factory>, figures=<factory>, sections=<factory>, typed_entities=<factory>, relations=<factory>, column_layout='single', page_role=None, primary_date=None, action_items=<factory>, definitions=<factory>, self_confidence=0.0, legibility='high', content_fidelity='verbatim', page_markdown=None)[source]

Bases: BaseModel

The model’s reading of the page. Never mixed into raw.

Parameters:

  • document_type (Literal['document', 'table', 'form', 'receipt', 'handwriting', 'code', 'chart', 'photo', 'screenshot', 'diagram', 'infographic', 'logo', 'stamp', 'mixed', 'blank', 'other'])

  • summary (str)

  • key_findings (List[str])

  • visual_notes (str)

  • page_title (str | None)

  • primary_message (str | None)

  • keywords (List[str])

  • figures (List[Figure])

  • sections (List[Section])

  • typed_entities (List[Entity])

  • relations (List[Relation])

  • column_layout (Literal['single', 'double', 'multi', 'complex'])

  • page_role (Literal['title', 'agenda', 'section_header', 'content', 'data', 'case_study', 'comparison', 'timeline', 'conclusion', 'appendix', 'other'] | None)

  • primary_date (str | None)

  • action_items (List[str])

  • definitions (List[KeyValue])

  • self_confidence (Annotated[float, Ge(ge=0.0), Le(le=1.0)])

  • legibility (Literal['high', 'medium', 'low'])

  • content_fidelity (Literal['verbatim', 'described', 'caption', 'skipped', 'mixed'])

  • page_markdown (str | None)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

A complete filled OCRPage (receipt)

The example below shows a fully populated page for a grocery receipt, with both the verbatim raw half and the model’s interpretation half.

from doc2mark.ocr.schema import (
    OCRPage, RawExtraction, Interpretation, Table, KeyValue, Metric,
)

page = OCRPage(
    raw=RawExtraction(
        text=(
            "ACME STORE\n"
            "123 Market St\n"
            "Apples  2  $3.00\n"
            "Bread   1  $2.50\n"
            "Milk    1  $4.00\n"
            "Total      $9.50\n"
            "Thank you for shopping!"
        ),
        tables=[
            Table(
                caption="Items",
                headers=["Item", "Qty", "Price"],
                rows=[
                    ["Apples", "2", "$3.00"],
                    ["Bread", "1", "$2.50"],
                    ["Milk", "1", "$4.00"],
                ],
            ),
        ],
        fields=[
            KeyValue(label="Store", value="ACME STORE"),
            KeyValue(label="Total", value="$9.50"),
        ],
        headings=["ACME STORE"],
        metrics=[Metric(label="Total", value="$9.50")],
        detected_language="en",
        has_handwriting=False,
    ),
    interpretation=Interpretation(
        document_type="receipt",
        summary=(
            "Grocery receipt from ACME Store listing three items "
            "with a total of $9.50."
        ),
        key_findings=["3 line items", "Total: $9.50"],
        primary_message="Three grocery items were purchased for $9.50.",
        self_confidence=0.94,
        legibility="high",
    ),
)

# Always-present raw half:
assert page.raw.fields[1].value == "$9.50"

# Interpretation present here because an LLM provider ran with full detail:
assert page.interpretation is not None
assert page.interpretation.document_type == "receipt"

# Single-string markdown view (back-compat OCRResult.text):
print(page.to_markdown())

Helpers

doc2mark.ocr.schema.sanitize_table_html(html)[source]

Sanitize model-produced table HTML to a strict table-only allowlist.

Keeps only table-structural tags, <br> and the colspan/rowspan/scope attributes (integer spans, the four scope keywords); drops scripts, styles, event handlers, URLs, comments, processing instructions and every other tag/attribute, keeping their text. Line structure inside a cell (<br>, <p>, <li>, <div> or a raw newline) becomes <br>, so words and numbers never run together. Text outside any table is kept: before a table it becomes that table’s <caption> (a title or a unit line such as Unit: NT$ thousand), after the last table it follows it. A Markdown pipe table put in the field is converted to an HTML table. Fails closed: returns "" when the input is empty or cannot be parsed, so unsanitized HTML is never emitted.

Parameters:

html (str)

Return type:

str

doc2mark.ocr.schema.router_invariants(page)[source]

Return the router firewall violations for a structured page (empty = OK).

Protects the BM42 invariant: real printed values are never withheld (marked illustrative) except on a high-confidence screenshot page. Verbatim substring checks are whitespace-normalized (see _norm_ws()). The OCR providers enforce the withholding subset (withholding_violations()) on every structured result; the full set also serves as a CI/eval assertion over recorded structured outputs.

Parameters:

page (OCRPage)

Return type:

List[str]

doc2mark.ocr.schema.withholding_violations(page, *, context_attached=None)[source]

The router-firewall violations that withhold printed values (empty = OK).

The subset of router_invariants() that the OCR providers enforce at run time on every structured result (a violating result is redone with a verbatim task): withheld (illustrative) values anywhere but on a high-confidence, legible screenshot, and a described/caption fidelity without the meaning in interpretation.summary. With context_attached=False (no neighbor-page PDF was attached to the request) any withholding is a violation too: the router’s third gate needs that context, and the prompt says so.

Parameters:

  • page (OCRPage)

  • context_attached (bool | None)

Return type:

List[str]