Contextual OCR strategy

When doc2mark OCRs a PDF page in isolation, the model sees only that one image. It has no way to know how a term was spelled two pages earlier, what language the surrounding document is written in, or whether a screenshot of round numbers is a real financial table or an illustrative product mock-up. The contextual OCR strategy fixes this: when transcribing the content on page k, doc2mark can attach a small PDF of the neighboring pages {k-1, k, k+1} as context – not as something to transcribe – so the model can anchor terminology, proper names, and language continuity against the document’s own neighbors.

This produces more consistent transcriptions (the same term spelled the same way across pages), keeps the output in the document’s own language, and gives the self-routing AUTO prompt enough signal to judge a host document’s purpose before it applies any non-verbatim policy.

The idea

The transcription target never changes: the page image is, and remains, the sole thing the model transcribes. The context PDF rides alongside it as a separate, clearly-labeled attachment. Both LLM providers prepend a strict instruction (_CONTEXT_PDF_INSTRUCTION in doc2mark/ocr/base.py) telling the model:

The IMAGE above is the PRIMARY and ONLY target to transcribe. The attached PDF contains the neighboring pages (previous/current/next) of the same document and is provided STRICTLY as CONTEXT for terminology, names, and language continuity. Do NOT transcribe, summarize, or quote the PDF. […] Transcribe ONLY what is visibly present in the image, and respond in the document’s own language.

A second clause (_ROUTER_CONFIDENCE_CLAUSE) is appended so the AUTO router uses the neighbors only to judge the host document’s purpose, and may apply its describe/screenshot policies only when its self_confidence is at least 0.7 and legibility is "high" – otherwise it falls back to verbatim transcription. Context augments the model’s judgement; it never licenses dropping real text.

The `context_pages` tiers

The feature is controlled by a single field on OCRConfig:

context_pages: int = 0

This integer is a scope tier, not a window size – the window is always fixed at {k-1, k, k+1}. The three tiers are:

context_pages = 0: Off (the default). Zero behavior change – the off path is byte-identical to a build without the feature.
context_pages = 1: Attach context to whole-page renders only (one PDF upload per rendered image page). This is the image-strategy route, where a page is mostly pictures and the whole page is rasterized and OCR’d.
context_pages = 2: Renders and non-decorative embedded images. Opt-in; adds an upload per embedded figure that survives the decorative filter. More uploads, more cost.

The tier is resolved once, from the OCR instance’s config, in the PDF loader:

self._context_tier = int(getattr(cfg, "context_pages", 0) or 0)

and gates context building per image in _collect_all_images: whole-page renders pass self._context_tier >= 1; embedded images pass self._context_tier >= 2.

How the window PDF is built

_build_window_pdf(k) (in doc2mark/pipelines/pymupdf_advanced_pipeline.py) constructs the context PDF for page index k:

Fixed, clamped window. It copies pages a = max(0, k-1) through b = min(n-1, k+1) inclusive into a fresh PDF with PyMuPDF’s insert_pdf. At the document edges the window simply clamps: page 0 yields {0, 1}, the last page yields {n-2, n-1}, and a single-page document yields {0}. It is therefore never more than three pages.
Compressed. The PDF is serialized with tobytes(deflate=True, garbage=3) to compress streams and drop orphaned objects.
Per-page de-duplicated. The result is cached in a small LRU (_WINDOW_CACHE_MAXLEN = 4) keyed by page index, so the window for a given page is built at most once even though several embedded images on that page share it, and overlapping windows on adjacent pages do not rebuild work.
Size-guarded. If the compressed PDF exceeds _CONTEXT_PDF_MAX_BYTES (18 MB – chosen to stay under Gemini’s ~20 MB inline request cap), the function logs a warning and returns None. Any failure to build the PDF also returns None. Callers treat None as “no context” and OCR the image normally, so the feature degrades gracefully and never blocks a page.

The output is raw base64 (no data: URI prefix), which makes it provider-independent – each provider wraps it in its own attachment format.

How each provider attaches the context PDF

The per-image context base64 travels positionally alongside the image bytes as a context_pdfs list, and the loader only sets that keyword when at least one image in the batch actually carries context (keeping the off-default path unchanged). Each provider then attaches it as a context-only part next to the image in the same HumanMessage.

OpenAI (doc2mark/ocr/openai.py) appends the instruction text and a nested file block:

content.append({
    "type": "file",
    "file": {
        "filename": "context.pdf",
        "file_data": f"data:application/pdf;base64,{context_pdf}",
    },
})

This is attached only when the PDF is present and the model can ingest a PDF part. PDF capability is gated by _model_supports_pdf against the prefixes gpt-4o, gpt-4.1, gpt-5, and o1 – so the default model, gpt-5.4-mini, qualifies, while a non-PDF model silently skips the attachment instead of 400-ing.

Gemini / Vertex AI (doc2mark/ocr/vertex_ai.py) appends the same instruction text and a media part carrying the raw base64:

content.append({
    "type": "media",
    "mime_type": "application/pdf",
    "data": context_pdf,
})

In both providers the image stays the first and only transcription target; the PDF is an extra part the model is told to read for context only.

It augments – it never replaces the verbatim layer

Context is purely additive. The deterministic, rule-based text and table layer that doc2mark extracts from a text-authoritative PDF is always preserved verbatim for the BM42 RAG flow – the context PDF only rides on the OCR calls that handle page renders (tier 1) and embedded figures (tier 2). Turning the feature on does not alter, paraphrase, or re-derive any text that the rule-based layer already captured; it only sharpens what the vision model produces for the images it was already going to OCR.

Enabling it

Because context is built from neighboring PDF pages, it applies to PDF sources only, and you enable it by passing an OCRConfig with context_pages set through the loader:

from doc2mark import UnifiedDocumentLoader
from doc2mark.ocr.base import OCRConfig

loader = UnifiedDocumentLoader(
    ocr_provider="openai",                 # gpt-5.4-mini is PDF-capable
    ocr_config=OCRConfig(context_pages=1),  # context on whole-page renders
)

doc = loader.load(
    "scanned-report.pdf",
    extract_images=True,
    ocr_images=True,
)

Set context_pages=2 to also attach context to non-decorative embedded figures. Use Gemini/Vertex AI or a PDF-capable OpenAI model so the attachment is actually consumed.

Cost and payload, honestly

Context is not free. Every OCR call that carries it uploads an extra PDF – up to three pages – in addition to the page image:

At tier 1, that is one context PDF per rendered image page.
At tier 2, it is additionally one per surviving embedded figure, so a page with several figures multiplies the uploads (they share the same cached window PDF, but each call still ships it).

The window PDF is capped at 18 MB and compressed, but it still adds tokens and bytes to every call, which means more latency and higher per-call cost. Enable it when cross-page consistency, naming, and language continuity matter; leave it at the default 0 when they do not.