UnifiedDocumentLoader

UnifiedDocumentLoader is the single entry point for turning documents into Markdown, JSON, or plain text. It auto-detects the input format from the file extension, dispatches to the right internal processor (Office, PDF, text, markup, legacy, image, and optional email), and optionally runs an OCR provider over extracted images. The same instance can convert one file with load() or whole directories / explicit file lists with the batch helpers, which now support opt-in thread-based parallelism and progress reporting.

All examples assume:

from doc2mark import UnifiedDocumentLoader, OutputFormat

Constructor

UnifiedDocumentLoader(
    ocr_provider='openai',
    api_key=None,
    ocr_config=None,
    cache_dir=None,
    ocr_cache=None,
    model='gpt-5.4-mini',
    temperature=0,
    max_tokens=8192,
    max_workers=5,
    prompt_template=PromptTemplate.DEFAULT,
    timeout=30,
    max_retries=3,
    top_p=1.0,
    frequency_penalty=0.0,
    presence_penalty=0.0,
    base_url=None,
    project=None,
    location='global',
    default_prompt=None,
    task=None,
    structured=None,
    detail=None,
    table_style=None,
)

The constructor builds (and configures) the OCR provider eagerly, so most OCR tuning happens here. Pass ocr_provider=None (or the strings 'none' / 'disabled') to disable OCR entirely.

Core parameters

ocr_providerstr | OCRProvider | BaseOCR | None

OCR backend to use. A name or OCRProvider enum ('openai', 'vertex_ai', …), a pre-built OCR instance, or None to disable OCR. Defaults to 'openai'.

api_keystr | None

API key for the OCR provider. For OpenAI it falls back to the OPENAI_API_KEY environment variable when omitted.

ocr_configOCRConfig | None

Base OCR configuration. When omitted a default config is created so the structured-output defaults still apply.

cache_dirstr | None

Directory for caching converted documents on disk (keyed by file path, mtime, size, and conversion options). Distinct from ocr_cache.

ocr_cacheOCRCache | None

Request-scoped cache for individual OCR image calls. See OCR Result Caching.

table_stylestr | None

How complex tables with merged cells are rendered: 'minimal_html' (default), 'markdown_grid', or 'styled_html'.

OCR / model tuning

modelstr

Model name. Defaults to 'gpt-5.4-mini' for OpenAI; for vertex_ai the default 'gpt-5.4-mini' is automatically swapped for a Gemini model.

temperaturefloat

Sampling temperature (default 0).

max_tokensint

Maximum response tokens (default 4096).

max_workersint

Concurrency used inside the OCR provider for batched image OCR (default 5). This is independent of the max_workers argument on the batch methods, which controls document-level concurrency.

prompt_templatestr | PromptTemplate

Built-in prompt preset (e.g. 'default', 'table_focused', 'document_focused', 'multilingual', 'receipt_focused').

timeout / max_retriesint

Per-request timeout in seconds and retry budget for OCR calls.

top_p / frequency_penalty / presence_penaltyfloat

Additional OpenAI sampling controls.

base_urlstr | None

Override base URL for OpenAI-compatible endpoints.

project / locationstr | None

Vertex AI Google Cloud project ID (falls back to GOOGLE_CLOUD_PROJECT) and region (default 'global').

default_promptstr | None

Custom prompt string that overrides the built-in templates.

Structured-OCR knobs

These override the matching field on the resolved OCRConfig for LLM providers. Leaving them as None preserves the supplied config (or the config default) unchanged.

taskstr | Task | None

Override the OCR task (e.g. 'receipt').

structuredbool | None

Toggle structured-output mode. The config default is True.

detailstr | None

Interpretation detail, 'raw' or 'full'.

Note

max_workers here (OCR-internal) and the max_workers argument on batch_process() / batch_process_files() (document-level threads) are two separate dials.

load()

load(
    file_path,
    output_format=OutputFormat.MARKDOWN,
    extract_images=False,
    ocr_images=False,
    show_progress=False,
    encoding='utf-8',
    delimiter=None,
) -> ProcessedDocument

Convert a single document and return a ProcessedDocument.

file_pathstr | Path

Path to the document. Raises FileNotFoundError if it does not exist.

output_formatstr | OutputFormat

MARKDOWN (default), JSON, or TEXT. Strings are accepted and normalized.

extract_imagesbool

Extract embedded images as base64. Only meaningful for Office and PDF inputs.

ocr_imagesbool

Run OCR over extracted images. Requires extract_images=True and a configured OCR provider.

show_progressbool

Emit per-step progress log messages.

encodingstr

Text encoding for text/markup inputs (default 'utf-8').

delimiterstr | None

CSV delimiter; auto-detected when None.

Raises UnsupportedFormatError for unknown formats and ProcessingError if conversion fails.

loader = UnifiedDocumentLoader(ocr_provider=None)  # OCR not needed here
doc = loader.load('report.docx')
print(doc.content)            # Markdown string
print(doc.metadata.format)    # DocumentFormat.DOCX

OCR-enabled loading:

loader = UnifiedDocumentLoader(
    ocr_provider='openai',
    api_key='sk-...',          # or set OPENAI_API_KEY
    prompt_template='table_focused',
)
doc = loader.load(
    'scanned.pdf',
    extract_images=True,
    ocr_images=True,           # transcribe image content into the output
)
print(doc.content)

batch_process()

batch_process(
    input_dir,
    output_dir=None,
    output_format=OutputFormat.MARKDOWN,
    extract_images=False,
    ocr_images=False,
    recursive=True,
    show_progress=True,
    save_files=True,
    encoding='utf-8',
    delimiter=None,
    max_workers=None,
    progress_callback=None,
) -> Dict[str, Dict[str, Any]]

Discover every supported file under input_dir and convert each one. Returns a dict keyed by the string file path; the keys are returned in deterministic input order regardless of completion order.

input_dirstr | Path

Directory to scan. Raises FileNotFoundError if missing.

output_dirstr | Path | None

Where outputs are written. Defaults to input_dir.

recursivebool

Recurse into subdirectories (default True).

save_filesbool

Write .md / .json (and any extracted images) to output_dir.

max_workersint | None

Opt-in parallelism. When set to a value greater than 1 and more than one file is found, documents are converted concurrently with a thread pool (capped at the file count). None (default) keeps the original sequential behavior.

progress_callbackCallable[[int, int, str], None] | None

Invoked as callback(done, total, path) after each file finishes — whether it succeeded or failed. path is the same string key used in the returned dict.

extract_images, ocr_images, output_format, show_progress, encoding, and delimiter behave as in load().

Result dict shape

Each value is a per-file result dict. On success:

{
    'status': 'success',
    'format': 'docx',              # DocumentFormat value
    'content_length': 1234,        # len(content), characters
    'duration': 0.42,              # seconds spent on this file
    'output_files': ['out/report.md'],   # written paths (empty if save_files=False)
    'metadata': {
        'images_extracted': 3,
        'tables_found': 1,
        'pages': 2,                # batch_process only
    },
}

On failure the entry is compact:

{
    'status': 'failed',
    'error': 'Processing failed: ...',
    'format': '.pdf',             # file suffix
}

Note

batch_process includes 'pages' in metadata; batch_process_files() omits it (it reports only images_extracted and tables_found).

Example with parallelism and a progress bar:

loader = UnifiedDocumentLoader(ocr_provider=None)

def on_progress(done, total, path):
    print(f'[{done}/{total}] {path}')

results = loader.batch_process(
    'docs/',
    output_dir='out/',
    max_workers=4,               # convert up to 4 files concurrently
    progress_callback=on_progress,
)

ok = sum(1 for r in results.values() if r['status'] == 'success')
print(f'{ok}/{len(results)} converted')

batch_process_files()

batch_process_files(
    file_paths,
    output_dir=None,
    output_format=OutputFormat.MARKDOWN,
    extract_images=False,
    ocr_images=False,
    show_progress=True,
    save_files=True,
    encoding='utf-8',
    delimiter=None,
    max_workers=None,
    progress_callback=None,
) -> Dict[str, Dict[str, Any]]

Like batch_process(), but converts an explicit list of files instead of scanning a directory (so there is no recursive argument). It returns the same per-file result dict described above, except each success entry’s metadata contains only images_extracted and tables_found (no pages). An empty file_paths returns an empty dict.

file_pathslist[str | Path]

Files to convert. Order is preserved in the returned dict.

output_dirstr | Path | None

Output directory. When None (or save_files=False) nothing is written to disk.

max_workers / progress_callback

Same opt-in parallelism and (done, total, path) callback as batch_process().

loader = UnifiedDocumentLoader(ocr_provider='openai')
results = loader.batch_process_files(
    ['a.pdf', 'b.docx', 'c.pptx'],
    output_dir='out/',
    extract_images=True,
    ocr_images=True,
    max_workers=3,
    progress_callback=lambda d, t, p: print(f'{d}/{t}'),
)
for path, info in results.items():
    print(path, info['status'])

API

class doc2mark.core.loader.UnifiedDocumentLoader(ocr_provider='openai', api_key=None, ocr_config=None, cache_dir=None, ocr_cache=None, model='gpt-5.4-mini', temperature=0, max_tokens=8192, max_workers=5, prompt_template=PromptTemplate.DEFAULT, timeout=30, max_retries=3, top_p=1.0, frequency_penalty=0.0, presence_penalty=0.0, base_url=None, project=None, location='global', default_prompt=None, task=None, structured=None, detail=None, table_style=None)[source]

Bases: object

Main document loader with unified API for all formats and enhanced OCR configuration.

Parameters:

  • ocr_provider (str | OCRProvider | BaseOCR | None)

  • api_key (str | None)

  • ocr_config (OCRConfig | None)

  • cache_dir (str | None)

  • ocr_cache (OCRCache | None)

  • model (str)

  • temperature (float)

  • max_tokens (int)

  • max_workers (int)

  • prompt_template (str | PromptTemplate)

  • timeout (int)

  • max_retries (int)

  • top_p (float)

  • frequency_penalty (float)

  • presence_penalty (float)

  • base_url (str | None)

  • project (str | None)

  • location (str)

  • default_prompt (str | None)

  • task (str | Task | None)

  • structured (bool | None)

  • detail (str | None)

  • table_style (str | None)

batch_process(input_dir, output_dir=None, output_format=OutputFormat.MARKDOWN, extract_images=False, ocr_images=False, recursive=True, show_progress=True, save_files=True, encoding='utf-8', delimiter=None, max_workers=None, progress_callback=None)[source]

Batch process multiple documents in a directory with full result tracking.

Parameters:

  • input_dir (str | Path) – Directory containing documents

  • output_dir (str | Path | None) – Optional output directory (default: same as input)

  • output_format (str | OutputFormat) – Output format (MARKDOWN, JSON, TEXT)

  • extract_images (bool) – Whether to extract images from documents (Office/PDF only)

  • ocr_images (bool) – Whether to perform OCR on extracted images (requires extract_images=True)

  • recursive (bool) – Whether to process subdirectories

  • show_progress (bool) – Whether to show progress messages

  • save_files (bool) – Whether to save output files

  • encoding (str) – Text encoding for text/markup files

  • delimiter (str | None) – CSV delimiter (auto-detect if None)

  • max_workers (int | None) – Optional number of worker threads. When set to a value > 1 (and more than one file is found), documents are processed concurrently with a thread pool. Defaults to None, which preserves the original sequential behavior.

  • progress_callback (Callable[[int, int, str], None] | None) – Optional callable invoked as (done, total, path) after each file finishes (whether it succeeded or failed). path is the string key used in the returned results dict.

Returns:

Dictionary mapping input paths to processing results

Return type:

Dict[str, Dict[str, Any]]

Examples

# Process with image extraction but no OCR
loader.batch_process("docs/", extract_images=True, ocr_images=False)

# Process with batch OCR
loader.batch_process("docs/", extract_images=True, ocr_images=True)

# Process concurrently with a progress bar
loader.batch_process("docs/", max_workers=4,
                     progress_callback=lambda d, t, p: print(f"{d}/{t}"))
batch_process_files(file_paths, output_dir=None, output_format=OutputFormat.MARKDOWN, extract_images=False, ocr_images=False, show_progress=True, save_files=True, encoding='utf-8', delimiter=None, max_workers=None, progress_callback=None)[source]

Batch process a specific list of files.

Parameters:

  • file_paths (List[str | Path]) – List of file paths to process

  • output_dir (str | Path | None) – Optional output directory

  • output_format (str | OutputFormat) – Output format (MARKDOWN, JSON, TEXT)

  • extract_images (bool) – Whether to extract images from documents (Office/PDF only)

  • ocr_images (bool) – Whether to perform OCR on extracted images (requires extract_images=True)

  • show_progress (bool) – Whether to show progress messages

  • save_files (bool) – Whether to save output files

  • encoding (str) – Text encoding for text/markup files

  • delimiter (str | None) – CSV delimiter (auto-detect if None)

  • max_workers (int | None) – Optional number of worker threads. When set to a value > 1 (and more than one file is provided), files are processed concurrently with a thread pool. Defaults to None, which preserves the original sequential behavior.

  • progress_callback (Callable[[int, int, str], None] | None) – Optional callable invoked as (done, total, path) after each file finishes (whether it succeeded or failed). path is the string key used in the returned results dict.

Returns:

Dictionary mapping input paths to processing results

Return type:

Dict[str, Dict[str, Any]]

Examples

# Process specific files with OCR
files = ["doc1.pdf", "doc2.docx"]
loader.batch_process_files(files, extract_images=True, ocr_images=True)

# Process concurrently with a progress callback
loader.batch_process_files(files, max_workers=4,
                           progress_callback=lambda d, t, p: print(f"{d}/{t}"))
get_available_prompt_templates()[source]

Get available prompt templates for OCR.

Returns:

Dictionary of template names and descriptions

Return type:

Dict[str, str]

get_ocr_configuration()[source]

Get current OCR configuration summary.

Returns:

Dictionary with OCR configuration details

Return type:

Dict[str, Any]

load(file_path, output_format=OutputFormat.MARKDOWN, extract_images=False, ocr_images=False, show_progress=False, encoding='utf-8', delimiter=None)[source]

Load and process a document.

Parameters:

  • file_path (str | Path) – Path to the document

  • output_format (str | OutputFormat) – Desired output format (MARKDOWN, JSON, TEXT)

  • extract_images (bool) – Whether to extract images as base64 (Office/PDF only)

  • ocr_images (bool) – Whether to perform OCR on extracted images (requires extract_images=True)

  • show_progress (bool) – Whether to show progress messages during processing

  • parameters (# Format-specific)

  • encoding (str) – Text encoding for text/markup files (default: ‘utf-8’)

  • delimiter (str | None) – Delimiter for CSV files (auto-detect if None)

Returns:

ProcessedDocument with content and metadata

Raises:
Return type:

ProcessedDocument

Note

  • extract_images and ocr_images only work with Office and PDF formats

  • encoding and delimiter only apply to text-based formats

For advanced OCR configuration, use the constructor parameters or update_ocr_configuration() method.

load_directory(directory, pattern='*', recursive=True, output_format=OutputFormat.MARKDOWN, **kwargs)[source]

Load all documents from a directory.

Parameters:

  • directory (str | Path) – Directory path

  • pattern (str) – Glob pattern for files

  • recursive (bool) – Whether to search recursively

  • output_format (str | OutputFormat) – Desired output format

  • **kwargs – Additional processor options

Returns:

List of processed documents

Return type:

List[ProcessedDocument]

set_ocr_provider(provider, api_key=None, config=None, ocr_cache=None)[source]

Change OCR provider.

Parameters:

  • provider (str | OCRProvider | BaseOCR | None) – New OCR provider

  • api_key (str | None) – API key for provider

  • config (OCRConfig | None) – OCR configuration

  • ocr_cache (OCRCache | None) – Optional OCR cache handler. Reuses existing handler when omitted.

property supported_formats: List[str]

Get list of supported formats.

Returns:

List of format extensions

update_ocr_configuration(**kwargs)[source]

Update OCR configuration dynamically.

Parameters:

**kwargs – Configuration parameters to update

Available for OpenAI OCR:

  • model: str

  • temperature: float

  • max_tokens: int

  • max_workers: int

  • prompt_template: str

  • enable_langchain: bool

  • timeout: int

  • max_retries: int

validate_ocr()[source]

Validate OCR provider configuration.

Returns:

True if OCR is properly configured

Return type:

bool

validate_ocr_setup()[source]

Validate OCR setup and return status information.

Returns:

Dictionary with validation results

Return type:

Dict[str, Any]