A minimal RAG pipeline

Retrieval-augmented generation needs three things from a document converter: clean text, pieces of it small enough to embed, and enough context on each piece to cite it. This page builds that pipeline with doc2mark in a few lines. The search at the end is a keyword toy so the example runs anywhere; put your embedding model and vector store in its place.

Convert and chunk

from doc2mark import ChunkingConfig, UnifiedDocumentLoader

loader = UnifiedDocumentLoader(ocr_provider=None, judge="none")   # nothing leaves the machine
config = ChunkingConfig(max_chunk_size=1200, overlap=150)

records = []
for path in ["report.pdf", "report.docx", "slides.pptx"]:
    result = loader.load(path)
    for chunk in result.get_chunks(config):
        records.append({
            "id": f"{path}#{chunk.chunk_index}",
            "text": chunk.content,
            "source": result.metadata.filename,
            "section": " > ".join(chunk.section_hierarchy),
            "pages": (chunk.page_start, chunk.page_end),
        })

print(len(records), "chunks")
print(records[0]["id"], "|", records[0]["section"], "|", records[0]["pages"])

def search(query, k=3):
    """Keyword overlap: replace with embeddings and a vector store."""
    words = set(query.lower().split())
    return sorted(records, key=lambda r: -len(words & set(r["text"].lower().split())))[:k]

for hit in search("structured table with merged cells"):
    print(hit["id"], hit["pages"], hit["section"])

ProcessedDocument.get_chunks() splits the document at its title and section headings, packs the items of a section into chunks of about max_chunk_size characters (items are not cut, so a long paragraph or a table can be larger) and prepends the end of the previous chunk as overlap. Each Chunk carries its section path and page span, which is what you store next to the vector. Chunking and token usage describes the options, token-based sizes and the edge cases.

What the converter already did for retrieval

• Tables stay tables. Merged cells are kept as rowspan / colspan in an HTML table inside the Markdown, and a line break in a cell is <br>, so a row never turns into a run of loose words (Tables).

• Page furniture is thinned, not guessed away. Bare page numbers are dropped; a running header or footer keeps its first copy and its repeats stay out of content and out of the chunks (PDF text structure).

• Hidden text is not indexed. Invisible text over nothing the page shows (a known prompt injection route) is left out and reported in metadata.extra["hidden_text"].

• Reading order follows the layout: multi-column pages are read column by column.

Scanned documents

Turn OCR on for documents that may contain scans or text in pictures. doc2mark OCRs only the pages and pictures that need it, and records what it did:

from doc2mark import UnifiedDocumentLoader

loader = UnifiedDocumentLoader(ocr_provider="tesseract")
result = loader.load("scan.pdf", ocr_images=True)
chunks = result.get_chunks()
print(len(chunks), chunks[0].content[:80])
print(result.metadata.extra.get("ocr_issues"))   # None: every image was read

With an LLM provider (ocr_provider="openai" or "vertex_ai") the loader also adds the provider’s token counts to metadata.extra["token_usage"], for cost accounting. Check metadata.extra["ocr_issues"] before indexing: it lists images the provider refused, could not read, or answered with withheld sample values (The result: ProcessedDocument).