Chunking and token usage

Chunks

chunk_content() turns a document’s json_content into Chunk objects; ProcessedDocument.get_chunks() calls it for you.

from doc2mark import ChunkingConfig, chunk_content, load

result = load("report.pdf")
chunks = chunk_content(result.json_content, ChunkingConfig(max_chunk_size=800, overlap=100))
for chunk in chunks:
    print(chunk.chunk_index, chunk.section_title, chunk.page_start, chunk.page_end,
          len(chunk.content), sorted(chunk.content_types))

json_content is filled for every output format by the PDF, Word, Excel, PowerPoint and image processors. The other formats (text, CSV, JSON, HTML, XML, Markdown, e-mail) leave it empty with the default Markdown output; get_chunks() then returns the whole content as one chunk, and output_format="json" gives them a single text:normal item.

How a document is split

  1. Sections. A text:title item (level 1) or text:section item (level 2) starts a new section when its level is at most split_on_heading_level (default 2: titles and sections; 1: titles only). Every text:section item counts as level 2, whatever its own level; in chunk text a title is written # ... and every section heading ## ....

  2. Packing. The items of a section are rendered to Markdown and packed into a chunk until the next item would pass max_chunk_size. Chunks are cut only between items: a single paragraph longer than max_chunk_size is one chunk of its own, and a table is never cut. With keep_tables_whole=True (the default) a table that does not fit is still added to the current chunk; with False it starts a new one.

  3. Overlap. Every chunk after the first starts with the last overlap units of the chunk before it (from its first space on), across section boundaries too. Overlap is added after packing, so a chunk can be up to overlap longer than max_chunk_size.

  4. Footnotes. text:footnote items ([^1]: ...) are taken out of the flow and appended to the chunk that first references them as [^1]; unreferenced ones go to the last chunk. PDF body text marks footnotes as superscripts (^1^), so PDF footnotes end up in the last chunk.

Items of type text:header and text:footer (the repeated copies of a PDF’s running headers and footers) are left out; a Word header or footer is written once and chunked like other text. An image item, which is there only with extract_images=True and no OCR, is written as a data: URI Markdown image: leave image extraction off (or drop those items) when you index text. include_page_markers is accepted but has no effect.

page_start / page_end come from the items’ page numbers (pages for PDF, slides for PowerPoint, sheets for Excel); content_types is the set of item types in the chunk (it can name text:header / text:footer although their text is left out). section_hierarchy is the current title and section; in a document without a title the first section stays in it as if it were one.

Sizes in tokens

from doc2mark import ChunkingConfig, load

config = ChunkingConfig(max_chunk_size=512, overlap=64, size_unit="tokens", encoding_name="cl100k_base")
chunks = load("report.pdf").get_chunks(config)
print(len(chunks), "chunks of at most about 512 tokens")

size_unit="tokens" counts with tiktoken (pip install "doc2mark[tokenizers]"). tiktoken downloads an encoding’s data the first time it is used, so the first call needs network access (or a pre-filled TIKTOKEN_CACHE_DIR). Without tiktoken installed a warning is logged and sizes are counted in characters.

OCR token usage

When an LLM provider OCRs something during load(), the provider’s token counts for that document are added up in metadata.extra["token_usage"]:

from doc2mark import UnifiedDocumentLoader

loader = UnifiedDocumentLoader(ocr_provider="openai")
result = loader.load("scan.pdf", ocr_images=True)
print(result.metadata.extra["token_usage"])   # {'input_tokens': ..., 'output_tokens': ..., 'total_tokens': ...}

Only fresh calls count: an OCR cache hit and a result replayed from cache_dir cost nothing (a replayed document carries the original count as token_usage_cached instead). Tesseract reports no tokens, and a document that needed no OCR has no token_usage key. Each OCRResult from the OCR facade carries its own metadata["token_usage"].