Chunking

See Chunking and token usage for how a document is split (sections, packing, overlap, footnotes) and A minimal RAG pipeline for a pipeline.

doc2mark.chunk_content(json_content, config=None)

Split structured document content into section-aware chunks.

Parameters:

• json_content (List[Dict[str, Any]]) – The json_content list from a ProcessedDocument.

• config (ChunkingConfig | None) – Chunking parameters. Uses defaults when None.

Returns:

Ordered list of Chunk objects.

Return type:

List[Chunk]

class doc2mark.ChunkingConfig(max_chunk_size=1500, overlap=200, split_on_heading_level=2, keep_tables_whole=True, include_page_markers=False, size_unit='chars', encoding_name='cl100k_base')

Configuration for the chunking algorithm.

Fields (defaults): max_chunk_size (1500), overlap (200), split_on_heading_level (2), keep_tables_whole (True), include_page_markers (False, no effect), size_unit ("chars" or "tokens"), encoding_name ("cl100k_base", used with "tokens").

Parameters:

• max_chunk_size (int)

• overlap (int)

• split_on_heading_level (int)

• keep_tables_whole (bool)

• include_page_markers (bool)

• size_unit (Literal['chars', 'tokens'])

• encoding_name (str)

class doc2mark.Chunk(content, section_title=None, section_hierarchy=<factory>, page_start=None, page_end=None, content_types=<factory>, chunk_index=0)

A section-aware chunk of document content.

Fields: content (Markdown), section_title, section_hierarchy (the title and section headings above it), page_start, page_end, content_types (item types in the chunk), chunk_index (from 0).

Parameters:

• content (str)

• section_title (str | None)

• section_hierarchy (List[str])

• page_start (int | None)

• page_end (int | None)

• content_types (List[str])

• chunk_index (int)