Architecture¶
Both entry points — the document2md command line
(document2md.cli) and BatchConverter
(document2md.batch) used directly from Python — go through the
same pipeline. BatchConverter resolves which backend converts a given
document (mineru or pymupdf; see Backends), starting
MineruServer to keep a single
mineru-api process warm across a batch only for the mineru backend.
document2md.converter shells out to mineru and
document2md.pymupdf_backend reads a PDF’s own embedded text
layer; document2md.tables rewrites mineru’s raw HTML table
fallback into Markdown tables, and document2md.cutter optionally
crops the result down to one section by title, for either backend’s
output.
The sections below walk the pipeline in the order a conversion actually flows through it.
document2md.cli — command-line entry point¶
The document2md console script. Parses arguments and drives one
BatchConverter conversion, printing the
resolved backend and where the result was saved. See the user site’s
Command line
page for its flags and worked examples; document2md.cli for the full API
reference.
document2md.batch — Python entry point¶
BatchConverter is the package’s public entry
point when used from Python, re-exported off document2md itself.
As a context manager, entering it resolves the requested backend and
starts a persistent mineru-api server only when that backend is
mineru (skipped if a caller further up already has one running via
MINERU_API_URL — see document2md.mineru_server below); exiting
stops it. Calling it converts one document — a single PDF path, or a list
of image paths for a document spanning several scanned pages — to
Markdown. See Backends for how resolution and dispatch work, and
the user site’s Python
page for worked examples; document2md.batch for the full API reference.
document2md.mineru_server — keeping mineru-api warm¶
BatchConverter.__enter__ starts a
MineruServer, which launches
mineru-api as a subprocess, waits for it to report healthy, and points
every conversion in the batch at it via the MINERU_API_URL environment
variable — instead of the mineru CLI spinning up (and reloading all
layout/OCR models into) a fresh temporary server on every single
invocation. See document2md.mineru_server for the full API reference.
document2md.converter — running mineru¶
Each BatchConverter.__call__, under the mineru backend, shells out
to the mineru CLI — reusing the MINERU_API_URL server above when
set — to OCR a PDF (convert_to_markdown)
or one or more scanned page images
(convert_images_to_markdown), then hands
mineru’s raw Markdown to document2md.tables below before writing the
result to disk. See document2md.converter for the full API reference.
document2md.pymupdf_backend — reading a PDF’s own text layer¶
Under the pymupdf backend, BatchConverter.__call__ instead reads a
born-digital PDF’s own embedded text layer with
pymupdf4llm
(convert_to_markdown) — far faster
than OCR, at the cost of slightly worse structure, and with no image
extraction. See Backends for how a PDF qualifies for this backend
and document2md.pymupdf_backend for the full API reference.
document2md.tables — HTML tables to Markdown tables¶
mineru renders simple tables as Markdown but falls back to raw HTML
(<table>…</table> with rowspan/colspan) for anything complex; this
module rewrites those into GitHub Markdown tables so the mineru
backend’s output is Markdown all the way through. See document2md.tables
for the full API reference.
document2md.cutter — cropping to a single section¶
When BatchConverter is called with titulo, the last step before
the Markdown is written is slicing it down to the text between this
section’s title and the next section’s title, as they appear in the
source document’s own index — the converted text otherwise spans whatever
sections shared a page. Needing no file on disk, this is a real,
executed example rather than one marked # doctest: +SKIP:
>>> from document2md.cutter import cut_markdown_by_titles
>>> markdown = (
... "resto de la nota anterior.\n\n"
... "## Acuerdo de regularizacion de titulos\n\n"
... "Cuerpo del acuerdo.\n\n"
... "## Norma Oficial Mexicana NOM-042-NUCL\n\n"
... "Nota siguiente, excluir.\n"
... )
>>> cut = cut_markdown_by_titles(
... markdown,
... "Acuerdo de regularizacion de titulos",
... "Norma Oficial Mexicana NOM-042-NUCL",
... )
>>> print(cut)
## Acuerdo de regularizacion de titulos
Cuerpo del acuerdo.
See document2md.cutter for the full API reference.