Backends¶
BatchConverter and the document2md CLI
share one selector: a backend argument (--backend on the CLI)
naming who converts a given document — "auto" (default), "mineru"
or "pymupdf". This page is about how that selector is implemented, and
what it takes to add a fourth backend; see the user site’s
Backends
page for the auto policy and a comparison of the two backends that ship
today.
How resolution and dispatch work¶
Three steps, in order:
Validation, in
BatchConverter.__init__. The requested name is checked againstdocument2md.batch.BACKENDS; anything else raisesValueErrorimmediately, before any I/O happens.Resolution, in
BatchConverter.__enter__."auto"becomes"mineru"whenshutil.which("mineru")finds it onPATH, otherwise"pymupdf"; any other requested name passes through unchanged. The resolved name is stored asself.backend. Still in__enter__, the resolved backend’s own dependency is checked (each backend’s_require_*function — see below) and, only formineru, aMineruServeris started. Doing this in__enter__rather than lazily on the first call means a missing dependency is reported before any document is converted, not partway through a batch.Dispatch, in
BatchConverter.__call__. A plainif/elifonself.backendroutes to the right converter function. Thepymupdfbranch additionally rejects input it structurally cannot handle — a list of page images, or a PDF that failshas_text_layer— by raisingRuntimeErrorrather than silently falling back tomineru.
The contract a backend must satisfy¶
Concretely, from mineru/pymupdf as the two examples:
A conversion function taking an input path (or list of paths, for
mineru’s scanned-page-image case) and amd_path, writing Markdown tomd_pathand returning nothing —document2md.converter.convert_to_markdownanddocument2md.pymupdf_backend.convert_to_markdown.A
_require_*()function raisingRuntimeErrorwith an install hint when the backend’s own dependency isn’t available —document2md.converter._require_mineruanddocument2md.pymupdf_backend._require_pymupdf.BatchConverter.__enter__calls the right one for the resolved backend.Lazy imports of the backend’s own dependency, inside functions rather than at module level. The docs build installs the package
--no-deps(see Development), and autodoc must still be able to import every module even when a backend’s dependency isn’t installed at all.
Adding a backend¶
Write the new module (
document2md/<name>_backend.pyis the existing naming pattern, though not enforced), with a conversion function and a_require_*()function satisfying the contract above. Import the backend’s own dependency lazily, inside functions.Add the new name to
document2md.batch.BACKENDS.Extend
BatchConverter.__enter__’s resolution: decide what"auto"should do when this backend is also available (mineru is preferred over pymupdf today — see the user site’s Backends page for why), call the new_require_*()for the resolved backend, and start any server the backend needs (most won’t).Extend
BatchConverter.__call__’s dispatch with a new branch, and decide what it does with input it can’t handle — raisingRuntimeErrornaming what to install or use instead, aspymupdfdoes, rather than silently falling back to another backend.Add the new choice to the CLI’s
--backendchoices.Tests: backend validation and resolution in
tests/test_batch.py(following the existingmineru/pymupdfcases), CLI parsing and the progress-line/error-message wiring intests/test_cli.py, and a dedicated test module for the backend’s own conversion function and_require_*()check, run for real rather than mocked if the dependency is small enough to install directly (astests/test_pymupdf_backend.pydoes).Document it: a one-sentence API reference entry and API page, this page’s contract/dispatch description if the new backend needs anything beyond it, and the user site’s Backends and Install pages.