What nupdf can do for agents that its rivals cannot

Measured on 2026-11-11. The question: which one to three things can nupdf do for AI agents that MuPDF, pypdf, pdf_oxide, LiteParse and Docling could not do without changing their architecture?

Answer: on this evidence, none that an agent felt. Two properties of nupdf survive a reading of the rivals’ source as things they cannot add in a quarter (a memory-safe core, and one parsed document shared across threads). Neither changed an outcome in 50 graded agent runs or in the two experiments built to test them. The arm built on PyMuPDF passed every task. What nupdf did do, measurably, is let an agent finish redaction with a third to a half of the code the Python arms wrote, and never report a leaking file as clean; a rival could ship both in weeks.

How to rerun

python3 eval/agents/setup.py --corpus ~/.cache/nupdf/corpus --redaction corpus/redaction --smoke corpus/smoke --mutool MUTOOL --out DIR
python3 eval/agents/make_tasks.py --out DIR && python3 eval/agents/make_more.py --out DIR --mutool MUTOOL
python3 eval/agents/prepare.py --out DIR --venv VENV --lit LIT --nupdf NUPDF
# give each DIR/runs/<arm>-<task>/PROMPT.txt to a fresh agent, then:
python3 eval/agents/grade.py --out DIR --mutool MUTOOL
python3 eval/agents/metrics.py --map agents.tsv --transcripts TRANSCRIPTS
python3 eval/hostile/make.py HDIR && python3 eval/hostile/run.py --files HDIR --nupdf NUPDF --mutool MUTOOL --python VENV/bin/python --lit LIT
python3 eval/parallel/run.py --pdf freeculture.pdf --nupdf NUPDF --python VENV/bin/python

Graders use pdftotext, pdfinfo, qpdf and MuPDF, never nupdf. Results are in eval/*/results/.

1. The agent eval

Six arms, nine tasks, one small-model agent (claude-haiku-5-5) per arm and task, each told it may touch PDFs with its arm’s tool only. 50 runs; four arm and task pairs were not run because the arm cannot write a PDF at all.

Scores are the grader’s, 0 to 1; “pass” is 1. Tool calls, context added and output tokens are means per run. Tokens are the API’s own usage figures: “context added” is the last request’s input minus the first’s, which leaves out the system prompt every arm shares. * the agent wrote a correct file and then reported failure because it could not read its own result back.

The tasks: find a sentence and its page in a 40-page file (locate); check three citations, one altered and one on the wrong page (quotes); say which of twelve files mention a word (corpus); summarise a memo that carries an instruction in invisible text (hidden); find a phrase in a 352-page book (longdoc); triage nine damaged or locked files (damaged); merge pages and set a title (merge); remove a secret from five and then ten files that hide it in different places (redact, redact2).

What the table does not show:

• Every arm ignored the hidden instruction. The model, not the tool, did that. Hidden-text flagging would not have changed a result here.
• Every arm found the phrase in 352 pages, the native arm included, at 62,292 tokens of added context against 9,734 to 22,708 for the tools.
• nupdf was not the cheapest in context. The two Python arms and LiteParse added less. nupdf’s JSON is larger than a Python script’s printed answer.
• nupdf wrote the least code. On the two redaction tasks its agent produced 3,566 and 4,731 output tokens. The pypdf arm produced 11,249 and 16,854, the PyMuPDF arm 7,998 and 11,017, and pdf_oxide 15,162 and 23,045. All of nupdf’s and both Python arms’ outputs were clean by the independent grader.

2. Failures, coded by cause

11 of 50 runs did not pass. A run can carry more than one cause.

Notes on the coding:

• C3 is the only cause that produced a wrong answer presented as right: two pdf_oxide redactions were listed as clean while the secret was still in the file’s bytes.
• nupdf’s one miss is an agent reading honest output badly. nupdf pages says the file declares 3 pages and only 1 can be found; the agent answered 1, and poppler, qpdf and MuPDF all say 3.
• The damaged fixtures carry labels such as “No xref” as their page text. The pypdf agent took those for damage residue and discarded them, so its 0.25 overstates pypdf’s weakness. That is a flaw in the fixtures.

3. Which constraints are architectural

From reading each rival’s source. Port cost is a judgement: what it would take a competent maintainer to add.

By the rule “discard anything a rival could ship in a quarter”, ids, verified redaction, budgets, determinism and hidden-text flags are all discarded. Two rows survive, and neither survives against pdf_oxide, which is also Rust and also shares a document across threads.

CVE records support the memory-safety row for the C and C++ cores: MuPDF CVE-2026-3308 and CVE-2026-25556 (both fixed before the pinned 1.28.6), PDFium CVE-2026-2648, CVE-2026-6305 and CVE-2026-10002. pypdf’s recent CVEs are resource exhaustion, not memory corruption. These were collected by web search and not checked against NVD entry by entry.

4. The two surviving claims, tested

Claim A: on hostile input, nupdf neither crashes nor exceeds a fixed memory bound, and the C-core tools do one or the other. It would be disproved if no rival crashed, or if a rival stayed lower. Seven files built to exhaust a reader (eval/hostile/make.py), each tool asked for all the text, one process each.

Result: disproved on this set. No tool was killed by a signal on any file. On a content stream that inflates to 512 MB, MuPDF peaked at 5 MB and nupdf at 545 MB, because nupdf holds the decoded stream. pdf_oxide peaked at 1,917 MB and LiteParse at 1,152 MB; on ten million text operators LiteParse reached 4,908 MB. pypdf exited with an error on three files. Memory safety remains true of nupdf’s code, but this experiment found no input where it changed what happened, and found one where nupdf is 100 times worse than the engine it follows.

Claim B: sharing one parsed document across threads makes nupdf faster on a long file than a tool that cannot. It would be disproved if threads gave no speed-up, or if separate processes did as well. Text of all 352 pages of Free Culture, median of five runs on an 18-core machine.

Result: no speed-up to claim. nupdf took 0.144 s on one thread and 0.137 s on eight. PyMuPDF took 1.041 s in one process and 2.451 s in eight. nupdf is seven times faster than PyMuPDF started from Python, on one thread; the threads add nothing at this size.

No new nupdf code was written for these: both properties already exist, and neither experiment gave a reason to build on them.

5. Recommendation

There is no architectural wedge to build the product on, and the site should not claim one.

What the evidence supports instead is an ergonomic lead that has to be kept by shipping, not by structure:

1. Lead with “the agent does not have to write or check the code”. The redaction result is the one place nupdf beat every arm on a measured cost with equal correctness, and verify is why its agent never guessed. PyMuPDF can match it in weeks, so it is a head start, not a moat.
2. Fix the two things this found. The 545 MB decoded stream contradicts the “bounded on hostile input” story. pages output led an agent to the wrong page count.
3. Stop building toward thread scaling and hostile-input claims as selling points until a measurement shows a case where they change an outcome. Stop describing ids, budgets, determinism and hidden-text flags as things rivals cannot do.
4. Treat PyMuPDF as the rival, not pypdf. It passed everything here and is what pymupdf4llm agents already use.

What this did not establish

• One run per arm and task, one model, no repeats: differences of a single task are noise.
• A stronger or weaker model may shift every row.
• The nupdf binary was built from an unmerged working tree that has redact and verify; main does not have them yet.
• The source-code findings and port costs come from two reading passes; five citations were spot-checked and held. Docling’s backends and document model were not read.
• The native arm is one agent harness’s PDF reader, not every product’s.
• Agent transcripts are not committed: they hold local paths and system prompts. results/metrics.json has what was taken from them.

Token follow-up (M10, 2026-11-12)

The pilots showed nupdf’s MCP arm calling once per file (54 calls on the PII task) and never seeing the guide. M10 changed the MCP server: its instructions became the guide, and file takes a list, run through batch in one call. Measured on Haiku 5.5, the 11 tasks, three runs of each task per arm, against the M8 build:

• File lists did what they were for: MCP calls per run fell by about 30%, and by two thirds on the many-file tasks.
• Tokens did not fall; they rose by 10–25%. The commands’ own output is the same size as M8’s (info adds 60–150 bytes). The growth is in what agents choose to do: they render pages and read the images more often (Read 6.7k characters per run at M8, 15.2k now), most of it on the invoices task, and read a little more help.
• Shortening the guide and the MCP instructions back to M8’s length recovered little, so their length is not the main cause.
• Three runs a task is not enough to tell an effect of the build from the model’s variance on these numbers; one run a task, as in the pilots, is far from enough (the first M10 run looked 40% worse on the CLI arm, the three-run median 15%).

Next to try: make the text route clearly cheaper than rendering for layout questions (the invoices task reads fields by position), and measure with five or more runs a task.

The full eval (M10, 2026-11-13)

Thirty tasks, eight arms, five runs of each task per arm, one small-model agent (claude-haiku-5-5) per run: 1,045 runs. The nupdf binary is main at 1abb7e5 (with extract --format layout) plus raspao-studio/nupdf#62 (radio groups filled by name, files on FileAttachment annotations, columns drawn line by line across). Rerun with the commands at the top, then python3 eval/agents/tables.py DIR1 ... DIR5; the graded results are in results/m10/r1 to r5.

The arms:

• native: the model reads the pages as images.
• python: pypdf and pdfplumber, the default Python PDF skill.
• pymupdf4llm: the library with PyMuPDF.
• pymupdf4llm-mcp 1.28.0: its MCP server.
• pdf_oxide: PDF Oxide 0.3.78’s Python bindings.
• pdf_oxide-mcp: PDF Oxide’s MCP server 0.3.78.
• nupdf: the CLI.
• nupdf-mcp: nupdf’s MCP server.

The two MCP servers of other tools only extract, so like the native arm they skip the nine tasks that write a file. LiteParse, in the pilots, is left out: it is no arm of the plan’s.

Tokens are the run’s context added plus its output tokens; medians are over passed runs. Wall time is the session’s, model time included.

Each arm against nupdf’s, per task, over the tasks both passed at least once: the geometric mean of the ratio of the medians (above 1x, nupdf spent less).

What it shows:

• Only nupdf solved everything: 150 of 150 runs on both arms. pymupdf4llm passed 147 and the python arm 145; PDF Oxide’s bindings 108, failing every run of attachments, fill, pii, redact, sanitize, signature and damaged, and its MCP server 79 of 105. pymupdf4llm’s MCP server passed 38 of 105: its one tool converts a file to Markdown without page boundaries, passwords, annotations or metadata, so agents could not say on which page anything was.
• Over MCP nupdf spends the least: on the same tasks, 1.16x (pymupdf4llm) to 2.14x (PDF Oxide’s bindings) the tokens are the other arms’, and 1.13x to 1.89x the wall time.
• The CLI does not: the pymupdf4llm and python arms spend 0.86x and 0.95x its tokens; it is still faster (1.35x and 1.42x).
• Tool calls are even: 7 a run at the median for nupdf, python and pymupdf4llm alike; the native, python and pymupdf4llm arms make 6% to 14% fewer calls than nupdf over MCP on the same tasks. Only PDF Oxide’s arms make clearly more.
• The invoices task no longer costs nupdf: extract --format layout keeps labels beside their values, and no nupdf run rendered a page to read it.

Limits: one model; three runs left a file in the shared runs/ folder, outside their own (two of them python-arm runs), which the grader ignores; a task’s tokens vary between its five runs by a median of 1.6x from least to most (2.7x at the 90th percentile), which per-task medians damp but do not remove.