From 3af4b1af3ab09b7e4c31c4f4bcd6daf44492994f Mon Sep 17 00:00:00 2001 From: Stefan Wintermeyer Date: Fri, 11 Sep 2026 17:04:38 +0200 Subject: [PATCH 1/2] =?UTF-8?q?Ein=20Lebenslauf,=20der=20JavaScript=20erw?= =?UTF-8?q?=C3=A4hnt,=20ist=20kein=20gef=C3=A4hrliches=20PDF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Das PDF-Tor suchte die Namen /JavaScript, /OpenAction und /EmbeddedFile im ganzen Dokument, also auch im Seitentext, und wies deshalb einen Lebenslauf mit "HTML/CSS/JavaScript" als Programm ab, während pdfinfo JavaScript: no sagte. Die beiden Fragen, die poppler aus der geparsten Struktur beantwortet, stellt der Byte-Scan jetzt gar nicht mehr: gemessen erkennt pdfinfo ein Skript an jeder Aktionsstelle und pdfdetach jede angehängte Datei. Die vier übrigen Namen liest er aus der PDF-Syntax statt aus dem Text, Zeichenketten, Hex-Zeichenketten und Kommentare werden vorher geleert. Geleert wird aber kein Bereich, der ein << oder >> enthält, denn in einem Objekt-Stream liest ein Leser jedes Objekt an der Stelle, die dessen eigene Tabelle nennt, und eine Klammer im einen Objekt könnte sonst den Katalog im nächsten verstecken. Diesen Text hat ein KI-Agent in meinem Namen geschrieben. Ich weiß, dass das problematisch ist. Claude-Session: https://claude.ai/code/session_013VnEsuePUe2CB2QvDcshcp --- docs/architecture/attachments.md | 20 +- lib/vutuv/uploads/pdf_gate.ex | 291 +++++++++++++++++++++++++--- test/support/attachment_fixtures.ex | 252 ++++++++++++++++++++++-- test/vutuv/attachments_test.exs | 105 ++++++++++ 4 files changed, 622 insertions(+), 46 deletions(-) diff --git a/docs/architecture/attachments.md b/docs/architecture/attachments.md index 5afd8a7f7..c7f1e1508 100644 --- a/docs/architecture/attachments.md +++ b/docs/architecture/attachments.md @@ -36,9 +36,16 @@ never route a file past its own gate. The module's own doc has the rest. (encrypted), it runs code when opened, it does something to the reader when opened, it carries another file inside it — and **fails closed**: poppler missing, poppler failing, or a scan that could not be finished is a refusal. -Three of the four are poppler's answers (`pdfinfo` for encryption and -JavaScript, `pdfdetach -list` for embedded files); `/OpenAction` no poppler -tool reports, so it is read from the bytes. +**Which question goes to whom is the whole design** (#2136), and the split is +measured rather than assumed. Poppler parses: it resolves the cross-reference +table and walks the objects, so `pdfinfo` answers `JavaScript: yes` wherever +the script hangs — name tree, `/OpenAction`, a catalog's or a page's `/AA`, an +annotation. So `/JavaScript` left the byte scan, and that is what stopped a CV +being refused for listing "HTML/CSS/JavaScript". `pdfdetach -list` is *not* as +complete: it answers 0 for a file reached through `/AF` or a `/RichMedia` +annotation, so `/EmbeddedFile` stayed in the scan beside it. Nothing reports an +action, so `/OpenAction`, `/Launch`, `/SubmitForm` and `/ImportData` are the +bytes' own. The module records each measurement with its date. The one thing worth repeating outside that module: **a raw-byte scan alone is not enough, and this was measured.** One `qpdf --object-streams=generate` run @@ -50,6 +57,13 @@ tries every hostile PDF twice, as written and hidden that way; calibrated by removing the inflation pass, the `/OpenAction` file is then **accepted** while poppler still catches the other two. +And **what a page says is not what a document does**: a string literal, a hex +string and a comment are blanked out of every buffer before the names are +looked for, so `/JavaScript` in `(HTML/CSS/JavaScript)` or `/Launch` in a link +to `…/products/Launch` is a word. Which ranges may be blanked, and why an +object stream is the case that decides it, is in the module; both halves are +calibrated in `attachments_test.exs`. + It lives under `Vutuv.Uploads` rather than beside the context that added it because two other doors already take a member's PDF and hand it back verbatim (`Vutuv.QualificationDocument`, `Vutuv.JobReferenceDocument`), whose only check diff --git a/lib/vutuv/uploads/pdf_gate.ex b/lib/vutuv/uploads/pdf_gate.ex index 531767202..427e7e95d 100644 --- a/lib/vutuv/uploads/pdf_gate.ex +++ b/lib/vutuv/uploads/pdf_gate.ex @@ -30,22 +30,72 @@ defmodule Vutuv.Uploads.PdfGate do * **It carries another file inside it** — `pdfdetach -list`, which counts them. - ## Why a raw-byte scan is not enough, measured - - The obvious implementation greps the file for `/JavaScript`, `/OpenAction` - and `/EmbeddedFile`. One `qpdf --object-streams=generate` run defeats all - three: every dictionary moves into a Flate-compressed object stream and the - three strings are simply not in the file any more (`grep -ac` says 0 for - each, checked on 2026-09-10), while the document goes on doing exactly what - it did. - - So the two questions poppler can answer are asked of **poppler**, which - parses the structure and sees through object streams. The one it cannot — - `/OpenAction` — is asked of the raw bytes **and of every stream this can - inflate**, and since that pass is running anyway it looks for the other two - names as well; removing the inflation leaves only the `/OpenAction` case - red, which is how the test calibrates it. `#XX` escapes are tolerated, - because `/Open#41ction` is the same name to a reader. + ## Names are structure, words are not (issue #2136) + + A PDF **name** (`/Launch`) is structure. A PDF **string** (`(Launch)`) is + content: every visible character a document shows is drawn out of one, and so + is every title, bookmark and link target. Reading both alike refused a CV for + listing web skills — two macOS-produced PDFs whose text said + "TypeScript/JavaScript" came back `:javascript` while `pdfinfo` said + `JavaScript: no`. + + So a string literal, a hex string and a comment are blanked out of every + buffer before the names are looked for, byte for byte, so that the offsets + the `/OpenAction` rule reads afterwards still line up. Blanking cannot lose + an action a reader would run: inside a balanced string a name is a string, + here and in every parser alike. The one way it could is an **object stream**, + where a reader picks each object out at the offset the stream's own table + records rather than reading front to back, so a `(` planted in one object and + a `)` in the next would blank the catalog between them. That is why a + candidate range spanning a `<<` or a `>>` is **not** blanked: everything + dangerous on the far side of an object boundary is a dictionary, so a range + holding one has not proven itself content and stays in the scan. An + unterminated `(` is likewise left as the byte it is. + + The cheaper fix considered and not taken was to anchor each name on what must + stand beside it — `/S` before `/Launch`, `<<` or `[` after `/OpenAction` — + since no prose writes `/S /Launch`. It would have cost none of this code and + would also cover the XMP case below. It was passed over because it enumerates + a spelling rather than naming the effect: the anchor has to be re-guessed for + every name added, and a file is free to put a comment, an indirect reference + or 200 bytes of whitespace between the two halves. Blanking says what is + actually true — a string is not structure — and errs toward refusing. + + ## Which question is asked of whom, measured + + Blanking is not enough on its own, because a document's words also reach the + file outside PDF syntax — an XMP metadata packet is XML, so + `…HTML/CSS/JavaScript` is not a string this can blank + (Ghostscript wrote exactly that, measured 2026-09-11). What settles it is + which question a **parser** can answer, and that is measured rather than + assumed: + + * **JavaScript: poppler, alone.** `pdfinfo` answers `JavaScript: yes` for a + script in the catalog's `/Names /JavaScript` tree, in `/OpenAction`, in + the catalog's `/AA`, in a page's `/AA` and on an annotation's `/A` alike + (2026-09-11). It sees through object streams and incremental updates, and + a byte scan that cannot tell a title from a script has no business voting + beside it. `/JavaScript` is therefore **not** one of the names below. + * **Embedded files: poppler *and* the bytes.** `pdfdetach -list` counts a + file in the `/EmbeddedFiles` name tree, in a `/Collection`, and on a + `/FileAttachment` annotation with or without its `/Type /EmbeddedFile` — + but answers **0** for a filespec reached through `/AF` on the catalog, + through `/AF` on a page, or through a `/RichMedia` annotation's assets + (all measured 2026-09-11). So `/EmbeddedFile` stays in the byte scan, + which is what catches those three. No ordinary document writes that word. + * **An action: the bytes, alone.** No poppler tool reports `/OpenAction`, so + it and the three acting names are read from the file and from every + stream that inflates — one `qpdf --object-streams=generate` run moves + every dictionary into a Flate-compressed object stream, after which a + plain grep finds nothing in a file that still does the same thing + (`grep -ac` said 0, checked 2026-09-10). + + Both gates were run end to end over 4,542 real local PDFs on 2026-09-11: ten + answers changed, nine of them a refusal lifted and one refused for a + different reason, and **none** newly refused. The one pre-existing document + among the nine is a 312-page programming book that was refused for linking to + `developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/…` from a `/URI` + action. ## Where the set is not closed @@ -56,11 +106,23 @@ defmodule Vutuv.Uploads.PdfGate do encrypted file is refused before this runs), but the possibility is real and named here rather than pretended away. + An `/AcroForm /XFA` form carries its script in an XML stream rather than in a + PDF action, and neither poppler's `JavaScript:` field nor any of these names + sees it (measured 2026-09-11). Refusing XFA outright would refuse the + interactive government forms people do attach, so it is named here and left + to a decision of its own. + + An embedded file reached through `/AF` or `/RichMedia` whose stream *also* + drops its `/Type /EmbeddedFile` is seen by nothing here: poppler does not + count it and there is no name left to match. Closing that would mean refusing + `/EF` or `/Filespec` outright, which is a widening with a cost of its own and + wants measuring first. + `/AA` (additional actions) is deliberately **not** blocked as a name: it sits on the widgets of every ordinary form, and it appeared in 2 of 1,051 real - PDFs measured on this machine on 2026-09-10. What is blocked is the *action* - inside it — see `@acting_names` — so a page whose `/AA` opens a calculator is - refused while a form's widgets are not. + PDFs measured on 2026-09-10. What is blocked is the *action* inside it — see + `@acting_names` — so a page whose `/AA` opens a calculator is refused while a + form's widgets are not. """ require Logger @@ -74,6 +136,13 @@ defmodule Vutuv.Uploads.PdfGate do @stream_limit 8_000_000 @total_limit 96_000_000 + # How far a `(` may reach before this stops believing it opened a string, and + # equally how far a `<` may sit from its `>`. A cost bound rather than a + # judgement: past it the bytes are scanned as what they are, which can only + # refuse more, and without it a buffer of nothing but brackets would make the + # blanking pass quadratic. + @string_limit 65_536 + # Actions that do something to the *machine* rather than move the reader # inside the document, refused wherever they stand. Naming the effect rather # than one spelling of it: leaving these out let two files through that the @@ -91,8 +160,13 @@ defmodule Vutuv.Uploads.PdfGate do # matches `/OpenAction` and `/Open#41ction` alike, so nothing has to decode a # 20 MB buffer to find the second spelling. No `u` modifier anywhere in this # module: these patterns run over slices of a PDF, which are not text. - @name_regexes (for name <- ~w(JavaScript EmbeddedFile OpenAction) ++ @acting_names, - into: %{} do + # + # `/JavaScript` is deliberately **not** in this list — see the moduledoc: + # poppler answers that question from the parsed document, wherever the script + # hangs, and a name that is also an ordinary English word in a document title + # cost a CV its upload. `/EmbeddedFile` is still here because poppler's answer + # has three measured gaps. + @name_regexes (for name <- ["EmbeddedFile", "OpenAction" | @acting_names], into: %{} do pattern = "/" <> Enum.map_join(String.to_charlist(name), fn char -> @@ -103,6 +177,13 @@ defmodule Vutuv.Uploads.PdfGate do {name, Regex.compile!(pattern)} end) + # The same names in one alternation, for the cheap question asked first: is + # there anything here to argue about at all? + @any_name_regex @name_regexes + |> Map.values() + |> Enum.map_join("|", &Regex.source/1) + |> Regex.compile!() + # `stream` … `endstream`, bytes rather than characters (no `u` modifier, so # `.` matches any byte and a slice of a PDF cannot break the match). @stream_regex ~r/stream\r?\n(.*?)endstream/s @@ -254,12 +335,23 @@ defmodule Vutuv.Uploads.PdfGate do end end - defp scan_buffer(buffer) do + # The blanking is part of the scan rather than something each caller pipes in + # front of it: a buffer this has not read as syntax is a buffer it has read as + # text, and that is the bug this module was fixed for. + # + # It is also the expensive half — blanking a 22 MB file costs 411 ms against + # 21 ms for the names themselves (measured over 12 real PDFs on 2026-09-11) — + # so it runs only where it could change the answer. A name that is not in the + # raw bytes cannot be in the blanked ones, because blanking only ever removes, + # and not one of those 12 documents carries any of these names at all. + defp scan_buffer(raw) do + if Regex.match?(@any_name_regex, raw), do: scan_syntax(without_content(raw)), else: :ok + end + + defp scan_syntax(buffer) do cond do - Regex.match?(@name_regexes["JavaScript"], buffer) -> {:error, :javascript} Regex.match?(@name_regexes["EmbeddedFile"], buffer) -> {:error, :embedded_files} - acting?(buffer) -> {:error, :open_action} - open_action?(buffer) -> {:error, :open_action} + acting?(buffer) or open_action?(buffer) -> {:error, :open_action} true -> :ok end end @@ -302,6 +394,155 @@ defmodule Vutuv.Uploads.PdfGate do |> Enum.map(fn [{start, _length}] -> {bytes, start, total - start} end) end + ## Syntax, not text + + # Replaces every string literal, hex string and comment with the same number + # of blanks. What a page shows is drawn out of strings, and so are titles, + # bookmarks and link targets, so `/JavaScript` in `(HTML/CSS/JavaScript)` and + # `/Launch` in `(…/products/Launch)` are words rather than actions — for this + # scan and for every parser that reads the same file. + defp without_content(buffer) do + # Compiled once and carried down: `:binary.match/3` builds a fresh + # Aho-Corasick automaton for every literal *list* it is handed, which over a + # 20 MB file is 400,000 rebuilds and two thirds of this pass (407 ms against + # 138 ms precompiled, measured 2026-09-11). A compiled pattern is a + # reference, so it cannot be a module attribute. + patterns = { + :binary.compile_pattern(["(", "<<", "<", "%"]), + :binary.compile_pattern(["(", ")", "\\"]), + :binary.compile_pattern(["\n", "\r"]), + :binary.compile_pattern(["<<", ">>"]) + } + + {parts, cursor} = + buffer + |> content_ranges(0, [], patterns) + |> Enum.reverse() + |> Enum.reduce({[], 0}, fn {start, length}, {acc, cursor} -> + {[blanks(length), binary_part(buffer, cursor, start - cursor) | acc], start + length} + end) + + IO.iodata_to_binary( + Enum.reverse([binary_part(buffer, cursor, byte_size(buffer) - cursor) | parts]) + ) + end + + # A sub-binary of one shared run rather than a fresh copy per range: a 20 MB + # file yields ~22,000 ranges holding 11 MB between them, and none of it needs + # to be allocated twice. + @blanks :binary.copy(" ", 4096) + defp blanks(length) when length <= 4096, do: binary_part(@blanks, 0, length) + defp blanks(length), do: :binary.copy(" ", length) + + # Descending, so the caller reverses once. Every branch resumes at or past the + # last byte it read, which is what keeps the pass linear — see + # `blank_unless_dictionary/5`. + defp content_ranges(buffer, from, acc, patterns) do + size = byte_size(buffer) + + if from >= size do + acc + else + # Leftmost-longest, so `<<` is read as a dictionary opening rather than as + # a hex string that would swallow the dictionary's first key. + case :binary.match(buffer, elem(patterns, 0), scope: {from, size - from}) do + :nomatch -> acc + {at, 2} -> content_ranges(buffer, at + 2, acc, patterns) + {at, 1} -> content_range(:binary.at(buffer, at), buffer, at, acc, patterns) + end + end + end + + defp content_range(?(, buffer, at, acc, patterns) do + case literal_end(buffer, at + 1, at, 1, patterns) do + {:ok, stop} -> blank_unless_dictionary(buffer, at, stop, acc, patterns) + {:none, resume} -> content_ranges(buffer, resume, acc, patterns) + end + end + + defp content_range(?<, buffer, at, acc, patterns) do + # Bounded rather than "wherever the next `>` is": an unclosed `<` in the + # middle of a stream's bytes must not cost a scan of the rest of the file. + reach = min(@string_limit, byte_size(buffer) - at - 1) + + case :binary.match(buffer, ">", scope: {at + 1, reach}) do + {stop, _length} -> blank_unless_dictionary(buffer, at, stop + 1, acc, patterns) + :nomatch -> content_ranges(buffer, at + reach, acc, patterns) + end + end + + defp content_range(?%, buffer, at, acc, patterns) do + size = byte_size(buffer) + + stop = + case :binary.match(buffer, elem(patterns, 2), scope: {at, size - at}) do + {eol, _length} -> eol + :nomatch -> size + end + + blank_unless_dictionary(buffer, at, stop, acc, patterns) + end + + # A blanked range is an **exemption** from the scan, and an exemption that + # cannot be proven does not apply. What proves it is that the range holds no + # dictionary: a linear reader and a real parser part company only across an + # object boundary — inside an object stream a reader picks each object out at + # the offset the stream's own table records — and every dangerous thing on the + # other side of such a boundary is a dictionary (`/OpenAction << … >>`, + # `<< /S /Launch >>`). So a candidate string that spans a `<<` or a `>>` is + # left in place and scanned as it stands, which can only refuse more. Without + # this, a `(` in one object of an object stream and a `)` in the next would + # blank the catalog between them. + # + # Either way the scan resumes **past** the range rather than one byte into it. + # Starting over inside it is what a file can exploit: 60 KB of `%` with a `<<` + # behind them and one `/Launch` to arm the pass took 2.8 seconds of a + # LiveView's own process, quadrupling with every doubling, which puts a file + # at the 20 MB cap in the region of days (measured 2026-09-11). Skipping the + # candidates inside a range this would not exempt anyway only ever blanks + # less, which only ever refuses more. + defp blank_unless_dictionary(buffer, at, stop, acc, patterns) do + range = binary_part(buffer, at, stop - at) + + acc = + if :binary.match(range, elem(patterns, 3)) == :nomatch, + do: [{at, stop - at} | acc], + else: acc + + content_ranges(buffer, stop, acc, patterns) + end + + # `\` escapes the next byte and `(` nests, exactly as a reader parses it: + # anything else would let one file mean two things. An unterminated string is + # not a string, so the scan carries on through the `(` — and past the stretch + # it just read. + defp literal_end(buffer, from, start, depth, patterns) do + size = byte_size(buffer) + + if from >= size do + {:none, size} + else + case :binary.match(buffer, elem(patterns, 1), scope: {from, size - from}) do + :nomatch -> {:none, size} + {at, _length} when at - start > @string_limit -> {:none, at} + {at, _length} -> literal_step(:binary.at(buffer, at), buffer, at, start, depth, patterns) + end + end + end + + defp literal_step(?\\, buffer, at, start, depth, patterns), + do: literal_end(buffer, at + 2, start, depth, patterns) + + defp literal_step(?(, buffer, at, start, depth, patterns), + do: literal_end(buffer, at + 1, start, depth + 1, patterns) + + defp literal_step(?), _buffer, at, _start, 1, _patterns), do: {:ok, at + 1} + + defp literal_step(?), buffer, at, start, depth, patterns), + do: literal_end(buffer, at + 1, start, depth - 1, patterns) + + ## zlib + # Inflates one stream, stopping at `@stream_limit` rather than letting a # deliberately tiny deflate stream expand into gigabytes. defp inflate(payload) do diff --git a/test/support/attachment_fixtures.ex b/test/support/attachment_fixtures.ex index 30093d977..5c39f22f0 100644 --- a/test/support/attachment_fixtures.ex +++ b/test/support/attachment_fixtures.ex @@ -17,6 +17,136 @@ defmodule Vutuv.AttachmentFixtures do @doc "A PDF with nothing in it but a page." def plain_pdf(dir), do: write(dir, "plain.pdf", pdf(:plain)) + @doc """ + A CV whose visible words are PDF token names: the page lists + "HTML/CSS/JavaScript", a link annotation points at MDN's + `/docs/Web/JavaScript` and the document title says it again. Nothing in it is + an action — `pdfinfo` answers `JavaScript: no` and `pdfdetach` counts none — + so it is the file issue #2136 is about. `compress: false` leaves the page's + content stream uncompressed, which puts those words in the file's own bytes + rather than behind zlib. + """ + def web_skills_cv_pdf(dir, opts \\ []) do + compress? = Keyword.get(opts, :compress, true) + + text = + "BT /F1 12 Tf 72 700 Td (Jane Doe, frontend developer) Tj " <> + "0 -16 Td (Skills: HTML/CSS/JavaScript, TypeScript/JavaScript, Elixir) Tj " <> + "0 -16 Td (Wrote the team notes on /OpenAction, /Launch and /EmbeddedFile) Tj ET" + + contents = + if compress?, + do: stream_object(:zlib.compress(text), " /Filter /FlateDecode"), + else: stream_object(text) + + write( + dir, + if(compress?, do: "cv.pdf", else: "cv-uncompressed.pdf"), + pdf(:web_skills_cv, [{4, contents}], "/Info 7 0 R ") + ) + end + + @doc """ + A PDF whose JavaScript hangs somewhere other than the catalog's name tree: + `:open_action`, `:catalog_aa`, `:page_aa` or `:annotation`. The byte scan no + longer looks for `/JavaScript` at all (issue #2136), so each of these rests + on poppler's answer alone. + """ + def action_javascript_pdf(dir, where) + when where in [:open_action, :catalog_aa, :page_aa, :annotation], + do: write(dir, "js-#{where}.pdf", pdf(:"js_#{where}")) + + @doc """ + A PDF carrying another file on a `/FileAttachment` annotation rather than in + the `/EmbeddedFiles` name tree — with `typed: false`, without the + `/Type /Filespec` and `/Type /EmbeddedFile` that would name it as one. Both + rest on `pdfdetach -list` alone since #2136. + """ + def file_attachment_pdf(dir, opts \\ []) do + kind = + if Keyword.get(opts, :typed, true), do: :file_attachment, else: :file_attachment_untyped + + write(dir, "#{kind}.pdf", pdf(kind)) + end + + @doc """ + A PDF carrying another file as a PDF 2.0 **associated file**, hung off the + catalog's `/AF`. `pdfdetach -list` answers `0 embedded files` for this one + (measured 2026-09-11), so it is the byte scan's `/EmbeddedFile` that catches + it — which is why that name stayed in the scan when `/JavaScript` left it. + """ + def associated_file_pdf(dir), do: write(dir, "associated.pdf", pdf(:associated_file)) + + @doc """ + A PDF whose `/OpenAction` and `/Launch` are spelled with `#XX` escapes — + `/Open#41ction << /S /L#61unch >>` — which is the same name to a reader. + """ + def hex_escaped_launch_pdf(dir), do: write(dir, "hex-escaped.pdf", pdf(:hex_escaped)) + + @doc """ + A clean PDF with a second revision appended: a new catalog carrying a launch + action, a second cross-reference section pointing at it and a `/Prev` back to + the first. The bytes of the clean revision are untouched. + """ + def incremental_launch_pdf(dir) do + base = pdf(:plain) + [_all, previous] = Regex.run(~r/startxref\s+(\d+)/, base) + + added = + "6 0 obj\n<< /Type /Catalog /Pages 2 0 R " <> + "/OpenAction << /S /Launch /F (calc.exe) >> >>\nendobj\n" + + at = byte_size(base) + + write( + dir, + "incremental.pdf", + base <> + added <> + "xref\n0 1\n0000000000 65535 f \n6 1\n" <> + String.pad_leading(Integer.to_string(at), 10, "0") <> + " 00000 n \ntrailer\n<< /Size 7 /Root 6 0 R /Prev #{previous} >>\n" <> + "startxref\n#{at + byte_size(added)}\n%%EOF\n" + ) + end + + @doc """ + A launch action inside a Flate stream whose inflated bytes open a string + before it and close one after it. A blanking pass that took those brackets at + face value would blank the dictionary between them and never see the action; + one that refuses to exempt a range holding a `<<` reads it (issue #2136). + """ + def string_wrapped_launch_pdf(dir) do + flate_pdf(dir, "string-wrapped.pdf", [ + "(a decoy string that opens here\n", + "<< /Type /Catalog /Pages 2 0 R /OpenAction << /S /Launch /F (calc.exe) >> >>\n", + ") and closes here\n" + ]) + end + + @doc "A stream of a few hundred bytes that inflates to 20 MB." + def decompression_bomb_pdf(dir), + do: flate_pdf(dir, "bomb.pdf", :binary.copy("A", 20_000_000)) + + @doc """ + `bytes` of `%` with a `<<` behind them and no newline anywhere: one comment + that runs to the end of the buffer, holding a dictionary, so the blanking + pass may not exempt it. A pass that started over one byte later after + refusing it re-read the whole run each time, and a file only has to name + `/Launch` once — in a string, harmlessly — for that pass to run at all. + """ + def comment_flood_pdf(dir, bytes \\ 30_000) do + flate_pdf(dir, "comment-flood-#{bytes}.pdf", [ + "(a link to /Launch, which is only a word)", + :binary.copy("%", bytes), + "<< /Type /Catalog >>" + ]) + end + + @doc "A PDF header with nothing readable behind it." + def header_then_garbage_pdf(dir), + do: write(dir, "broken.pdf", "%PDF-1.7\n" <> :binary.copy(<<0xFF>>, 512)) + @doc "A PDF whose catalog carries a document-level JavaScript action." def javascript_pdf(dir), do: write(dir, "javascript.pdf", pdf(:javascript)) @@ -130,8 +260,11 @@ defmodule Vutuv.AttachmentFixtures do body = <<0x78, 0x9C>> <> stored_block(decoy) <> :zlib.zip(real) <> adler ^full = :zlib.uncompress(body) - stream = "<< /Length #{byte_size(body)} /Filter /FlateDecode >>\nstream\n#{body}\nendstream" - write(dir, "endstream-decoy.pdf", pdf(:plain, [{6, stream}])) + write( + dir, + "endstream-decoy.pdf", + pdf(:plain, [{6, stream_object(body, " /Filter /FlateDecode")}]) + ) end # One uncompressed deflate block (BTYPE 00), never the final one, so a real @@ -206,38 +339,60 @@ defmodule Vutuv.AttachmentFixtures do path end + # A one-page PDF whose object 6 is `payload`, Flate-compressed. + defp flate_pdf(dir, name, payload) do + stream = payload |> IO.iodata_to_binary() |> :zlib.compress() + + write(dir, name, pdf(:plain, [{6, stream_object(stream, " /Filter /FlateDecode")}])) + end + + defp stream_object(payload, extra \\ ""), + do: "<< /Length #{byte_size(payload)}#{extra} >>\nstream\n#{payload}\nendstream" + ## The PDFs themselves @content "BT /F1 24 Tf 72 700 Td (hello) Tj ET" + @js_action "<< /S /JavaScript /JS (app.alert\\(1\\);) >>" - defp pdf(kind, more \\ []) do + defp pdf(kind, more \\ [], trailer_extra \\ "") do {root, extra} = catalog(kind) - build( + objects = [ {1, root}, {2, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"}, {3, page(kind)}, - {4, "<< /Length #{byte_size(@content)} >>\nstream\n#{@content}\nendstream"}, + {4, stream_object(@content)}, {5, "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>"} | extra ] ++ more - ) - end - # The page dictionary. `:page_action` hangs an additional action off it, which - # is where a launch goes when there is no `/OpenAction` to put it in. - defp page(:page_action) do - "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R " <> - "/AA << /O << /S /Launch /F (calc.exe) >> >> " <> - "/Resources << /Font << /F1 5 0 R >> >> >>" + # Later entries win, so a fixture can hand `pdf/3` its own object 4. + objects |> Map.new() |> Map.to_list() |> build(trailer_extra) end - defp page(_kind) do + # The page dictionary, with whatever this kind hangs off it in the middle: + # `:page_action` an additional action, which is where a launch goes when there + # is no `/OpenAction` to put it in, and the annotation kinds an `/Annots`. + defp page(kind) do "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R " <> - "/Resources << /Font << /F1 5 0 R >> >> >>" + page_extra(kind) <> "/Resources << /Font << /F1 5 0 R >> >> >>" end + defp page_extra(:page_action), do: "/AA << /O << /S /Launch /F (calc.exe) >> >> " + defp page_extra(:js_page_aa), do: "/AA << /O #{@js_action} >> " + + defp page_extra(kind) + when kind in [ + :js_annotation, + :file_attachment, + :file_attachment_untyped, + :web_skills_cv + ], + do: "/Annots [6 0 R] " + + defp page_extra(_kind), do: "" + # A document of `count` pages, each with its own content stream saying which # page it is, so a rendered preview can be told from its neighbours. Object # numbers: 1 catalog, 2 the page tree, 3 the shared font, then a page and a @@ -271,7 +426,20 @@ defmodule Vutuv.AttachmentFixtures do defp catalog(:javascript) do {"<< /Type /Catalog /Pages 2 0 R /Names << /JavaScript << /Names [(a) 6 0 R] >> >> >>", - [{6, "<< /S /JavaScript /JS (app.alert\\(1\\);) >>"}]} + [{6, @js_action}]} + end + + # Every word in this one is inside a PDF string, which is what the gate blanks + # before it looks for a name: the page text, the `/URI` it links to and the + # `/Title` the trailer points at (issue #2136). + defp catalog(:web_skills_cv) do + {"<< /Type /Catalog /Pages 2 0 R >>", + [ + {6, + "<< /Type /Annot /Subtype /Link /Rect [72 690 300 710] " <> + "/A << /S /URI /URI (https://developer.mozilla.org/en-US/docs/Web/JavaScript) >> >>"}, + {7, "<< /Title (Curriculum Vitae - HTML/CSS/JavaScript) >>"} + ]} end defp catalog(:open_action) do @@ -289,6 +457,54 @@ defmodule Vutuv.AttachmentFixtures do defp catalog(:page_action), do: {"<< /Type /Catalog /Pages 2 0 R >>", []} + defp catalog(:js_open_action), + do: {"<< /Type /Catalog /Pages 2 0 R /OpenAction #{@js_action} >>", []} + + defp catalog(:js_catalog_aa), + do: {"<< /Type /Catalog /Pages 2 0 R /AA << /WC #{@js_action} >> >>", []} + + defp catalog(:js_page_aa), do: {"<< /Type /Catalog /Pages 2 0 R >>", []} + + defp catalog(:js_annotation) do + {"<< /Type /Catalog /Pages 2 0 R >>", + [{6, "<< /Type /Annot /Subtype /Link /Rect [0 0 10 10] /A #{@js_action} >>"}]} + end + + defp catalog(:associated_file) do + {"<< /Type /Catalog /Pages 2 0 R /AF [6 0 R] >>", + [ + {6, "<< /Type /Filespec /F (secret.txt) /EF << /F 7 0 R >> >>"}, + {7, "<< /Type /EmbeddedFile /Length 6 >>\nstream\nhidden\nendstream"} + ]} + end + + defp catalog(:hex_escaped) do + {"<< /Type /Catalog /Pages 2 0 R " <> + "/Open#41ction << /S /L#61unch /F (calc.exe) >> >>", []} + end + + defp catalog(:file_attachment) do + {"<< /Type /Catalog /Pages 2 0 R >>", + [ + {6, + "<< /Type /Annot /Subtype /FileAttachment /Rect [0 0 10 10] " <> + "/FS << /Type /Filespec /F (secret.txt) /EF << /F 7 0 R >> >> >>"}, + {7, "<< /Type /EmbeddedFile /Length 6 >>\nstream\nhidden\nendstream"} + ]} + end + + # The same attachment with nothing naming it as one: no `/Type /Filespec`, no + # `/Type /EmbeddedFile`. `pdfdetach` still counts it (measured 2026-09-11). + defp catalog(:file_attachment_untyped) do + {"<< /Type /Catalog /Pages 2 0 R >>", + [ + {6, + "<< /Type /Annot /Subtype /FileAttachment /Rect [0 0 10 10] " <> + "/FS << /F (secret.txt) /EF << /F 7 0 R >> >> >>"}, + {7, "<< /Length 6 >>\nstream\nhidden\nendstream"} + ]} + end + # 9 MB of filler in the catalog, so the object stream qpdf builds from it # inflates past `PdfGate`'s per-stream cut. The padding compresses to nothing, # which is what makes the finished file ~10 KB. @@ -310,7 +526,7 @@ defmodule Vutuv.AttachmentFixtures do # Serialises numbered objects with a cross-reference table. Nothing clever: # the offsets have to be right or poppler will not read the file, which is # what makes these fixtures worth having. - defp build(objects) do + defp build(objects, trailer_extra \\ "") do objects = Enum.sort_by(objects, &elem(&1, 0)) {body, offsets} = @@ -332,6 +548,6 @@ defmodule Vutuv.AttachmentFixtures do body <> "xref\n0 #{size}\n0000000000 65535 f \n" <> entries <> - "trailer\n<< /Size #{size} /Root 1 0 R >>\nstartxref\n#{byte_size(body)}\n%%EOF\n" + "trailer\n<< /Size #{size} #{trailer_extra}/Root 1 0 R >>\nstartxref\n#{byte_size(body)}\n%%EOF\n" end end diff --git a/test/vutuv/attachments_test.exs b/test/vutuv/attachments_test.exs index 1a0531ba1..40564eef9 100644 --- a/test/vutuv/attachments_test.exs +++ b/test/vutuv/attachments_test.exs @@ -21,6 +21,7 @@ defmodule Vutuv.AttachmentsTest do alias Vutuv.AttachmentStore alias Vutuv.MediaJobs.MediaJob alias Vutuv.Repo + alias Vutuv.WorkCounter setup do tmp = Path.join(System.tmp_dir!(), "vutuv_attachments_#{System.unique_integer([:positive])}") @@ -87,6 +88,20 @@ defmodule Vutuv.AttachmentsTest do assert {:ok, _attachment} = upload(user, Fixtures.destination_pdf(files)) end + # Issue #2136. This CV's page says "HTML/CSS/JavaScript", its title says it + # again and a link annotation points at MDN's `/docs/Web/JavaScript`, and + # none of that is an action — `pdfinfo` answers `JavaScript: no`. Every one + # of those words is inside a PDF **string**, which is what the gate now + # blanks before it looks for a name. Calibration: stop `scan_buffer/1` in + # `Vutuv.Uploads.PdfGate` from calling `without_content/1` and both halves + # go red with `{:error, :embedded_files}`, on the sentence about print PDFs. + test "a CV that lists web skills is not a program", %{user: user, files: files} do + assert {:ok, _compressed} = upload(user, Fixtures.web_skills_cv_pdf(files)) + + assert {:ok, _uncompressed} = + upload(user, Fixtures.web_skills_cv_pdf(files, compress: false)) + end + test "the intake writes a media job", %{user: user, files: files} do assert {:ok, _attachment} = upload(user, Fixtures.plain_pdf(files)) @@ -179,6 +194,96 @@ defmodule Vutuv.AttachmentsTest do assert {:error, :embedded_files} = upload(user, Fixtures.embedded_file_pdf(files)) end + # Since #2136 the byte scan does not look for `/JavaScript` at all, because + # a scan that reads a page's words cannot tell a name from a sentence. Each + # of these four rests on `pdfinfo`'s answer alone, so this is where that + # rests. Calibration: take the `JavaScript:` line out of `PdfGate.check/1` + # and all four go red — three with `{:ok, _}`, `:open_action` with its own + # reason. + test "JavaScript is refused wherever the action hangs", %{user: user, files: files} do + got = + for where <- [:open_action, :catalog_aa, :page_aa, :annotation] do + {where, upload(user, Fixtures.action_javascript_pdf(files, where))} + end + + assert Enum.all?(got, &match?({_where, {:error, :javascript}}, &1)), + "the gate answered: #{inspect(got)}" + end + + # `/EmbeddedFile` is the other half of that decision and went the other way: + # poppler counts a file on a `/FileAttachment` annotation even when nothing + # names it as one, so the first two of these are its answer — and answers + # **0** for one reached through the catalog's `/AF`, so the third is the + # byte scan's. Calibration: take `"EmbeddedFile"` out of `@name_regexes` in + # `Vutuv.Uploads.PdfGate` and the `/AF` line alone goes red with `{:ok, _}`. + test "a file carried inside is refused however it is hung", %{user: user, files: files} do + assert {:error, :embedded_files} = upload(user, Fixtures.file_attachment_pdf(files)) + + assert {:error, :embedded_files} = + upload(user, Fixtures.file_attachment_pdf(files, typed: false)) + + assert {:error, :embedded_files} = upload(user, Fixtures.associated_file_pdf(files)) + end + + # A blanking pass reads a buffer front to back; a reader picks each object + # out of an object stream at the offset its table records. Where those two + # part company, a `(` in one object and a `)` in another would blank the + # catalog between them — so a candidate string holding a `<<` is not + # blanked. Calibration: let `blank_unless_dictionary/4` in + # `Vutuv.Uploads.PdfGate` exempt every range and this goes red with + # `{:ok, _}`. + test "a launch wrapped in brackets is still a launch", %{user: user, files: files} do + assert {:error, :open_action} = upload(user, Fixtures.string_wrapped_launch_pdf(files)) + end + + # The three below are not calibrated against #2136 — the narrowing does not + # touch them, and they pass with and without it. They are here because the + # narrowing had to be shown to lose nothing, and each names a different + # layer: the `#XX` alternation in `@name_regexes`, the raw-file half of the + # scan (an incremental update leaves the first revision's bytes alone), and + # `pdfinfo` itself on a file no parser can open. + test "a launch spelled with hex escapes is refused", %{user: user, files: files} do + assert {:error, :open_action} = upload(user, Fixtures.hex_escaped_launch_pdf(files)) + end + + test "a second revision that appends a launch is refused", %{user: user, files: files} do + assert {:error, :open_action} = upload(user, Fixtures.incremental_launch_pdf(files)) + end + + test "a PDF header with nothing readable behind it is refused", %{user: user, files: files} do + assert {:error, :unreadable} = upload(user, Fixtures.header_then_garbage_pdf(files)) + end + + # `@stream_limit` without qpdf: the fixture below needs no external tool, so + # this is the one that still runs on CI. + test "a stream that inflates to 20 MB is refused", %{user: user, files: files} do + assert {:error, :unreadable} = upload(user, Fixtures.decompression_bomb_pdf(files)) + end + + # The blanking pass reads the file, so a file can make it work. Reductions + # rather than a clock, because the suite runs twenty cases at once (see + # `Vutuv.WorkCounter`). Calibrated both ways on 2026-09-11: as it stands, + # 47,915 reductions for 30 KB and 64,525 for 60 KB, the difference being + # mostly the upload around it. Make `blank_unless_dictionary/5` in + # `Vutuv.Uploads.PdfGate` resume at `at + 1` again and the same two are + # **90 million** and **321 million**, quadrupling with every doubling: 2.8 + # seconds of a LiveView's own process for a 60 KB file, and days for one at + # the 20 MB cap. + test "a file cannot make the blanking pass quadratic", %{user: user, files: files} do + {small, _answer} = + WorkCounter.count_reductions(fn -> + upload(user, Fixtures.comment_flood_pdf(files, 30_000)) + end) + + {large, _answer} = + WorkCounter.count_reductions(fn -> + upload(user, Fixtures.comment_flood_pdf(files, 60_000)) + end) + + assert large < 5_000_000, + "60 KB of comments cost #{large} reductions, 30 KB cost #{small}" + end + # The `/OpenAction` rule reads what follows the name and lets a destination # through, so an action that is not spelled `/OpenAction <>` walked # past it: `pdfinfo` answers `JavaScript: no` for both of these and the gate From 37f805c75599e921cdd9dc095d088a18db902acc Mon Sep 17 00:00:00 2001 From: Stefan Wintermeyer Date: Fri, 11 Sep 2026 18:38:12 +0200 Subject: [PATCH 2/2] =?UTF-8?q?Das=20PDF-Tor=20verl=C3=A4sst=20sich=20nich?= =?UTF-8?q?t=20mehr=20auf=20pdfinfo=20allein?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ein blinder Prüfer fand fünf Konstrukte, die das Tor annahm und speicherte: ein Öffnungsskript auf Seite 2, 3 und 20, und eine JavaScript-Aktion, die hinter /Next an einem Ziel hängt, als Dictionary wie als Array. Gemessen liest pdfinfo ohne Seitenbereich nur Seite 1, folgt keiner /Next-Kette und meldet je nach Version Verschiedenes; also bekommt der Aufruf -f 1 -l 999999, /JavaScript steht wieder neben ihm im Byte-Scan, und eine Ausgabe ohne die Felder, die wir lesen, gilt als keine Antwort statt als Unbedenklichkeitsbescheinigung. Dazu zwei Fehler dieser Änderung selbst: ein einzelnes < als letztes Byte ließ den Ausblende-Durchgang auf derselben Stelle kreisen, und statt einer Abkürzung, die weniger ausblendet und dabei genau den Lebenslauf wieder abwies, trägt der Durchgang jetzt eine Obergrenze in gelesenen Bytes. Diesen Text hat ein KI-Agent in meinem Namen geschrieben. Ich weiß, dass das problematisch ist. Claude-Session: https://claude.ai/code/session_013VnEsuePUe2CB2QvDcshcp --- docs/architecture/attachments.md | 29 +-- lib/vutuv/uploads/pdf_gate.ex | 304 +++++++++++++++++++--------- test/support/attachment_fixtures.ex | 66 +++++- test/vutuv/attachments_test.exs | 90 ++++++-- 4 files changed, 355 insertions(+), 134 deletions(-) diff --git a/docs/architecture/attachments.md b/docs/architecture/attachments.md index c7f1e1508..fdb565228 100644 --- a/docs/architecture/attachments.md +++ b/docs/architecture/attachments.md @@ -36,16 +36,17 @@ never route a file past its own gate. The module's own doc has the rest. (encrypted), it runs code when opened, it does something to the reader when opened, it carries another file inside it — and **fails closed**: poppler missing, poppler failing, or a scan that could not be finished is a refusal. -**Which question goes to whom is the whole design** (#2136), and the split is -measured rather than assumed. Poppler parses: it resolves the cross-reference -table and walks the objects, so `pdfinfo` answers `JavaScript: yes` wherever -the script hangs — name tree, `/OpenAction`, a catalog's or a page's `/AA`, an -annotation. So `/JavaScript` left the byte scan, and that is what stopped a CV -being refused for listing "HTML/CSS/JavaScript". `pdfdetach -list` is *not* as -complete: it answers 0 for a file reached through `/AF` or a `/RichMedia` -annotation, so `/EmbeddedFile` stayed in the scan beside it. Nothing reports an -action, so `/OpenAction`, `/Launch`, `/SubmitForm` and `/ImportData` are the -bytes' own. The module records each measurement with its date. +**Every question has two answerers, because each one has measured gaps** +(#2136). `pdfinfo` without a page range reads **page 1 only**, so a script on +page 2 answers `JavaScript: no` — hence the `-f 1 -l 999999` — and even with +the range it does not follow an `/OpenAction`'s `/Next` chain, and what it +reports at all moves between poppler versions. `pdfdetach -list` answers 0 for +a file reached through `/AF` or a `/RichMedia` annotation. So both names stay in +the byte scan beside the tool, and nothing reports an action at all, which +leaves `/OpenAction` and the three acting names the bytes' own. Dropping +`/JavaScript` from the scan because poppler reports scripts was tried and +reverted the same day: five constructs walked straight through. The module +records each measurement with its date. The one thing worth repeating outside that module: **a raw-byte scan alone is not enough, and this was measured.** One `qpdf --object-streams=generate` run @@ -60,9 +61,11 @@ poppler still catches the other two. And **what a page says is not what a document does**: a string literal, a hex string and a comment are blanked out of every buffer before the names are looked for, so `/JavaScript` in `(HTML/CSS/JavaScript)` or `/Launch` in a link -to `…/products/Launch` is a word. Which ranges may be blanked, and why an -object stream is the case that decides it, is in the module; both halves are -calibrated in `attachments_test.exs`. +to `…/products/Launch` is a word. That blanking is the whole of the #2136 fix. +Which ranges may be blanked, why an object stream is the case that decides it, +and why the pass carries an allowance in bytes examined — a file can otherwise +make it re-read itself until a LiveView process is gone — are in the module; +each is calibrated in `attachments_test.exs`. It lives under `Vutuv.Uploads` rather than beside the context that added it because two other doors already take a member's PDF and hand it back verbatim diff --git a/lib/vutuv/uploads/pdf_gate.ex b/lib/vutuv/uploads/pdf_gate.ex index 427e7e95d..ecd6293ab 100644 --- a/lib/vutuv/uploads/pdf_gate.ex +++ b/lib/vutuv/uploads/pdf_gate.ex @@ -46,11 +46,11 @@ defmodule Vutuv.Uploads.PdfGate do here and in every parser alike. The one way it could is an **object stream**, where a reader picks each object out at the offset the stream's own table records rather than reading front to back, so a `(` planted in one object and - a `)` in the next would blank the catalog between them. That is why a - candidate range spanning a `<<` or a `>>` is **not** blanked: everything - dangerous on the far side of an object boundary is a dictionary, so a range - holding one has not proven itself content and stays in the scan. An - unterminated `(` is likewise left as the byte it is. + a `)` in the next would blank the catalog between them. That is why blanking + **stops at the first `<<` or `>>`** inside a candidate and the scan carries on + from there: everything dangerous on the far side of an object boundary is a + dictionary, so nothing this blanks can span one. An unterminated `(` is + likewise left as the byte it is. The cheaper fix considered and not taken was to anchor each name on what must stand beside it — `/S` before `/Launch`, `<<` or `[` after `/OpenAction` — @@ -61,39 +61,41 @@ defmodule Vutuv.Uploads.PdfGate do or 200 bytes of whitespace between the two halves. Blanking says what is actually true — a string is not structure — and errs toward refusing. - ## Which question is asked of whom, measured - - Blanking is not enough on its own, because a document's words also reach the - file outside PDF syntax — an XMP metadata packet is XML, so - `…HTML/CSS/JavaScript` is not a string this can blank - (Ghostscript wrote exactly that, measured 2026-09-11). What settles it is - which question a **parser** can answer, and that is measured rather than - assumed: - - * **JavaScript: poppler, alone.** `pdfinfo` answers `JavaScript: yes` for a - script in the catalog's `/Names /JavaScript` tree, in `/OpenAction`, in - the catalog's `/AA`, in a page's `/AA` and on an annotation's `/A` alike - (2026-09-11). It sees through object streams and incremental updates, and - a byte scan that cannot tell a title from a script has no business voting - beside it. `/JavaScript` is therefore **not** one of the names below. - * **Embedded files: poppler *and* the bytes.** `pdfdetach -list` counts a - file in the `/EmbeddedFiles` name tree, in a `/Collection`, and on a - `/FileAttachment` annotation with or without its `/Type /EmbeddedFile` — - but answers **0** for a filespec reached through `/AF` on the catalog, - through `/AF` on a page, or through a `/RichMedia` annotation's assets - (all measured 2026-09-11). So `/EmbeddedFile` stays in the byte scan, - which is what catches those three. No ordinary document writes that word. - * **An action: the bytes, alone.** No poppler tool reports `/OpenAction`, so - it and the three acting names are read from the file and from every - stream that inflates — one `qpdf --object-streams=generate` run moves - every dictionary into a Flate-compressed object stream, after which a - plain grep finds nothing in a file that still does the same thing - (`grep -ac` said 0, checked 2026-09-10). - - Both gates were run end to end over 4,542 real local PDFs on 2026-09-11: ten - answers changed, nine of them a refusal lifted and one refused for a + ## Two answerers per question, because each one has measured gaps + + Blanking is what fixed the CV, and it is the *only* thing that changed about + which names are looked for: all five are still looked for. That is deliberate, + and it is the lesson of the two measurements below — **a reporting tool with + known gaps keeps its byte-scan partner.** Dropping `/JavaScript` because + `pdfinfo` reports scripts was tried and reverted the same day; five constructs + walked straight through (2026-09-11). + + * **`pdfinfo` misses at least three ways.** Without a page range it reads + **page 1 only**, so a page's `/AA /O` script on page 2 answers + `JavaScript: no` — hence the `-f 1 -l 999999` below. It does not follow an + `/OpenAction`'s `/Next` chain, as a dictionary or as an array, even with + the range. And what it reports at all moves between versions: a script in + `/OpenAction` is `yes` on poppler 26.09 here and unreported on the build + CI runs, which for an installation nobody controls means no version can be + relied on. + * **`pdfdetach -list` misses three too.** It counts a file in the + `/EmbeddedFiles` name tree, in a `/Collection` and on a `/FileAttachment` + annotation with or without its `/Type /EmbeddedFile`, but answers **0** + for a filespec reached through `/AF` on the catalog, through `/AF` on a + page, or through a `/RichMedia` annotation's assets. + * **No poppler tool reports an action at all**, so `/OpenAction` and the + three acting names have only ever been the bytes'. + + And the bytes miss things the parsers catch, which is the other half of the + argument: one `qpdf --object-streams=generate` run moves every dictionary into + a Flate-compressed object stream, after which a plain grep finds nothing in a + file that still does the same thing (`grep -ac` said 0, checked 2026-09-10). + Neither side is sufficient; both stay. + + Both gates were run end to end over 4,738 real local PDFs on 2026-09-11: nine + answers changed, seven of them a refusal lifted and two refused for a different reason, and **none** newly refused. The one pre-existing document - among the nine is a 312-page programming book that was refused for linking to + among the seven is a 312-page programming book that was refused for linking to `developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/…` from a `/URI` action. @@ -112,11 +114,11 @@ defmodule Vutuv.Uploads.PdfGate do interactive government forms people do attach, so it is named here and left to a decision of its own. - An embedded file reached through `/AF` or `/RichMedia` whose stream *also* - drops its `/Type /EmbeddedFile` is seen by nothing here: poppler does not - count it and there is no name left to match. Closing that would mean refusing - `/EF` or `/Filespec` outright, which is a widening with a cost of its own and - wants measuring first. + A script or an embedded file that is **both** invisible to poppler and hidden + from the bytes — an `/AA /O` script inside an object stream, say, or an `/AF` + file whose stream also drops its `/Type /EmbeddedFile` — is seen by nothing + here. Every construct measured so far is caught by one side or the other, and + the two-answerer shape above is what keeps that true; it is not a proof. `/AA` (additional actions) is deliberately **not** blocked as a name: it sits on the widgets of every ordinary form, and it appeared in 2 of 1,051 real @@ -156,17 +158,20 @@ defmodule Vutuv.Uploads.PdfGate do # them, so naming them costs no ordinary document. @acting_names ~w(Launch SubmitForm ImportData) + # What `check/1` reads out of `pdfinfo`. Absent means unanswered, not clean. + @required_fields [~r/^Pages:/m, ~r/^Encrypted:/m, ~r/^JavaScript:/m] + # The names, with `#XX` escapes allowed for every character — one pass that # matches `/OpenAction` and `/Open#41ction` alike, so nothing has to decode a # 20 MB buffer to find the second spelling. No `u` modifier anywhere in this # module: these patterns run over slices of a PDF, which are not text. # - # `/JavaScript` is deliberately **not** in this list — see the moduledoc: - # poppler answers that question from the parsed document, wherever the script - # hangs, and a name that is also an ordinary English word in a document title - # cost a CV its upload. `/EmbeddedFile` is still here because poppler's answer - # has three measured gaps. - @name_regexes (for name <- ["EmbeddedFile", "OpenAction" | @acting_names], into: %{} do + # `/JavaScript` and `/EmbeddedFile` are here **beside** poppler's answers, not + # instead of them: both tools have measured gaps, and the moduledoc lists + # them. What stopped a CV being refused for the words on its page is the + # blanking below, not a shorter list. + @name_regexes (for name <- ["JavaScript", "EmbeddedFile", "OpenAction" | @acting_names], + into: %{} do pattern = "/" <> Enum.map_join(String.to_charlist(name), fn char -> @@ -235,10 +240,21 @@ defmodule Vutuv.Uploads.PdfGate do ## poppler + # `-f 1 -l 999999` is not decoration: without a page range `pdfinfo` reads + # **page 1 only**, so a script on a page's `/AA /O` answered `JavaScript: no` + # for page 2 of a three-page document and `yes` for the same script on page 1 + # (measured 2026-09-11). Most documents anybody attaches have more than one + # page. defp pdfinfo(path) do - case run(:pdfinfo, [path]) do + case run(:pdfinfo, ["-f", "1", "-l", "999999", path]) do {out, 0} -> - {:ok, out} + # An answer whose shape we do not recognise is not an answer. The three + # fields below are what `check/1` reads; a poppler that prints none of + # them (or one too old to know `JavaScript:`) exits 0 all the same, and + # a positive-match test would read that silence as "clean". + if Enum.all?(@required_fields, &Regex.match?(&1, out)), + do: {:ok, out}, + else: refused(:unreadable, out) {out, _status} -> # A user password stops poppler at the door: "Command Line Error: @@ -345,11 +361,22 @@ defmodule Vutuv.Uploads.PdfGate do # raw bytes cannot be in the blanked ones, because blanking only ever removes, # and not one of those 12 documents carries any of these names at all. defp scan_buffer(raw) do - if Regex.match?(@any_name_regex, raw), do: scan_syntax(without_content(raw)), else: :ok + if Regex.match?(@any_name_regex, raw) do + case without_content(raw) do + {:ok, buffer} -> scan_syntax(buffer) + # A buffer this could not finish reading is a buffer it did not scan, + # and an unfinished scan refuses — the same answer the inflation budget + # gives. + :unfinished -> {:error, :unreadable} + end + else + :ok + end end defp scan_syntax(buffer) do cond do + Regex.match?(@name_regexes["JavaScript"], buffer) -> {:error, :javascript} Regex.match?(@name_regexes["EmbeddedFile"], buffer) -> {:error, :embedded_files} acting?(buffer) or open_action?(buffer) -> {:error, :open_action} true -> :ok @@ -414,11 +441,15 @@ defmodule Vutuv.Uploads.PdfGate do :binary.compile_pattern(["<<", ">>"]) } + case content_ranges(buffer, 0, [], patterns, budget(buffer)) do + {:ok, ranges} -> {:ok, blank(buffer, Enum.reverse(ranges))} + :unfinished -> :unfinished + end + end + + defp blank(buffer, ranges) do {parts, cursor} = - buffer - |> content_ranges(0, [], patterns) - |> Enum.reverse() - |> Enum.reduce({[], 0}, fn {start, length}, {acc, cursor} -> + Enum.reduce(ranges, {[], 0}, fn {start, length}, {acc, cursor} -> {[blanks(length), binary_part(buffer, cursor, start - cursor) | acc], start + length} end) @@ -427,6 +458,19 @@ defmodule Vutuv.Uploads.PdfGate do ) end + # What this pass may spend on one buffer, counted in **bytes examined** rather + # than in steps: one step can read the whole buffer, so steps say nothing + # about work. A backstop rather than the defence — `blank_upto_dictionary/6` + # is what keeps the pass close to linear — and set from the corpus rather than + # guessed. Measured over 4,738 real local PDFs on 2026-09-11: at this ceiling + # exactly **one** document spends it, and that one is refused either way + # (`:open_action` with an unlimited allowance, `:unreadable` with this one). + # A buffer that spends the allowance is a buffer this did not finish reading, + # and an unfinished scan refuses. + @budget_per_byte 32 + @budget_floor 4_000_000 + defp budget(buffer), do: @budget_per_byte * byte_size(buffer) + @budget_floor + # A sub-binary of one shared run rather than a fresh copy per range: a 20 MB # file yields ~22,000 ranges holding 11 MB between them, and none of it needs # to be allocated twice. @@ -434,44 +478,75 @@ defmodule Vutuv.Uploads.PdfGate do defp blanks(length) when length <= 4096, do: binary_part(@blanks, 0, length) defp blanks(length), do: :binary.copy(" ", length) - # Descending, so the caller reverses once. Every branch resumes at or past the - # last byte it read, which is what keeps the pass linear — see - # `blank_unless_dictionary/5`. - defp content_ranges(buffer, from, acc, patterns) do + # Descending, so the caller reverses once. `left` is the byte allowance, and + # every scan below pays its own distance into it. + defp content_ranges(_buffer, _from, _acc, _patterns, left) when left <= 0, do: :unfinished + + defp content_ranges(buffer, from, acc, patterns, left) do size = byte_size(buffer) if from >= size do - acc + {:ok, acc} else # Leftmost-longest, so `<<` is read as a dictionary opening rather than as # a hex string that would swallow the dictionary's first key. case :binary.match(buffer, elem(patterns, 0), scope: {from, size - from}) do - :nomatch -> acc - {at, 2} -> content_ranges(buffer, at + 2, acc, patterns) - {at, 1} -> content_range(:binary.at(buffer, at), buffer, at, acc, patterns) + :nomatch -> + {:ok, acc} + + {at, 2} -> + content_ranges(buffer, at + 2, acc, patterns, left - (at - from) - 1) + + {at, 1} -> + content_range(:binary.at(buffer, at), buffer, at, acc, patterns, left - (at - from) - 1) end end end - defp content_range(?(, buffer, at, acc, patterns) do - case literal_end(buffer, at + 1, at, 1, patterns) do - {:ok, stop} -> blank_unless_dictionary(buffer, at, stop, acc, patterns) - {:none, resume} -> content_ranges(buffer, resume, acc, patterns) + # An unterminated `(` is not a string opener, so the scan carries on from the + # byte after it. Not from wherever the walk gave up: everything between is + # ordinary syntax, and on the CV it holds the very annotation this has to + # blank. + defp content_range(?(, buffer, at, acc, patterns, left) do + case literal_end(buffer, at + 1, at, 1, patterns, left) do + {:ok, stop, left} -> + blank_upto_dictionary(buffer, at, stop, acc, patterns, left) + + # Not a string opener, so the `(` is a byte. Resume at the next dictionary + # boundary rather than at the byte after it: everything between is inside + # the same object, and re-reading it from one byte later is what a file + # can spend a LiveView process on. + {:none, left} -> + reach = min(@string_limit, byte_size(buffer) - at - 1) + + case :binary.match(buffer, elem(patterns, 3), scope: {at + 1, reach}) do + {marker, _length} -> content_ranges(buffer, marker, acc, patterns, left - reach) + :nomatch -> content_ranges(buffer, at + max(reach, 1), acc, patterns, left - reach) + end + + :unfinished -> + :unfinished end end - defp content_range(?<, buffer, at, acc, patterns) do + defp content_range(?<, buffer, at, acc, patterns, left) do # Bounded rather than "wherever the next `>` is": an unclosed `<` in the # middle of a stream's bytes must not cost a scan of the rest of the file. reach = min(@string_limit, byte_size(buffer) - at - 1) case :binary.match(buffer, ">", scope: {at + 1, reach}) do - {stop, _length} -> blank_unless_dictionary(buffer, at, stop + 1, acc, patterns) - :nomatch -> content_ranges(buffer, at + reach, acc, patterns) + {stop, _length} -> + blank_upto_dictionary(buffer, at, stop + 1, acc, patterns, left - (stop - at)) + + # Past this `<`, never back onto it: a file whose last byte is a lone `<` + # leaves nothing to reach into, and resuming at `at + reach` resumed at + # `at` and looped for ever on a 700-byte document. + :nomatch -> + content_ranges(buffer, at + 1, acc, patterns, left - reach) end end - defp content_range(?%, buffer, at, acc, patterns) do + defp content_range(?%, buffer, at, acc, patterns, left) do size = byte_size(buffer) stop = @@ -480,7 +555,7 @@ defmodule Vutuv.Uploads.PdfGate do :nomatch -> size end - blank_unless_dictionary(buffer, at, stop, acc, patterns) + blank_upto_dictionary(buffer, at, stop, acc, patterns, left - (stop - at)) end # A blanked range is an **exemption** from the scan, and an exemption that @@ -494,52 +569,81 @@ defmodule Vutuv.Uploads.PdfGate do # this, a `(` in one object of an object stream and a `)` in the next would # blank the catalog between them. # - # Either way the scan resumes **past** the range rather than one byte into it. - # Starting over inside it is what a file can exploit: 60 KB of `%` with a `<<` - # behind them and one `/Launch` to arm the pass took 2.8 seconds of a - # LiveView's own process, quadrupling with every doubling, which puts a file - # at the 20 MB cap in the region of days (measured 2026-09-11). Skipping the - # candidates inside a range this would not exempt anyway only ever blanks - # less, which only ever refuses more. - defp blank_unless_dictionary(buffer, at, stop, acc, patterns) do + # Blanking a range is an **exemption** from the scan, and an exemption that + # cannot be proven does not apply. What proves it is that the range holds no + # dictionary: a linear reader and a real parser part company only across an + # object boundary — inside an object stream a reader picks each object out at + # the offset the stream's own table records — and every dangerous thing on the + # other side of such a boundary is a dictionary (`/OpenAction << … >>`, + # `<< /S /Launch >>`). So the blanking **stops at the first `<<` or `>>`** + # inside the candidate and the scan resumes there, reading the rest as it + # stands. Without that, a `(` in one object of an object stream and a `)` in + # the next would blank the catalog between them. + # + # Cutting rather than rejecting is also what keeps the pass affordable. The + # two obvious alternatives both cost: rejecting the range and resuming one + # byte later re-reads it, which 28 of 4,738 real local PDFs could not afford + # (one needed to be read 1,024 times over), and rejecting it and resuming + # *past* it drops the candidates inside — on the CV this module exists for, + # a stray `(` in a compressed stream reaches past the annotation whose `/URI` + # names JavaScript, and the CV was refused again (both measured 2026-09-11). + defp blank_upto_dictionary(buffer, at, stop, acc, patterns, left) do range = binary_part(buffer, at, stop - at) + left = left - (stop - at) - acc = - if :binary.match(range, elem(patterns, 3)) == :nomatch, - do: [{at, stop - at} | acc], - else: acc + cut = + case :binary.match(range, elem(patterns, 3)) do + {marker, _length} -> at + marker + :nomatch -> stop + end - content_ranges(buffer, stop, acc, patterns) + acc = if cut > at, do: [{at, cut - at} | acc], else: acc + + content_ranges(buffer, max(cut, at + 1), acc, patterns, left) end # `\` escapes the next byte and `(` nests, exactly as a reader parses it: - # anything else would let one file mean two things. An unterminated string is - # not a string, so the scan carries on through the `(` — and past the stretch - # it just read. - defp literal_end(buffer, from, start, depth, patterns) do + # anything else would let one file mean two things. + defp literal_end(_buffer, _from, _start, _depth, _patterns, left) when left <= 0, + do: :unfinished + + defp literal_end(buffer, from, start, depth, patterns, left) do size = byte_size(buffer) if from >= size do - {:none, size} + {:none, left} else case :binary.match(buffer, elem(patterns, 1), scope: {from, size - from}) do - :nomatch -> {:none, size} - {at, _length} when at - start > @string_limit -> {:none, at} - {at, _length} -> literal_step(:binary.at(buffer, at), buffer, at, start, depth, patterns) + :nomatch -> + {:none, left - (size - from)} + + {at, _length} when at - start > @string_limit -> + {:none, left - (at - from)} + + {at, _length} -> + literal_step( + :binary.at(buffer, at), + buffer, + at, + start, + depth, + patterns, + left - (at - from) - 1 + ) end end end - defp literal_step(?\\, buffer, at, start, depth, patterns), - do: literal_end(buffer, at + 2, start, depth, patterns) + defp literal_step(?\\, buffer, at, start, depth, patterns, left), + do: literal_end(buffer, at + 2, start, depth, patterns, left) - defp literal_step(?(, buffer, at, start, depth, patterns), - do: literal_end(buffer, at + 1, start, depth + 1, patterns) + defp literal_step(?(, buffer, at, start, depth, patterns, left), + do: literal_end(buffer, at + 1, start, depth + 1, patterns, left) - defp literal_step(?), _buffer, at, _start, 1, _patterns), do: {:ok, at + 1} + defp literal_step(?), _buffer, at, _start, 1, _patterns, left), do: {:ok, at + 1, left} - defp literal_step(?), buffer, at, start, depth, patterns), - do: literal_end(buffer, at + 1, start, depth - 1, patterns) + defp literal_step(?), buffer, at, start, depth, patterns, left), + do: literal_end(buffer, at + 1, start, depth - 1, patterns, left) ## zlib diff --git a/test/support/attachment_fixtures.ex b/test/support/attachment_fixtures.ex index 5c39f22f0..2813104c1 100644 --- a/test/support/attachment_fixtures.ex +++ b/test/support/attachment_fixtures.ex @@ -48,14 +48,43 @@ defmodule Vutuv.AttachmentFixtures do @doc """ A PDF whose JavaScript hangs somewhere other than the catalog's name tree: - `:open_action`, `:catalog_aa`, `:page_aa` or `:annotation`. The byte scan no - longer looks for `/JavaScript` at all (issue #2136), so each of these rests - on poppler's answer alone. + `:open_action`, `:catalog_aa`, `:page_aa`, `:annotation`, `:next_dict`, + `:next_array` or `:field_calculate`. The last three, and every one of the + multi-page shapes below, are constructs `pdfinfo` does **not** report + (measured 2026-09-11), which is why `/JavaScript` is in the byte scan beside + it rather than instead of it. """ def action_javascript_pdf(dir, where) - when where in [:open_action, :catalog_aa, :page_aa, :annotation], + when where in [ + :open_action, + :catalog_aa, + :page_aa, + :annotation, + :next_dict, + :next_array, + :field_calculate + ], do: write(dir, "js-#{where}.pdf", pdf(:"js_#{where}")) + @doc """ + A `pages`-page PDF whose page `on` opens with a JavaScript action. Bare + `pdfinfo` reads **page 1 only**, so it answers `JavaScript: no` for every one + of these while answering `yes` for the identical script on page 1 — which is + why the gate passes it a page range, and why a one-page fixture calibrates + nothing (issue #2136). + """ + def page_javascript_pdf(dir, pages, on) do + write(dir, "js-page-#{on}-of-#{pages}.pdf", multi_page(pages, on)) + end + + @doc """ + A PDF whose last byte is a lone `<`, with a name in the raw bytes so the + blanking pass runs at all. A pass that resumed at `at + reach` rather than + past the `<` had nothing to reach into and looped for ever on this. + """ + def dangling_bracket_pdf(dir), + do: write(dir, "dangling.pdf", pdf(:plain, [{6, "<< /Note (/Launch) >>"}]) <> "\n<") + @doc """ A PDF carrying another file on a `/FileAttachment` annotation rather than in the `/EmbeddedFiles` name tree — with `typed: false`, without the @@ -385,6 +414,7 @@ defmodule Vutuv.AttachmentFixtures do defp page_extra(kind) when kind in [ :js_annotation, + :js_field_calculate, :file_attachment, :file_attachment_untyped, :web_skills_cv @@ -397,18 +427,19 @@ defmodule Vutuv.AttachmentFixtures do # page it is, so a rendered preview can be told from its neighbours. Object # numbers: 1 catalog, 2 the page tree, 3 the shared font, then a page and a # content object per page. - defp multi_page(count) do + defp multi_page(count, script_on \\ nil) do pages = for index <- 0..(count - 1), do: {4 + index * 2, 5 + index * 2} kids = Enum.map_join(pages, " ", fn {page, _content} -> "#{page} 0 R" end) page_objects = Enum.flat_map(Enum.with_index(pages, 1), fn {{page, content}, number} -> text = "BT /F1 96 Tf 72 400 Td (#{number}) Tj ET" + opens = if number == script_on, do: "/AA << /O #{@js_action} >> ", else: "" [ {page, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents #{content} 0 R " <> - "/Resources << /Font << /F1 3 0 R >> >> >>"}, + opens <> "/Resources << /Font << /F1 3 0 R >> >> >>"}, {content, "<< /Length #{byte_size(text)} >>\nstream\n#{text}\nendstream"} ] end) @@ -470,6 +501,29 @@ defmodule Vutuv.AttachmentFixtures do [{6, "<< /Type /Annot /Subtype /Link /Rect [0 0 10 10] /A #{@js_action} >>"}]} end + # A destination that satisfies `destination?/1`, with the script chained + # behind it. `pdfinfo` answers `JavaScript: no` for both of these, page range + # or not (measured 2026-09-11). + defp catalog(:js_next_dict) do + {"<< /Type /Catalog /Pages 2 0 R " <> + "/OpenAction << /S /GoTo /D [3 0 R /Fit] /Next #{@js_action} >> >>", []} + end + + defp catalog(:js_next_array) do + {"<< /Type /Catalog /Pages 2 0 R " <> + "/OpenAction << /S /GoTo /D [3 0 R /Fit] /Next [#{@js_action}] >> >>", []} + end + + # A form field that recalculates itself with a script. + defp catalog(:js_field_calculate) do + {"<< /Type /Catalog /Pages 2 0 R /AcroForm << /Fields [6 0 R] >> >>", + [ + {6, + "<< /Type /Annot /Subtype /Widget /FT /Tx /T (total) /Rect [0 0 10 10] " <> + "/AA << /C #{@js_action} >> >>"} + ]} + end + defp catalog(:associated_file) do {"<< /Type /Catalog /Pages 2 0 R /AF [6 0 R] >>", [ diff --git a/test/vutuv/attachments_test.exs b/test/vutuv/attachments_test.exs index 40564eef9..fa82eb994 100644 --- a/test/vutuv/attachments_test.exs +++ b/test/vutuv/attachments_test.exs @@ -194,15 +194,31 @@ defmodule Vutuv.AttachmentsTest do assert {:error, :embedded_files} = upload(user, Fixtures.embedded_file_pdf(files)) end - # Since #2136 the byte scan does not look for `/JavaScript` at all, because - # a scan that reads a page's words cannot tell a name from a sentence. Each - # of these four rests on `pdfinfo`'s answer alone, so this is where that - # rests. Calibration: take the `JavaScript:` line out of `PdfGate.check/1` - # and all four go red — three with `{:ok, _}`, `:open_action` with its own - # reason. + # Seven places a script can hang, two of which `pdfinfo` does not report at + # all — a `/Next` chain behind a destination, as a dictionary or as an array + # — and one it reports only on some builds. What answers for those is + # `/JavaScript` in `@name_regexes`, which is why it is there beside poppler + # rather than instead of it (#2136). Calibration: take `"JavaScript"` back + # out of `@name_regexes` and the two `/Next` lines go red with `{:ok, _}`. + # + # Note which way that calibration does *not* run: disabling poppler's + # `JavaScript:` answer leaves all seven green, because every one of these + # fixtures also carries the name in bytes this can read. What poppler alone + # answers is the object-stream variant below, and only where qpdf can build + # the fixture. test "JavaScript is refused wherever the action hangs", %{user: user, files: files} do + wheres = [ + :open_action, + :catalog_aa, + :page_aa, + :annotation, + :next_dict, + :next_array, + :field_calculate + ] + got = - for where <- [:open_action, :catalog_aa, :page_aa, :annotation] do + for where <- wheres do {where, upload(user, Fixtures.action_javascript_pdf(files, where))} end @@ -210,6 +226,34 @@ defmodule Vutuv.AttachmentsTest do "the gate answered: #{inspect(got)}" end + # Bare `pdfinfo` reads page 1 and stops, so a one-page fixture proves + # nothing about a two-page CV. Calibration: drop `-f 1 -l 999999` from + # `pdfinfo/1` and all three go red — with `{:ok, _}` if `"JavaScript"` is + # also out of `@name_regexes`, which is the pair of changes that let five + # such files through on 2026-09-11. + test "a script on a page nobody looks at is still a script", %{user: user, files: files} do + got = + for {pages, on} <- [{3, 2}, {3, 3}, {20, 20}] do + {{pages, on}, upload(user, Fixtures.page_javascript_pdf(files, pages, on))} + end + + assert Enum.all?(got, &match?({_where, {:error, :javascript}}, &1)), + "the gate answered: #{inspect(got)}" + end + + # `pdfinfo` exiting 0 with nothing to say is not a clean bill: a + # positive-match test reads that silence as "no JavaScript here". `true(1)` + # is the cheapest poppler that lies. Calibration: accept any exit-0 output + # in `pdfinfo/1` and this goes red with `{:ok, _}` — the gate storing a file + # it never checked. + test "a poppler that answers nothing has not answered", %{user: user, files: files} do + Fixtures.put_config(pdfinfo: "/usr/bin/true") + Attachments.forget_capability() + on_exit(&Attachments.forget_capability/0) + + assert {:error, :unreadable} = upload(user, Fixtures.plain_pdf(files)) + end + # `/EmbeddedFile` is the other half of that decision and went the other way: # poppler counts a file on a `/FileAttachment` annotation even when nothing # names it as one, so the first two of these are its answer — and answers @@ -269,19 +313,35 @@ defmodule Vutuv.AttachmentsTest do # **90 million** and **321 million**, quadrupling with every doubling: 2.8 # seconds of a LiveView's own process for a 60 KB file, and days for one at # the 20 MB cap. + # A lone `<` at the end of the file leaves the hex-string branch nothing to + # reach into, so a resume computed from that reach lands back on the same + # byte. Calibration: resume at `at + reach` again and this goes red with + # `{:error, :unreadable}` — a 639-byte document the pass reads until its + # whole allowance is gone, and before that allowance existed, for ever. + test "a file that ends in a bracket still finishes", %{user: user, files: files} do + assert {:ok, _attachment} = upload(user, Fixtures.dangling_bracket_pdf(files)) + end + + # A comment that runs to the end of the buffer and holds a `<<` is the + # candidate this pass is most easily made to re-read, and a file only has to + # name `/Launch` once — in a string, harmlessly — to arm the pass at all. + # Blanking up to the dictionary and resuming there reads it once. + # + # Reductions rather than a clock, because the suite runs twenty cases at + # once (see `Vutuv.WorkCounter`). Calibrated both ways on 2026-09-11: as it + # stands 60 KB costs 77,683 reductions and the file is accepted, which is + # the right answer for it. Make `blank_upto_dictionary/6` in + # `Vutuv.Uploads.PdfGate` reject the range and resume at `at + 1` instead + # and the same file costs **321 million** and 2.8 seconds of a LiveView's + # own process, quadrupling with every doubling. test "a file cannot make the blanking pass quadratic", %{user: user, files: files} do - {small, _answer} = - WorkCounter.count_reductions(fn -> - upload(user, Fixtures.comment_flood_pdf(files, 30_000)) - end) - - {large, _answer} = + {work, answer} = WorkCounter.count_reductions(fn -> upload(user, Fixtures.comment_flood_pdf(files, 60_000)) end) - assert large < 5_000_000, - "60 KB of comments cost #{large} reductions, 30 KB cost #{small}" + assert match?({:ok, _attachment}, answer), "the gate answered: #{inspect(answer)}" + assert work < 20_000_000, "60 KB of comments cost #{work} reductions" end # The `/OpenAction` rule reads what follows the name and lets a destination