diff --git a/config/config.exs b/config/config.exs index 5aafa7d27..e053d4f1a 100644 --- a/config/config.exs +++ b/config/config.exs @@ -126,6 +126,16 @@ config :vutuv, :screenshot_blocklist, ["reddit.com", "heise.de"] # (config/runtime.exs). config :vutuv, :moderate_images, true config :vutuv, :ollama_url, "http://localhost:11434" +# How long the scanner may be unreachable before a post waiting on one of its +# verdicts stops telling its author that a check is in progress (issue #2149). +# The queue itself is unchanged — a service outage still retries for ever and +# still releases nothing — so this is a ceiling on the *sentence*, not on the +# post: the moment Ollama answers, the verdict lands and the post publishes +# itself. Half an hour, which is what a scanner that answers but cannot judge +# an image already spends before it gives up (5 tries, 120+240+480+960 s), and +# far enough above the 300 s retry pace that a blip is never named. Runtime +# override: AI_CHECK_STALL_SECONDS (config/runtime.exs). +config :vutuv, :ai_check_stall_seconds, 1_800 config :vutuv, :ollama_vision_model, "qwen3-vl:8b" # Whether a link-preview capture is also judged on whether it shows the PAGE diff --git a/config/runtime.exs b/config/runtime.exs index adbf9aa95..5c3cfad26 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -631,6 +631,14 @@ if config_env() == :prod do config :vutuv, :moderate_images, false end + # How long that outage may run before a post waiting on a verdict stops + # saying a check is in progress and says plainly that it cannot run + # (issue #2149). Nothing is refused or released at the ceiling; an + # installation whose GPU box is regularly away for an hour raises it. + if seconds = System.get_env("AI_CHECK_STALL_SECONDS") do + config :vutuv, :ai_check_stall_seconds, String.to_integer(seconds) + end + # The link-preview page check: judges each capture on whether it shows the # page or a consent/ad/login wall, and blocklists the site when it does not. # SCREENSHOT_PAGE_CHECK=false switches it off (an installation whose Ollama diff --git a/docs/ADMINS.md b/docs/ADMINS.md index f4f821657..b12b114e9 100644 --- a/docs/ADMINS.md +++ b/docs/ADMINS.md @@ -227,6 +227,7 @@ Everything else has a default (the vutuv.de production value): | `SAVED_SEARCHES_MAX_PER_MEMBER` | `10` | Most saved searches (with e-mail alerts) one member may store (anti-abuse). A member at the cap is asked to delete one first | | `GEO_COUNTRIES` | `DE,AT,CH` | Comma-separated ISO 3166-1 alpha-2 codes whose bundled GeoNames postal data is loaded for offline zip → coordinate resolution on job postings. To add a country, drop its GeoNames zip export (`download.geonames.org/export/zip/.zip` → extracted `.txt`, optionally gzipped to `.txt.gz`) into `priv/geo/` and add the code here. Fully offline — no outbound calls | | `IMAGE_MODERATION_ENABLED` | `true` | `false` turns AI image moderation off (images publish immediately, as before the feature). While enabled, **every** image — avatars, covers, post / job-posting / organization images and the automatic link and homepage screenshots — waits invisible to everyone but its owner until a local Ollama vision model approves it; an unsafe image is deleted on the spot and the owner notified. Fail-closed: with Ollama unreachable, new images queue up and are scanned automatically once it is back — nothing is ever auto-approved. Set `false` only on installations without Ollama | +| `AI_CHECK_STALL_SECONDS` | `1800` | How long the scanner may be unreachable before a post waiting on one of its verdicts stops telling its author that a check is in progress and says plainly that the check cannot run. It changes **words, not the queue**: nothing is refused, nothing is released early, the retry goes on at its usual five-minute pace, and the post publishes itself the moment a verdict lands. Raise it on an installation whose GPU box is routinely away for an hour, so a normal absence is not called a stall; the chip in the app bar stops counting such a post as work in flight either way | | `IMAGE_PIXELATION_WINDOW_SECONDS` | `3600` | How long a picture waiting for that verdict shows readers a **pixelated preview** of itself — a separately stored file reduced to 64 cells on its long edge, not the picture behind a blur filter, so what reaches a reader carries none of the detail. It keeps a post card whole while the scan runs, and the real picture replaces it live the moment the verdict lands. Past this window the card falls back to a grey "being checked" tile, so a derivative of an unvetted picture never sits on a public page indefinitely. `0` switches the pixelated preview off entirely, which is the strictest posture | | `OLLAMA_URL` | `http://localhost:11434` | Base URL of the Ollama instance every AI feature talks to (image scan, translations, tag merge assist, employment-reference analysis). May be a **comma-separated list** (`http://gpu-box:11434,http://second-gpu:11434,http://localhost:11434`), which is read two ways at once. For a single call it is a **priority list**: every instance but the last is tried with a 30 s budget and skipped on any failure, the last one is the patient fallback (120 s, covers a CPU cold load). For calls that overlap it is also a **pool**: the second one starts on the least busy instance, so a second GPU takes work rather than waiting for the first to break. Verdicts are identical either way — the list only buys speed | | `OLLAMA_CONCURRENCY` | all `OLLAMA_URL` entries but the last | How many instances at the head of the list are treated as workers — which is both how deep the pool goes and how many calls a background sweep may have in flight. The default holds the **last** entry back as the fallback of record, because that is typically the web server's own CPU Ollama: `http://gpu-box:11434,http://localhost:11434` is one worker (unchanged from before this existed), `http://gpu1:11434,http://gpu2:11434,http://localhost:11434` is two. Set it to the number of entries when the list is all GPUs and nothing is held in reserve, or higher when one box can genuinely run two calls at once | diff --git a/docs/architecture/attachments.md b/docs/architecture/attachments.md index fdb565228..d6db48714 100644 --- a/docs/architecture/attachments.md +++ b/docs/architecture/attachments.md @@ -158,6 +158,21 @@ the clip now one case of it. `Vutuv.Posts.Pending` owns the whole question: stops waiting, and the author is offered the text without the refused file or neither. Both are `phx-click` events, never links: each destroys state, and a state-destroying GET dies on a Back button or a link prefetch. +* **A check that cannot run is said out loud** (#2149). An unreachable scanner + is retried for ever by design — nothing may be released without a verdict — + so the wait itself has no ceiling, and the *sentence* used not to either: on + an installation whose Ollama was down the card said "our AI is checking 1 + picture" at ten minutes, at a day and at thirty days. `image_scans` now + records **since when** the scanner has been unreachable + (`service_failing_since`, stamped on the first service error and cleared the + moment Ollama answers at all — even to say it cannot judge that file), and + past `ImageScans.stall_after_seconds/0` (`AI_CHECK_STALL_SECONDS`, 1800 s) + the stage is `:stalled`: the card says the check cannot be reached, the + app-bar chip stops counting the row as work in flight, and the clip's own + "our AI is checking it" line steps aside. The queue is untouched, so an + outage is a delay and never a refused post — a blip is 15 times under the + ceiling and never named, and whenever the scanner returns the verdict lands + and the post publishes itself. * **Surviving a deploy.** The publish is claimed by a compare-and-set on `status`, and the claim writes `minted_post_id` — the id the post is about to get — so a slot killed between the insert and the bookkeeping is resumed by diff --git a/docs/architecture/images.md b/docs/architecture/images.md index 698e5d96c..8bcf39da0 100644 --- a/docs/architecture/images.md +++ b/docs/architecture/images.md @@ -1541,7 +1541,12 @@ The moving parts (all under `Vutuv.Moderation`): said what the model actually saw). Two error classes: `{:service, _}` (Ollama down — retry forever, fail-closed) vs `{:image, _}` (this file can't be judged — capped, then rejected; an unverifiable image is never - released). + released). A service error also stamps `service_failing_since` on the scan + row, kept from the first failure of the run and cleared as soon as Ollama + answers at all; `ImageScans.stalled_subjects/1` reads it so whoever is + *waiting* on a verdict (a post with a file, #2149) can stop claiming a check + is in progress after `stall_after_seconds/0`. It bounds the sentence, never + the queue: nothing is released or refused when it passes. - `ImageScanWorker` — boot-resume + poll + nudge, mirroring `Vutuv.Posts.ScreenshotWorker`; hourly `repair_drift/0` re-enqueues any asset stranded in `pending`, and `ImageSubjects.settle_stranded_quarantine/0` diff --git a/lib/vutuv/moderation/image_scan.ex b/lib/vutuv/moderation/image_scan.ex index b4b405df8..38e18495d 100644 --- a/lib/vutuv/moderation/image_scan.ex +++ b/lib/vutuv/moderation/image_scan.ex @@ -50,6 +50,12 @@ defmodule Vutuv.Moderation.ImageScan do field(:attempts, :integer, default: 0) field(:next_attempt_at, :utc_datetime) field(:last_error, :string) + # Since when the *scanner* — not this image — has been failing, unbroken. + # Set on the first service error and cleared the moment Ollama answers at + # all, even to say it cannot judge this file, so it measures one outage + # rather than a lifetime of them. `Vutuv.Posts.Pending` reads it to stop + # telling an author a check is in progress when it is not (issue #2149). + field(:service_failing_since, :utc_datetime) field(:category, :string) # The model's own one-line description of the image. For a rejection it is # the only surviving record of what was deleted (the files are gone), and diff --git a/lib/vutuv/moderation/image_scans.ex b/lib/vutuv/moderation/image_scans.ex index 9cc9b622e..86fc06d4e 100644 --- a/lib/vutuv/moderation/image_scans.ex +++ b/lib/vutuv/moderation/image_scans.ex @@ -23,7 +23,11 @@ defmodule Vutuv.Moderation.ImageScans do scan (the backstop for a crash between verdict and application — and for any future upload path that forgets to enqueue, since the gallery tables default their `moderation` column to `pending`). When Ollama is down the - queue retries forever; nothing is ever auto-approved. With + queue retries forever; nothing is ever auto-approved. The row does record + **since when** it has been unreachable (`service_failing_since`), which is + what lets a member waiting on that verdict be told the check cannot run + instead of that it is running — see `stall_after_seconds/0` and issue #2149. + With `:moderate_images` off (tests, installations without Ollama) assets are created `approved` and this module is dormant. """ @@ -50,8 +54,12 @@ defmodule Vutuv.Moderation.ImageScans do @image_error_cap 5 # Service failures (Ollama down/unreachable) retry forever at this pace. @service_retry_seconds 300 + # How long such an outage may run before whoever waits on the verdict is told + # plainly that the check cannot run (issue #2149). See `stall_after_seconds/0`. + @default_stall_seconds 1800 @kinds ImageScan.kinds() + @open_statuses ImageScan.open_statuses() # The member personally chose these images; a rejection deletes their # content, so they get the notice. Machine captures (link screenshots) are # our artifact of a third-party page — silently showing no preview is the @@ -71,6 +79,47 @@ defmodule Vutuv.Moderation.ImageScans do @doc "Whether AI image moderation is enabled on this installation." def enabled?, do: Application.get_env(:vutuv, :moderate_images, true) + @doc """ + How long the scanner may be unreachable before whoever is *waiting* on one of + its verdicts should stop being told a check is in progress (issue #2149). + + Not a cap on the queue: a service outage still retries at + `@service_retry_seconds` for ever, and nothing is released or refused when + this passes. It is the point at which the wait stops being a wait and becomes + an open question for the member, and the config key is there because an + installation whose GPU box is regularly away for an hour should not call that + a stall. + """ + def stall_after_seconds, + do: Application.get_env(:vutuv, :ai_check_stall_seconds, @default_stall_seconds) + + @doc """ + Of these scan subjects, the ones whose check cannot run: an **open** scan + that has been failing against the service — not against the picture — for + longer than `stall_after_seconds/0`, unbroken. + + Answered for a whole batch at once, because the readers are page-shaped: one + waiting card per post, and a member may have several. A subject with no open + scan, or one whose last attempt reached Ollama at all, is simply not in the + set — `service_failing_since` is cleared the moment the scanner answers, even + to say it cannot judge that file, so a stamp this old means every retry since + failed to reach it. + """ + def stalled_subjects([]), do: MapSet.new() + + def stalled_subjects(subject_ids) when is_list(subject_ids) do + cutoff = DateTime.add(DateTime.utc_now(:second), -stall_after_seconds(), :second) + + from(s in ImageScan, + where: + s.subject_id in ^subject_ids and s.status in ^@open_statuses and + not is_nil(s.service_failing_since) and s.service_failing_since < ^cutoff, + select: s.subject_id + ) + |> Repo.all() + |> MapSet.new() + end + @doc """ The moderation state a freshly stored image starts in: `"pending"` (limbo) when moderation is enabled, `"approved"` otherwise. @@ -125,6 +174,7 @@ defmodule Vutuv.Moderation.ImageScans do attempts: 0, next_attempt_at: nil, last_error: nil, + service_failing_since: nil, updated_at: now ] ], @@ -179,8 +229,23 @@ defmodule Vutuv.Moderation.ImageScans do cutoff = NaiveDateTime.add(NaiveDateTime.utc_now(), -@stuck_after_seconds, :second) {count, _} = - from(s in ImageScan, where: s.status == "scanning" and s.updated_at < ^cutoff) - |> Repo.update_all(set: [status: "pending", updated_at: NaiveDateTime.utc_now(:second)]) + from(s in ImageScan, + where: s.status == "scanning" and s.updated_at < ^cutoff, + update: [ + set: [ + status: "pending", + updated_at: ^NaiveDateTime.utc_now(:second), + # A claim this old means the scanner never answered — an Ollama + # that accepts the connection and then hangs leaves no service + # error to stamp, and whoever waits on this verdict would never + # learn the check is not running (issue #2149). `coalesce` keeps an + # outage already being measured. + service_failing_since: + fragment("coalesce(?, ?)", s.service_failing_since, ^DateTime.utc_now(:second)) + ] + ] + ) + |> Repo.update_all([]) count end @@ -397,7 +462,12 @@ defmodule Vutuv.Moderation.ImageScans do update_claimed(scan, status: "pending", next_attempt_at: retry_at, - last_error: error_string(reason) + last_error: error_string(reason), + # Since when this outage has been running, kept from the first failure of + # the run rather than rewritten each time: it is what lets a waiting post + # tell a two-minute blip from a host whose scanner has been gone for a + # month (issue #2149). Cleared the moment Ollama answers at all. + service_failing_since: scan.service_failing_since || DateTime.utc_now(:second) ) :ok @@ -433,7 +503,10 @@ defmodule Vutuv.Moderation.ImageScans do status: "pending", attempts: attempts, next_attempt_at: retry_at, - last_error: error_string(reason) + last_error: error_string(reason), + # The scanner answered — it just could not judge this file. Whatever + # outage was running is over, and the next one starts its own clock. + service_failing_since: nil ) :ok @@ -538,7 +611,7 @@ defmodule Vutuv.Moderation.ImageScans do should be. A reader's own translation request is never stood down. """ def busy? do - Repo.exists?(from(s in ImageScan, where: s.status in ^ImageScan.open_statuses())) + Repo.exists?(from(s in ImageScan, where: s.status in ^@open_statuses)) end @doc "Queue totals for the admin dashboard: %{pending: n, rejected_7d: n}." @@ -546,7 +619,7 @@ defmodule Vutuv.Moderation.ImageScans do week_ago = DateTime.add(DateTime.utc_now(), -7 * 86_400, :second) pending = - Repo.aggregate(from(s in ImageScan, where: s.status in ^ImageScan.open_statuses()), :count) + Repo.aggregate(from(s in ImageScan, where: s.status in ^@open_statuses), :count) rejected = Repo.aggregate( diff --git a/lib/vutuv/posts/pending.ex b/lib/vutuv/posts/pending.ex index 3e1e6a605..6b52c0a9d 100644 --- a/lib/vutuv/posts/pending.ex +++ b/lib/vutuv/posts/pending.ex @@ -44,6 +44,21 @@ defmodule Vutuv.Posts.Pending do tell "the post is already there" from "it never happened" instead of writing the member's post twice. + ## When the check cannot run + + A file's preview pages, and a clip's frames, wait on the AI scanner, and an + unreachable scanner is retried for ever by design — nothing may be released + without a verdict. So the wait itself has no ceiling, and until #2149 neither + did the sentence: a post on an installation whose Ollama was down said "our + AI is checking 1 picture" at ten minutes, at a day and at thirty days. The + **pipeline is unchanged**; what has a ceiling is the claim. Past + `Vutuv.Moderation.ImageScans.stall_after_seconds/0` of unbroken + service failure the stage is `:stalled`, the app bar stops counting the row + as work in flight, and the author is told plainly. Nothing is refused and + nothing is dropped, so a blip cannot cost a post: the stamp the ceiling reads + is cleared the moment the scanner answers, and the verdict then publishes the + post by itself. + ## The clock advances on every outcome `due/1` is oldest-clock-first, and `sweep/1` stamps `checked_at` on every row @@ -61,6 +76,7 @@ defmodule Vutuv.Posts.Pending do alias Vutuv.Attachments.Attachment alias Vutuv.Attachments.Pages alias Vutuv.Images.Image + alias Vutuv.Moderation.ImageScans alias Vutuv.Posts.PendingPost alias Vutuv.Posts.Post alias Vutuv.Posts.PostVideo @@ -239,7 +255,9 @@ defmodule Vutuv.Posts.Pending do A row whose medium was refused is waiting for its *author*, not for us, so it is not counted: the chip is on every page and would otherwise sit there - amber for ever while the card for that same row says it was refused. + amber for ever while the card for that same row says it was refused. A + `:stalled` row is out for the same reason and a stronger one — nothing is + working on it at all (issue #2149). """ def in_progress_summary(user_id) when is_binary(user_id) do rows = waiting_for(user_id) @@ -271,10 +289,12 @@ defmodule Vutuv.Posts.Pending do `stage` is where the pipeline is, as data a surface turns into a sentence — `{:video, video}`, `{:rendering, done, total}`, `{:checking, count}`, - `:refused` or `:ready`. `state` is that classified for the publisher: - `:ready`, `:refused` (it can never become ready, so the author has to - choose) or `:working`. **Derived from the stage rather than read again**, so - a new medium or a new stage word is one edit, not three. + `:stalled`, `:refused` or `:ready`. `state` is that classified for the + publisher: `:ready`, `:refused` (it can never become ready, so the author has + to choose), `:stalled` (the scanner has been unreachable past the ceiling, so + nobody can say when it will) or `:working`. **Derived from the stage rather + than read again**, so a new medium or a new stage word is one edit, not + three. Surfaces take the whole map and pass it down; nothing asks twice. """ @@ -288,11 +308,17 @@ defmodule Vutuv.Posts.Pending do """ def readings(pendings) when is_list(pendings) do pendings = preload_media(pendings) - ids = for pending <- pendings, file <- pending.attachments, do: file.id - checking = page_counts(ids, "pending") - rendered = page_counts(ids, :any) + pages = page_rows(for pending <- pendings, file <- pending.attachments, do: file.id) + awaiting = awaiting_verdict(pages) + + batch = %{ + awaiting: awaiting, + rendered: page_counts(pages), + stalled: + ImageScans.stalled_subjects(Enum.flat_map(pendings, &awaited_verdicts(&1, awaiting))) + } - Map.new(pendings, &{&1.id, read_one(&1, checking, rendered)}) + Map.new(pendings, &{&1.id, read_one(&1, batch)}) end @doc "Whether this row can be published — `reading/1`'s `state`." @@ -304,10 +330,10 @@ defmodule Vutuv.Posts.Pending do @doc "The files this row is waiting on, in upload order." def attachments(%PendingPost{} = pending), do: reading(pending).files - defp read_one(%PendingPost{} = pending, checking, rendered) do + defp read_one(%PendingPost{} = pending, batch) do files = Enum.sort_by(pending.attachments, & &1.inserted_at, NaiveDateTime) - file_states = Map.new(files, &{&1.id, file_state(&1, checking)}) - stage = stage_of(pending.video, files, file_states, checking, rendered) + file_states = Map.new(files, &{&1.id, waiting_file_state(&1, batch)}) + stage = stage_of(pending.video, files, file_states, batch) %{ stage: stage, @@ -330,7 +356,11 @@ defmodule Vutuv.Posts.Pending do video = video_state(pending.video) states = Map.values(file_states) - survivors_settled? = video in [:done, :refused] and Enum.all?(states, &(&1 != :working)) + # `:done` and `:refused` are the only settled answers — a file still being + # worked on, or one whose check cannot be reached, would be carried into + # the insert and roll it back. + survivors_settled? = + video in [:done, :refused] and Enum.all?(states, &(&1 in [:done, :refused])) # `video_state/1` answers `:done` for a post that has no clip at all, so # the clip only counts as content when there really is one. @@ -341,21 +371,60 @@ defmodule Vutuv.Posts.Pending do survivors_settled? and keeps_something? end + defp stage_of(video, files, file_states, batch), + do: video |> pipeline_stage(files, file_states, batch) |> unreachable_check(batch) + # The clip comes first when there is one, because it is the slowest and the - # only stage that can name a percent. - defp stage_of(video, files, file_states, checking, rendered) do + # only stage that can name a percent. `:stalled` sits exactly where the files + # would otherwise be reported as being checked — a file carries its own + # stalled state — and never above a refusal: a decided medium stays decided. + defp pipeline_stage(video, files, file_states, batch) do video_state = video_state(video) - still_checking = files |> Enum.map(&Map.get(checking, &1.id, 0)) |> Enum.sum() + states = Map.values(file_states) + still_checking = files |> Enum.map(&length(awaiting(batch, &1))) |> Enum.sum() cond do - video_state == :refused or :refused in Map.values(file_states) -> :refused + video_state == :refused or :refused in states -> :refused video_state == :working -> {:video, video} - Enum.any?(files, &(&1.stage in ~w(stored rendering))) -> rendering(files, rendered) + Enum.any?(files, &(&1.stage in ~w(stored rendering))) -> rendering(files, batch.rendered) + :stalled in states -> :stalled still_checking > 0 -> {:checking, still_checking} true -> :ready end end + # The clip's half of the same rule, and the one stage that has to be caught + # from outside: a clip at `checking` outranks every file above, so its + # frames' stall would otherwise be reported as "our AI is checking it" + # (issue #2149). A clip still being converted is waiting for ffmpeg, not for + # Ollama, and is left alone. + defp unreachable_check({:video, %PostVideo{stage: "checking"} = video}, batch) do + if stalled?(frame_verdicts(video), batch.stalled), do: :stalled, else: {:video, video} + end + + defp unreachable_check(stage, _batch), do: stage + + # Every picture the AI check still owes this row a verdict on: the preview + # pages of its files that no verdict has reached, and — only while the clip + # is at `checking`, since a clip being converted is waiting for ffmpeg and + # not for Ollama — that clip's frames. + defp awaited_verdicts(%PendingPost{} = pending, awaiting) do + frame_verdicts(pending.video) ++ + Enum.flat_map(pending.attachments, &Map.get(awaiting, &1.id, [])) + end + + defp frame_verdicts(%PostVideo{stage: "checking", frames: frames}) when is_list(frames), + do: Enum.map(frames, & &1.id) + + defp frame_verdicts(_video), do: [] + + # One unreachable-scanner stamp older than the ceiling is enough. If the + # service were back, the 300-second retry would have cleared every one of + # these stamps long before the ceiling — so a single survivor is the outage, + # not a page that got unlucky. + defp stalled?(subject_ids, stalled), + do: Enum.any?(subject_ids, &MapSet.member?(stalled, &1)) + defp rendering(files, rendered), do: {:rendering, page_sum(rendered, files), wanted_pages(files)} @@ -363,23 +432,34 @@ defmodule Vutuv.Posts.Pending do do: files |> Enum.map(&Map.get(counts, &1.id, 0)) |> Enum.sum() defp state_of(:refused), do: :refused + defp state_of(:stalled), do: :stalled defp state_of(:ready), do: :ready defp state_of(_working), do: :working - defp page_counts([], _moderation), do: %{} + # Every preview page of these files, in **one** query: which of them a + # verdict has not reached (the ids, because the ids are what the scan queue + # is asked about) and how many exist at all ("page 2 of 3"). Five pages per + # file at most, so carrying both answers back is cheaper than a second round + # trip over the same rows. + defp page_rows([]), do: [] - defp page_counts(ids, moderation) do + defp page_rows(ids) do Image |> where([i], i.kind == ^Pages.kind() and i.attachment_id in ^ids) - |> page_moderation(moderation) - |> group_by([i], i.attachment_id) - |> select([i], {i.attachment_id, count(i.id)}) + |> select([i], {i.attachment_id, i.id, i.moderation}) |> Repo.all() - |> Map.new() end - defp page_moderation(query, :any), do: query - defp page_moderation(query, moderation), do: where(query, [i], i.moderation == ^moderation) + defp awaiting_verdict(pages) do + for {file_id, page_id, "pending"} <- pages, reduce: %{} do + acc -> Map.update(acc, file_id, [page_id], &[page_id | &1]) + end + end + + defp page_counts(pages), + do: Enum.frequencies_by(pages, fn {file_id, _id, _state} -> file_id end) + + defp awaiting(batch, %Attachment{id: id}), do: Map.get(batch.awaiting, id, []) # How many pages the renders are aiming for, so "page 2 of 3" can be said. defp wanted_pages(files), do: files |> Enum.map(&Pages.wanted_count/1) |> Enum.sum() @@ -400,7 +480,7 @@ defmodule Vutuv.Posts.Pending do publisher cannot each answer it differently — a chip reading "ready" beside a post that then parks is exactly the confusion this whole issue is about. """ - def file_state(%Attachment{} = file), do: file_state(file, page_counts([file.id], "pending")) + def file_state(%Attachment{} = file), do: Map.fetch!(file_states([file]), file.id) @doc """ The same for a list, as `%{attachment_id => state}` and in **one** query — a @@ -408,20 +488,33 @@ defmodule Vutuv.Posts.Pending do single form would run a query each. """ def file_states(files) when is_list(files) do - checking = page_counts(Enum.map(files, & &1.id), "pending") + batch = %{awaiting: awaiting_verdict(page_rows(Enum.map(files, & &1.id)))} - Map.new(files, &{&1.id, file_state(&1, checking)}) + Map.new(files, &{&1.id, file_state(&1, batch)}) end - defp file_state(%Attachment{stage: stage} = file, checking) do + defp file_state(%Attachment{stage: stage} = file, batch) do cond do Attachment.refused?(file) -> :refused stage in ~w(stored rendering) -> :working - Map.get(checking, file.id, 0) > 0 -> :working + awaiting(batch, file) != [] -> :working true -> :done end end + # The same, plus the fourth answer only a waiting row asks for: a file whose + # pages nobody can get a verdict on. It is deliberately **not** in + # `file_state/1` — that one answers the *gate's* question ("may this be + # shown?"), where a stalled file and a working one are the same closed door, + # and giving it a fourth value would put a scan-queue lookup on every file + # read. This one answers the *author's* ("is anybody working on it?"). + defp waiting_file_state(%Attachment{} = file, batch) do + case file_state(file, batch) do + :working -> if stalled?(awaiting(batch, file), batch.stalled), do: :stalled, else: :working + settled -> settled + end + end + @doc """ Whether these files are all finished — rendered, and every preview page past the AI check. What the composer asks before deciding whether the post can go diff --git a/lib/vutuv_web/components/pending_post_components.ex b/lib/vutuv_web/components/pending_post_components.ex index a217a13e9..11b920422 100644 --- a/lib/vutuv_web/components/pending_post_components.ex +++ b/lib/vutuv_web/components/pending_post_components.ex @@ -55,6 +55,18 @@ defmodule VutuvWeb.PendingPostComponents do end def stage_text(:refused), do: gettext("Something in this post was refused.") + + # Said in the author's terms, not the operator's: what they can act on is + # that the wait now has no end anybody can name, and that nothing of theirs + # was lost or refused. No duration in it — the ceiling is ours, the outage is + # somebody's server, and neither number means anything here. + def stage_text(:stalled) do + gettext( + "Our AI check cannot be reached at the moment. Your post is kept here and " <> + "goes out by itself as soon as the check runs again." + ) + end + def stage_text(:ready), do: gettext("Publishing it now") @doc "The stage as a live region, for a surface that shows one post." @@ -113,20 +125,31 @@ defmodule VutuvWeb.PendingPostComponents do """ def file_label(:refused), do: gettext("refused") def file_label(:working), do: gettext("being prepared") + # Never "being prepared": nothing is preparing it, and an amber word saying + # so sat directly under a headline saying the check cannot run (issue #2149). + def file_label(:stalled), do: gettext("waiting for the check") def file_label(:done), do: gettext("ready") @doc "The colour that word takes." def file_tone(:refused), do: "text-red-700 dark:text-red-300" def file_tone(:working), do: "text-amber-700 dark:text-amber-300" + def file_tone(:stalled), do: "text-slate-600 dark:text-slate-300" def file_tone(:done), do: "text-slate-500 dark:text-slate-400" # Its own msgid per subject, never one shared phrase: a post waiting on a # clip and a post waiting on files are two different sentences, and the # German for the clip one says "Video". - defp waiting_headline(true, _files), do: gettext("This post is still waiting for you") - defp waiting_headline(false, []), do: gettext("Your post appears as soon as the video is ready") + defp waiting_headline(:refused, _files), do: gettext("This post is still waiting for you") - defp waiting_headline(false, _files), + # A stalled row must not promise "as soon as its files are ready": nothing is + # working on them, and that promise is the whole complaint in issue #2149. + defp waiting_headline(:stalled, _files), + do: gettext("This post is waiting for a check that cannot run") + + defp waiting_headline(_working, []), + do: gettext("Your post appears as soon as the video is ready") + + defp waiting_headline(_working, _files), do: gettext("Your post appears as soon as its files are ready") ## The waiting card @@ -136,6 +159,12 @@ defmodule VutuvWeb.PendingPostComponents do tile when there is one, a line per file, the stage, and a way out — cancel while it works, publish without what was refused or drop it once something was. The host handles the two events. + + Three shapes, and `data-pending-status` names which: `working` (the server is + on it), `refused` (a verdict went against a medium, so the choice is the + author's) and `stalled` (the AI check has been unreachable past its ceiling, + issue #2149 — nobody is working on it and nobody can say when they will, so + the card stops claiming a stage and says that instead). """ attr(:pending, PendingPost, required: true) attr(:body_html, :any, required: true, doc: "the rendered text") @@ -154,18 +183,20 @@ defmodule VutuvWeb.PendingPostComponents do |> assign(:files, reading.files) |> assign(:file_states, reading.file_states) |> assign(:stage, reading.stage) - |> assign(:refused?, reading.state == :refused) + |> assign(:state, reading.state) |> assign(:leftover?, reading.publishable_without_refused?) ~H""" - <.card - class="mt-3" - data-pending-post={@pending.id} - data-pending-status={(@refused? && "refused") || "working"} - > + <.card class="mt-3" data-pending-post={@pending.id} data-pending-status={status_key(@state)}>

- <.hourglass :if={!@refused?} class="h-4 w-4 text-amber-600 dark:text-amber-400" /> - {waiting_headline(@refused?, @files)} + <%!-- The hourglass means "we are on it". A refused row is waiting for + its author and a stalled one for a scanner nobody can reach, so neither + gets one. --%> + <.hourglass + :if={@state not in [:refused, :stalled]} + class="h-4 w-4 text-amber-600 dark:text-amber-400" + /> + {waiting_headline(@state, @files)}

{@body_html} @@ -173,7 +204,14 @@ defmodule VutuvWeb.PendingPostComponents do
<.video_tile video={@video} class="w-full sm:w-64 sm:shrink-0" />
- <.stage_line video={@video} class="text-sm text-slate-700 dark:text-slate-200" /> + <%!-- The clip's own line says "our AI is checking it" while it is at + `checking`, which is the one sentence a stalled card must not carry + twice over. The card's stage line below says what is really true. --%> + <.stage_line + :if={@state != :stalled} + video={@video} + class="text-sm text-slate-700 dark:text-slate-200" + />

{gettext("About %{minutes} min of video", minutes: minutes_up(@video))}

@@ -194,7 +232,7 @@ defmodule VutuvWeb.PendingPostComponents do />
<.button - :if={@refused? and @leftover?} + :if={@state == :refused and @leftover?} type="button" phx-click="publish-without-refused" phx-value-id={@pending.id} @@ -208,10 +246,18 @@ defmodule VutuvWeb.PendingPostComponents do phx-click={JS.push("cancel-pending-post", value: %{id: @pending.id})} data-cancel-pending-post > - {if @refused?, do: gettext("Delete this post"), else: gettext("Cancel")} + {if @state == :refused, do: gettext("Delete this post"), else: gettext("Cancel")}
""" end + + # What the card is, for a test and for anybody reading the DOM: `working` + # covers every stage the server is actually on, and the two that are not are + # named apart — a refused medium is the author's decision to make, a stalled + # check is nobody's yet. + defp status_key(:refused), do: "refused" + defp status_key(:stalled), do: "stalled" + defp status_key(_working), do: "working" end diff --git a/priv/gettext/de/LC_MESSAGES/default.po b/priv/gettext/de/LC_MESSAGES/default.po index 9b355f8c4..bcda35d2d 100644 --- a/priv/gettext/de/LC_MESSAGES/default.po +++ b/priv/gettext/de/LC_MESSAGES/default.po @@ -109,7 +109,7 @@ msgstr "Geburtstag" msgid "Birthday" msgstr "Geburtstag" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #: lib/vutuv_web/components/saved_search_components.ex:104 #: lib/vutuv_web/components/ui.ex:4935 #: lib/vutuv_web/live/admin/newsletter_broadcast_live.ex:210 @@ -22873,7 +22873,7 @@ msgstr "bandbreite langsam internet mobil daten sparen datensparmodus volumen sc msgid "Appearance" msgstr "Darstellung" -#: lib/vutuv_web/components/pending_post_components.ex:178 +#: lib/vutuv_web/components/pending_post_components.ex:216 #, elixir-autogen, elixir-format msgid "About %{minutes} min of video" msgstr "Etwa %{minutes} Min. Video" @@ -22888,7 +22888,7 @@ msgstr "Video hinzufügen" msgid "Converting · %{percent} %" msgstr "Wird umgewandelt · %{percent} %" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #, elixir-autogen, elixir-format msgid "Delete this post" msgstr "Diesen Beitrag löschen" @@ -22950,7 +22950,7 @@ msgstr "Das Video konnte nicht angehängt werden." msgid "The video is longer than %{seconds} seconds." msgstr "Das Video ist länger als %{seconds} Sekunden." -#: lib/vutuv_web/components/pending_post_components.ex:126 +#: lib/vutuv_web/components/pending_post_components.ex:142 #, elixir-autogen, elixir-format msgid "This post is still waiting for you" msgstr "Dieser Beitrag wartet noch auf Sie" @@ -25156,12 +25156,12 @@ msgid_plural "Our AI is checking %{formatted} pictures." msgstr[0] "Unsere KI prüft gerade %{formatted} Bild." msgstr[1] "Unsere KI prüft gerade %{formatted} Bilder." -#: lib/vutuv_web/components/pending_post_components.ex:203 +#: lib/vutuv_web/components/pending_post_components.ex:241 #, elixir-autogen, elixir-format msgid "Post without it" msgstr "Trotzdem veröffentlichen" -#: lib/vutuv_web/components/pending_post_components.ex:58 +#: lib/vutuv_web/components/pending_post_components.ex:70 #, elixir-autogen, elixir-format msgid "Publishing it now" msgstr "Wird jetzt veröffentlicht" @@ -25181,7 +25181,7 @@ msgstr "Etwas an diesem Beitrag wurde abgelehnt." msgid "Waiting" msgstr "Wartet" -#: lib/vutuv_web/components/pending_post_components.ex:130 +#: lib/vutuv_web/components/pending_post_components.ex:153 #, elixir-autogen, elixir-format msgid "Your post appears as soon as its files are ready" msgstr "Ihr Beitrag erscheint, sobald seine Dateien fertig sind" @@ -25192,22 +25192,22 @@ msgstr "Ihr Beitrag erscheint, sobald seine Dateien fertig sind" msgid "Your uploads" msgstr "Ihre Uploads" -#: lib/vutuv_web/components/pending_post_components.ex:115 +#: lib/vutuv_web/components/pending_post_components.ex:127 #, elixir-autogen, elixir-format msgid "being prepared" msgstr "wird vorbereitet" -#: lib/vutuv_web/components/pending_post_components.ex:116 +#: lib/vutuv_web/components/pending_post_components.ex:131 #, elixir-autogen, elixir-format msgid "ready" msgstr "fertig" -#: lib/vutuv_web/components/pending_post_components.ex:114 +#: lib/vutuv_web/components/pending_post_components.ex:126 #, elixir-autogen, elixir-format msgid "refused" msgstr "abgelehnt" -#: lib/vutuv_web/components/pending_post_components.ex:127 +#: lib/vutuv_web/components/pending_post_components.ex:150 #, elixir-autogen, elixir-format msgid "Your post appears as soon as the video is ready" msgstr "Ihr Beitrag erscheint, sobald das Video fertig ist" @@ -25610,3 +25610,18 @@ msgstr "Dieses Logo enthält %{marker}, was vutuv als eingebettete Datei liest u #, elixir-autogen, elixir-format msgid "This logo has a file embedded in it, usually a web font, that vutuv cannot check. Export it again with the text converted to outlines." msgstr "Dieses Logo enthält eine eingebettete Datei, meist eine Web-Schrift, die vutuv nicht prüfen kann. Bitte exportieren Sie es erneut mit in Pfade umgewandelter Schrift." + +#: lib/vutuv_web/components/pending_post_components.ex:64 +#, elixir-autogen, elixir-format +msgid "Our AI check cannot be reached at the moment. Your post is kept here and goes out by itself as soon as the check runs again." +msgstr "Unsere KI-Prüfung ist zurzeit nicht erreichbar. Ihr Beitrag bleibt hier und erscheint von selbst, sobald die Prüfung wieder läuft." + +#: lib/vutuv_web/components/pending_post_components.ex:147 +#, elixir-autogen, elixir-format +msgid "This post is waiting for a check that cannot run" +msgstr "Dieser Beitrag wartet auf eine Prüfung, die nicht läuft" + +#: lib/vutuv_web/components/pending_post_components.ex:130 +#, elixir-autogen, elixir-format +msgid "waiting for the check" +msgstr "wartet auf die Prüfung" diff --git a/priv/gettext/default.pot b/priv/gettext/default.pot index c1f90944a..c12b2adb0 100644 --- a/priv/gettext/default.pot +++ b/priv/gettext/default.pot @@ -111,7 +111,7 @@ msgstr "" msgid "Birthday" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #: lib/vutuv_web/components/saved_search_components.ex:104 #: lib/vutuv_web/components/ui.ex:4935 #: lib/vutuv_web/live/admin/newsletter_broadcast_live.ex:210 @@ -22088,7 +22088,7 @@ msgstr "" msgid "Appearance" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:178 +#: lib/vutuv_web/components/pending_post_components.ex:216 #, elixir-autogen, elixir-format msgid "About %{minutes} min of video" msgstr "" @@ -22103,7 +22103,7 @@ msgstr "" msgid "Converting · %{percent} %" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #, elixir-autogen, elixir-format msgid "Delete this post" msgstr "" @@ -22165,7 +22165,7 @@ msgstr "" msgid "The video is longer than %{seconds} seconds." msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:126 +#: lib/vutuv_web/components/pending_post_components.ex:142 #, elixir-autogen, elixir-format msgid "This post is still waiting for you" msgstr "" @@ -24323,12 +24323,12 @@ msgid_plural "Our AI is checking %{formatted} pictures." msgstr[0] "" msgstr[1] "" -#: lib/vutuv_web/components/pending_post_components.ex:203 +#: lib/vutuv_web/components/pending_post_components.ex:241 #, elixir-autogen, elixir-format msgid "Post without it" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:58 +#: lib/vutuv_web/components/pending_post_components.ex:70 #, elixir-autogen, elixir-format msgid "Publishing it now" msgstr "" @@ -24348,7 +24348,7 @@ msgstr "" msgid "Waiting" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:130 +#: lib/vutuv_web/components/pending_post_components.ex:153 #, elixir-autogen, elixir-format msgid "Your post appears as soon as its files are ready" msgstr "" @@ -24359,22 +24359,22 @@ msgstr "" msgid "Your uploads" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:115 +#: lib/vutuv_web/components/pending_post_components.ex:127 #, elixir-autogen, elixir-format msgid "being prepared" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:116 +#: lib/vutuv_web/components/pending_post_components.ex:131 #, elixir-autogen, elixir-format msgid "ready" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:114 +#: lib/vutuv_web/components/pending_post_components.ex:126 #, elixir-autogen, elixir-format msgid "refused" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:127 +#: lib/vutuv_web/components/pending_post_components.ex:150 #, elixir-autogen, elixir-format msgid "Your post appears as soon as the video is ready" msgstr "" @@ -24777,3 +24777,18 @@ msgstr "" #, elixir-autogen, elixir-format msgid "This logo has a file embedded in it, usually a web font, that vutuv cannot check. Export it again with the text converted to outlines." msgstr "" + +#: lib/vutuv_web/components/pending_post_components.ex:64 +#, elixir-autogen, elixir-format +msgid "Our AI check cannot be reached at the moment. Your post is kept here and goes out by itself as soon as the check runs again." +msgstr "" + +#: lib/vutuv_web/components/pending_post_components.ex:147 +#, elixir-autogen, elixir-format +msgid "This post is waiting for a check that cannot run" +msgstr "" + +#: lib/vutuv_web/components/pending_post_components.ex:130 +#, elixir-autogen, elixir-format +msgid "waiting for the check" +msgstr "" diff --git a/priv/gettext/en/LC_MESSAGES/default.po b/priv/gettext/en/LC_MESSAGES/default.po index 93f5fdebf..7289e569e 100644 --- a/priv/gettext/en/LC_MESSAGES/default.po +++ b/priv/gettext/en/LC_MESSAGES/default.po @@ -109,7 +109,7 @@ msgstr "" msgid "Birthday" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #: lib/vutuv_web/components/saved_search_components.ex:104 #: lib/vutuv_web/components/ui.ex:4935 #: lib/vutuv_web/live/admin/newsletter_broadcast_live.ex:210 @@ -22086,7 +22086,7 @@ msgstr "" msgid "Appearance" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:178 +#: lib/vutuv_web/components/pending_post_components.ex:216 #, elixir-autogen, elixir-format msgid "About %{minutes} min of video" msgstr "" @@ -22101,7 +22101,7 @@ msgstr "" msgid "Converting · %{percent} %" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #, elixir-autogen, elixir-format, fuzzy msgid "Delete this post" msgstr "" @@ -22163,7 +22163,7 @@ msgstr "" msgid "The video is longer than %{seconds} seconds." msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:126 +#: lib/vutuv_web/components/pending_post_components.ex:142 #, elixir-autogen, elixir-format msgid "This post is still waiting for you" msgstr "" @@ -24321,12 +24321,12 @@ msgid_plural "Our AI is checking %{formatted} pictures." msgstr[0] "" msgstr[1] "" -#: lib/vutuv_web/components/pending_post_components.ex:203 +#: lib/vutuv_web/components/pending_post_components.ex:241 #, elixir-autogen, elixir-format, fuzzy msgid "Post without it" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:58 +#: lib/vutuv_web/components/pending_post_components.ex:70 #, elixir-autogen, elixir-format, fuzzy msgid "Publishing it now" msgstr "" @@ -24346,7 +24346,7 @@ msgstr "" msgid "Waiting" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:130 +#: lib/vutuv_web/components/pending_post_components.ex:153 #, elixir-autogen, elixir-format, fuzzy msgid "Your post appears as soon as its files are ready" msgstr "" @@ -24357,22 +24357,22 @@ msgstr "" msgid "Your uploads" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:115 +#: lib/vutuv_web/components/pending_post_components.ex:127 #, elixir-autogen, elixir-format msgid "being prepared" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:116 +#: lib/vutuv_web/components/pending_post_components.ex:131 #, elixir-autogen, elixir-format, fuzzy msgid "ready" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:114 +#: lib/vutuv_web/components/pending_post_components.ex:126 #, elixir-autogen, elixir-format, fuzzy msgid "refused" msgstr "" -#: lib/vutuv_web/components/pending_post_components.ex:127 +#: lib/vutuv_web/components/pending_post_components.ex:150 #, elixir-autogen, elixir-format, fuzzy msgid "Your post appears as soon as the video is ready" msgstr "" @@ -24775,3 +24775,18 @@ msgstr "" #, elixir-autogen, elixir-format msgid "This logo has a file embedded in it, usually a web font, that vutuv cannot check. Export it again with the text converted to outlines." msgstr "" + +#: lib/vutuv_web/components/pending_post_components.ex:64 +#, elixir-autogen, elixir-format +msgid "Our AI check cannot be reached at the moment. Your post is kept here and goes out by itself as soon as the check runs again." +msgstr "" + +#: lib/vutuv_web/components/pending_post_components.ex:147 +#, elixir-autogen, elixir-format +msgid "This post is waiting for a check that cannot run" +msgstr "" + +#: lib/vutuv_web/components/pending_post_components.ex:130 +#, elixir-autogen, elixir-format, fuzzy +msgid "waiting for the check" +msgstr "" diff --git a/priv/gettext/it/LC_MESSAGES/default.po b/priv/gettext/it/LC_MESSAGES/default.po index be14e4d3c..4abd69132 100644 --- a/priv/gettext/it/LC_MESSAGES/default.po +++ b/priv/gettext/it/LC_MESSAGES/default.po @@ -111,7 +111,7 @@ msgstr "Data di nascita" msgid "Birthday" msgstr "Compleanno" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #: lib/vutuv_web/components/saved_search_components.ex:104 #: lib/vutuv_web/components/ui.ex:4935 #: lib/vutuv_web/live/admin/newsletter_broadcast_live.ex:210 @@ -23552,7 +23552,7 @@ msgstr "banda lento internet mobile risparmio dati volume compressione immagini msgid "Appearance" msgstr "Aspetto" -#: lib/vutuv_web/components/pending_post_components.ex:178 +#: lib/vutuv_web/components/pending_post_components.ex:216 #, elixir-autogen, elixir-format msgid "About %{minutes} min of video" msgstr "Circa %{minutes} min di video" @@ -23567,7 +23567,7 @@ msgstr "Aggiungi video" msgid "Converting · %{percent} %" msgstr "Conversione · %{percent} %" -#: lib/vutuv_web/components/pending_post_components.ex:211 +#: lib/vutuv_web/components/pending_post_components.ex:249 #, elixir-autogen, elixir-format msgid "Delete this post" msgstr "Elimina questo post" @@ -23629,7 +23629,7 @@ msgstr "Il video non può essere allegato." msgid "The video is longer than %{seconds} seconds." msgstr "Il video dura più di %{seconds} secondi." -#: lib/vutuv_web/components/pending_post_components.ex:126 +#: lib/vutuv_web/components/pending_post_components.ex:142 #, elixir-autogen, elixir-format msgid "This post is still waiting for you" msgstr "Questo post La sta ancora aspettando" @@ -25834,12 +25834,12 @@ msgid_plural "Our AI is checking %{formatted} pictures." msgstr[0] "La nostra IA sta controllando %{formatted} immagine." msgstr[1] "La nostra IA sta controllando %{formatted} immagini." -#: lib/vutuv_web/components/pending_post_components.ex:203 +#: lib/vutuv_web/components/pending_post_components.ex:241 #, elixir-autogen, elixir-format msgid "Post without it" msgstr "Pubblica lo stesso" -#: lib/vutuv_web/components/pending_post_components.ex:58 +#: lib/vutuv_web/components/pending_post_components.ex:70 #, elixir-autogen, elixir-format msgid "Publishing it now" msgstr "Pubblicazione in corso" @@ -25859,7 +25859,7 @@ msgstr "Qualcosa in questo post è stato rifiutato." msgid "Waiting" msgstr "In attesa" -#: lib/vutuv_web/components/pending_post_components.ex:130 +#: lib/vutuv_web/components/pending_post_components.ex:153 #, elixir-autogen, elixir-format msgid "Your post appears as soon as its files are ready" msgstr "Il suo post apparirà non appena i file saranno pronti" @@ -25870,22 +25870,22 @@ msgstr "Il suo post apparirà non appena i file saranno pronti" msgid "Your uploads" msgstr "I suoi caricamenti" -#: lib/vutuv_web/components/pending_post_components.ex:115 +#: lib/vutuv_web/components/pending_post_components.ex:127 #, elixir-autogen, elixir-format msgid "being prepared" msgstr "in preparazione" -#: lib/vutuv_web/components/pending_post_components.ex:116 +#: lib/vutuv_web/components/pending_post_components.ex:131 #, elixir-autogen, elixir-format msgid "ready" msgstr "pronto" -#: lib/vutuv_web/components/pending_post_components.ex:114 +#: lib/vutuv_web/components/pending_post_components.ex:126 #, elixir-autogen, elixir-format msgid "refused" msgstr "rifiutato" -#: lib/vutuv_web/components/pending_post_components.ex:127 +#: lib/vutuv_web/components/pending_post_components.ex:150 #, elixir-autogen, elixir-format msgid "Your post appears as soon as the video is ready" msgstr "Il suo post apparirà non appena il video sarà pronto" @@ -26290,3 +26290,18 @@ msgstr "Questo logo contiene %{marker}, che vutuv legge come file incorporato e #, elixir-autogen, elixir-format msgid "This logo has a file embedded in it, usually a web font, that vutuv cannot check. Export it again with the text converted to outlines." msgstr "Questo logo contiene un file incorporato, di solito un font web, che vutuv non può verificare. Lo esporti di nuovo convertendo il testo in tracciati." + +#: lib/vutuv_web/components/pending_post_components.ex:64 +#, elixir-autogen, elixir-format +msgid "Our AI check cannot be reached at the moment. Your post is kept here and goes out by itself as soon as the check runs again." +msgstr "Al momento il nostro controllo IA non è raggiungibile. Il Suo post resta qui e viene pubblicato da sé non appena il controllo riprende." + +#: lib/vutuv_web/components/pending_post_components.ex:147 +#, elixir-autogen, elixir-format +msgid "This post is waiting for a check that cannot run" +msgstr "Questo post attende un controllo che non può essere eseguito" + +#: lib/vutuv_web/components/pending_post_components.ex:130 +#, elixir-autogen, elixir-format +msgid "waiting for the check" +msgstr "in attesa del controllo" diff --git a/priv/repo/migrations/20260911173549_add_service_failing_since_to_image_scans.exs b/priv/repo/migrations/20260911173549_add_service_failing_since_to_image_scans.exs new file mode 100644 index 000000000..b86a03e28 --- /dev/null +++ b/priv/repo/migrations/20260911173549_add_service_failing_since_to_image_scans.exs @@ -0,0 +1,18 @@ +defmodule Vutuv.Repo.Migrations.AddServiceFailingSinceToImageScans do + use Ecto.Migration + + # When the scanner itself is unreachable the queue retries forever by design + # — the image is fine, nothing may be auto-approved, and the outage ends when + # the operator's Ollama comes back. What was written nowhere is *since when*, + # so a post waiting on that verdict could not tell a two-minute blip from a + # host whose scanner has been down for a month, and told its author "our AI + # is checking 1 picture" either way (issue #2149). + # + # Nullable and additive: the release still serving traffic through the + # blue/green switch never reads or writes it. + def change do + alter table(:image_scans) do + add(:service_failing_since, :utc_datetime) + end + end +end diff --git a/test/support/attachment_helpers.ex b/test/support/attachment_helpers.ex index 00fa0a201..38dc99bfc 100644 --- a/test/support/attachment_helpers.ex +++ b/test/support/attachment_helpers.ex @@ -12,12 +12,17 @@ defmodule Vutuv.AttachmentHelpers do whatever the test is named after. """ + import Ecto.Query, only: [from: 2] import ExUnit.Assertions alias Vutuv.Attachments alias Vutuv.Attachments.Attachment alias Vutuv.Attachments.Pages + alias Vutuv.Images.Image + alias Vutuv.Moderation.ImageScan + alias Vutuv.Moderation.ImageScans alias Vutuv.Repo + alias Vutuv.UUIDv7 @doc """ Renders `attachment`'s pages and hands back the row the pipeline settled, @@ -49,6 +54,64 @@ defmodule Vutuv.AttachmentHelpers do assert_settled(attachment) end + @doc """ + One preview page for `attachment`, in the state a verdict would leave it — + written rather than rendered, so a test about the *wait* does not depend on + poppler or Chromium being on the machine running the suite. + """ + def page!(%Attachment{} = attachment, moderation) do + now = NaiveDateTime.utc_now(:second) + + Repo.insert!(%Image{ + id: UUIDv7.generate(), + kind: Pages.kind(), + attachment_id: attachment.id, + user_id: attachment.user_id, + token: Vutuv.Uploads.gen_token(), + position: 0, + moderation: moderation, + width: 1240, + height: 1667, + content_type: "image/avif", + size_bytes: 1234, + inserted_at: now, + updated_at: now + }) + end + + @doc """ + The queue row an unreachable Ollama leaves behind for one preview page: open, + retrying, and failing against the **service** rather than against the picture + — which is the distinction the stall ceiling is drawn on (issue #2149). + + `seconds` is how long that one outage has been running, written rather than + waited for, so the gate is decided by the row and never by the time of day + the suite runs at. Replaces any open scan on the page, so a test can age the + same outage twice. + """ + def stalled_scan!(%Image{} = page, seconds) do + Repo.delete_all( + from(s in ImageScan, + where: s.subject_id == ^page.id and s.status in ^ImageScan.open_statuses() + ) + ) + + now = DateTime.utc_now(:second) + + Repo.insert!(%ImageScan{ + kind: Pages.kind(), + subject_id: page.id, + owner_user_id: page.user_id, + status: "pending", + next_attempt_at: DateTime.add(now, 300, :second), + last_error: "econnrefused", + service_failing_since: DateTime.add(now, -seconds, :second) + }) + end + + @doc "How long an outage has to run before a waiting post stops claiming a check." + def stall_after_seconds, do: ImageScans.stall_after_seconds() + # The row is re-read rather than taken from `render/1`: that function writes # the stage and answers a struct saying so in one breath, so its return value # cannot corroborate itself. diff --git a/test/vutuv/moderation/image_scans_test.exs b/test/vutuv/moderation/image_scans_test.exs index c2065c588..c15c6c1be 100644 --- a/test/vutuv/moderation/image_scans_test.exs +++ b/test/vutuv/moderation/image_scans_test.exs @@ -13,6 +13,7 @@ defmodule Vutuv.Moderation.ImageScansTest do import Ecto.Query import ExUnit.CaptureLog import Vutuv.PostsHelpers + import Vutuv.WebPushHelpers, only: [put_config: 2] alias Vutuv.Accounts alias Vutuv.Moderation.ImageScan @@ -73,6 +74,22 @@ defmodule Vutuv.Moderation.ImageScansTest do defp reload(user), do: Repo.get!(Vutuv.Accounts.User, user.id) + # The backed-off row, due again — a retry without waiting out the pace. + defp due_now do + Repo.update_all(from(s in ImageScan, where: s.status == "pending"), + set: [next_attempt_at: nil] + ) + end + + # How long the outage on this scan has been running. Written rather than + # waited for, so the ceiling is decided by the row and never by the time of + # day the suite runs at. + defp outage!(%ImageScan{id: id}, seconds) do + Repo.update_all(from(s in ImageScan, where: s.id == ^id), + set: [service_failing_since: DateTime.add(DateTime.utc_now(:second), -seconds, :second)] + ) + end + defp open_scan(kind, subject_id) do Repo.one( from(s in ImageScan, @@ -375,6 +392,49 @@ defmodule Vutuv.Moderation.ImageScansTest do assert scan.attempts == 0 end + test "the outage is stamped once and cleared the moment Ollama answers at all", + %{user: user} do + user = upload_avatar(user) + + ImageScans.deliver_due(judge: fn _path -> {:error, {:service, :econnrefused}} end) + + since = open_scan("avatar", user.id).service_failing_since + + assert since, + "a service failure recorded nothing about when the outage started, " <> + "so nobody waiting on this verdict can tell a blip from a dead host" + + # A second failure measures the same outage rather than restarting it, + # or a queue retrying every five minutes would never look old. + due_now() + ImageScans.deliver_due(judge: fn _path -> {:error, {:service, :timeout}} end) + assert open_scan("avatar", user.id).service_failing_since == since + + # Ollama answered — it just could not judge this picture. That is the + # scanner working, so the outage is over and the next one starts fresh. + due_now() + ImageScans.deliver_due(judge: fn _path -> {:error, {:image, :bad_verdict}} end) + refute open_scan("avatar", user.id).service_failing_since + end + + test "only an outage past the ceiling counts as a check that cannot run", %{user: user} do + user = upload_avatar(user) + ImageScans.deliver_due(judge: fn _path -> {:error, {:service, :econnrefused}} end) + scan = open_scan("avatar", user.id) + + # A second under the half hour: the check is late, not absent. + outage!(scan, ImageScans.stall_after_seconds() - 1) + assert ImageScans.stalled_subjects([scan.subject_id]) == MapSet.new() + + outage!(scan, ImageScans.stall_after_seconds() + 1) + assert ImageScans.stalled_subjects([scan.subject_id]) == MapSet.new([scan.subject_id]) + + # The ceiling is the installation's: a GPU box that is regularly away for + # an hour is not a stall there. + put_config(:ai_check_stall_seconds, 24 * 3_600) + assert ImageScans.stalled_subjects([scan.subject_id]) == MapSet.new() + end + test "an unjudgeable image caps out into rejection, never release", %{user: user} do user = upload_avatar(user) scan = open_scan("avatar", user.id) diff --git a/test/vutuv/posts/pending_test.exs b/test/vutuv/posts/pending_test.exs index d62457958..c0ef9b7cf 100644 --- a/test/vutuv/posts/pending_test.exs +++ b/test/vutuv/posts/pending_test.exs @@ -19,11 +19,14 @@ defmodule Vutuv.Posts.PendingTest do import Vutuv.WebPushHelpers, only: [put_config: 2] + import Vutuv.AttachmentHelpers, only: [page!: 2, stalled_scan!: 2, stall_after_seconds: 0] + alias Ecto.Adapters.SQL.Sandbox alias Vutuv.AttachmentFixtures, as: Fixtures alias Vutuv.Attachments alias Vutuv.Attachments.Attachment alias Vutuv.Images.Image + alias Vutuv.Moderation.ImageScan alias Vutuv.Posts alias Vutuv.Posts.Pending alias Vutuv.Posts.PendingPost @@ -31,7 +34,6 @@ defmodule Vutuv.Posts.PendingTest do alias Vutuv.Posts.PostVideo alias Vutuv.Posts.Publisher alias Vutuv.Repo - alias Vutuv.UUIDv7 alias VutuvWeb.Live.PendingPostActions setup do @@ -48,29 +50,6 @@ defmodule Vutuv.Posts.PendingTest do attachment end - # One preview page for `attachment`, in the state a verdict would leave it — - # written rather than rendered, so the wait is measured without depending on - # poppler or Chromium being on the machine running the suite. - defp page!(%Attachment{} = attachment, moderation) do - now = NaiveDateTime.utc_now(:second) - - Repo.insert!(%Image{ - id: UUIDv7.generate(), - kind: "attachment_page", - attachment_id: attachment.id, - user_id: attachment.user_id, - token: Vutuv.Uploads.gen_token(), - position: 0, - moderation: moderation, - width: 1240, - height: 1667, - content_type: "image/avif", - size_bytes: 1234, - inserted_at: now, - updated_at: now - }) - end - defp rendered!(%Attachment{} = attachment) do Repo.update_all(from(a in Attachment, where: a.id == ^attachment.id), set: [stage: "ready", worked_at: nil] @@ -368,6 +347,73 @@ defmodule Vutuv.Posts.PendingTest do end end + describe "when the AI check cannot run" do + test "past the ceiling the post stops claiming a check is in progress", + %{user: user, files: files} do + attachment = file!(user, files) + + {:ok, pending} = + Pending.create(user, "post", %{}, %{body: "Read this"}, attachments: [attachment]) + + rendered!(attachment) + page = page!(attachment, "pending") + + # A blip is not a ceiling. Under it the check is merely late, the post is + # work the server is doing, and the app bar says so. + stalled_scan!(page, stall_after_seconds() - 60) + assert Pending.stage(Pending.get(user, pending.id)) == {:checking, 1} + assert Pending.in_progress_summary(user.id).count == 1 + + # Past it the wait is unbounded, so the sentence changes and the row + # stops counting as work in flight — it is waiting for its author now. + stalled_scan!(page, stall_after_seconds() + 60) + reading = Pending.reading(Pending.get(user, pending.id)) + assert reading.stage == :stalled + assert reading.state == :stalled + assert Pending.in_progress_summary(user.id).count == 0 + + # The ceiling is the installation's to move, both ways. + put_config(:ai_check_stall_seconds, 24 * 3_600) + assert Pending.stage(Pending.get(user, pending.id)) == {:checking, 1} + end + + test "a stalled row keeps its text, keeps its place in the queue and publishes itself " <> + "when the scanner comes back", + %{user: user, files: files} do + attachment = file!(user, files) + + {:ok, pending} = + Pending.create(user, "post", %{}, %{body: "Read this"}, attachments: [attachment]) + + rendered!(attachment) + page = page!(attachment, "pending") + stalled_scan!(page, stall_after_seconds() + 3_600) + assert Pending.reading(Pending.get(user, pending.id)).state == :stalled + + # Nothing was refused, so nothing offers to throw the file away, and the + # sweeper leaves the row alone — but still advances its clock, or an + # unworkable row would hold the front of every oldest-first batch. + assert Pending.reading(Pending.get(user, pending.id)).publishable_without_refused? == false + assert Pending.sweep(4) == 1 + assert reload(pending).status == "waiting" + + assert Pending.due(4) == [], + "a row whose check cannot run is still at the front of the due query" + + # Ollama is back: the verdict lands and the post goes out by itself. An + # outage of any length is a delay, never a refused post. + Repo.update_all(from(i in Image, where: i.id == ^page.id), set: [moderation: "approved"]) + + Repo.update_all(from(s in ImageScan, where: s.subject_id == ^page.id), + set: [status: "approved", service_failing_since: nil] + ) + + assert {:ok, post} = Pending.media_changed(:attachment, attachment.id) + assert reload(pending).status == "published" + assert post.body == "Read this" + end + end + describe "the author's page" do test "lists what waits and what was published", %{user: user, files: files} do attachment = file!(user, files) diff --git a/test/vutuv_web/live/uploads_stalled_check_test.exs b/test/vutuv_web/live/uploads_stalled_check_test.exs new file mode 100644 index 000000000..402bede63 --- /dev/null +++ b/test/vutuv_web/live/uploads_stalled_check_test.exs @@ -0,0 +1,96 @@ +defmodule VutuvWeb.UploadsStalledCheckTest do + @moduledoc """ + What the author's own queue at `/system/uploads` says when the AI check + cannot run (issue #2149). + + The measurement that opened the issue was a sentence, not a state: on an + installation whose scanner is down, the page said "Unsere KI prüft gerade + 1 Bild." at ten minutes, at a day and at thirty days. So the assertions here + are the words, in both languages — the German most of all, because + `mix gettext.extract --merge` fuzzy-fills a new msgid with some unrelated + translation and nothing fails the build when it does. + + `async: false`: it flips `:uploads_dir_prefix`, which every uploader reads, + and `Application.put_env/3` is global state the SQL sandbox does not roll + back. + """ + + use VutuvWeb.ConnCase, async: false + + import Ecto.Query, only: [from: 2] + import Phoenix.LiveViewTest + import Vutuv.AttachmentHelpers, only: [page!: 2, stalled_scan!: 2, stall_after_seconds: 0] + import Vutuv.WebPushHelpers, only: [put_config: 2] + + alias Vutuv.AttachmentFixtures, as: Fixtures + alias Vutuv.Attachments + alias Vutuv.Attachments.Attachment + alias Vutuv.Posts.Pending + alias Vutuv.Repo + + setup %{conn: conn} do + tmp = Path.join(System.tmp_dir!(), "vutuv_stalled_ui_#{System.unique_integer([:positive])}") + files = Path.join(tmp, "files") + File.mkdir_p!(files) + put_config(:uploads_dir_prefix, tmp) + on_exit(fn -> File.rm_rf(tmp) end) + + # Files are an admin feature until an installation opens them up. + {conn, user} = create_and_login_admin(conn) + + {:ok, attachment} = + Attachments.create_pending(user, Fixtures.text_file(files), "wartebericht.txt") + + {:ok, _pending} = + Pending.create(user, "post", %{}, %{body: "Waiting"}, attachments: [attachment]) + + stalled_page!(attachment) + + %{conn: conn, user: user} + end + + # A file whose render is done and whose one preview page has been waiting on + # an unreachable scanner for longer than the ceiling. + defp stalled_page!(%Attachment{} = attachment) do + Repo.update_all(from(a in Attachment, where: a.id == ^attachment.id), set: [stage: "ready"]) + + attachment + |> page!("pending") + |> stalled_scan!(stall_after_seconds() + 60) + end + + test "the page says the check cannot run instead of that it is running", %{conn: conn} do + {:ok, _live, html} = live(conn, ~p"/system/uploads") + + assert html =~ ~s(data-pending-status="stalled") + assert html =~ "This post is waiting for a check that cannot run" + assert html =~ "Our AI check cannot be reached at the moment" + assert html =~ ~s(data-file-state="stalled") + assert html =~ "waiting for the check" + + refute html =~ "being prepared", + "the file row still says something is preparing it while the scanner is unreachable" + + refute html =~ "Our AI is checking", + "the queue page still claims a check is in progress while the scanner is unreachable" + end + + test "and says it in German", %{conn: conn} do + {:ok, _live, html} = + conn + |> Phoenix.ConnTest.recycle() + |> Plug.Conn.put_req_header("accept-language", "de-DE,de") + |> live(~p"/system/uploads") + + assert html =~ "Dieser Beitrag wartet auf eine Prüfung, die nicht läuft" + assert html =~ "Unsere KI-Prüfung ist zurzeit nicht erreichbar" + + # The one the browser caught: a msgid added after the last extract renders + # in English on a German page, and `gettext.extract --merge` then filled it + # with "Wartet auf den Besitzer" — waiting for somebody else entirely. + assert html =~ "wartet auf die Prüfung" + + refute html =~ "Unsere KI prüft gerade" + refute html =~ "waiting for the check" + end +end