A post says so when the AI check cannot run, instead of waiting forever

config/config.exs

@@ -126,6 +126,16 @@ config :vutuv, :screenshot_blocklist, ["reddit.com", "heise.de"]
 # (config/runtime.exs).
 config :vutuv, :moderate_images, true
 config :vutuv, :ollama_url, "http://localhost:11434"
+# How long the scanner may be unreachable before a post waiting on one of its
+# verdicts stops telling its author that a check is in progress (issue #2149).
+# The queue itself is unchanged — a service outage still retries for ever and
+# still releases nothing — so this is a ceiling on the *sentence*, not on the
+# post: the moment Ollama answers, the verdict lands and the post publishes
+# itself. Half an hour, which is what a scanner that answers but cannot judge
+# an image already spends before it gives up (5 tries, 120+240+480+960 s), and
+# far enough above the 300 s retry pace that a blip is never named. Runtime
+# override: AI_CHECK_STALL_SECONDS (config/runtime.exs).
+config :vutuv, :ai_check_stall_seconds, 1_800
 config :vutuv, :ollama_vision_model, "qwen3-vl:8b"
 
 # Whether a link-preview capture is also judged on whether it shows the PAGE

config/runtime.exs

@@ -631,6 +631,14 @@ if config_env() == :prod do
     config :vutuv, :moderate_images, false
   end
 
+  # How long that outage may run before a post waiting on a verdict stops
+  # saying a check is in progress and says plainly that it cannot run
+  # (issue #2149). Nothing is refused or released at the ceiling; an
+  # installation whose GPU box is regularly away for an hour raises it.
+  if seconds = System.get_env("AI_CHECK_STALL_SECONDS") do
+    config :vutuv, :ai_check_stall_seconds, String.to_integer(seconds)
+  end
+
   # The link-preview page check: judges each capture on whether it shows the
   # page or a consent/ad/login wall, and blocklists the site when it does not.
   # SCREENSHOT_PAGE_CHECK=false switches it off (an installation whose Ollama

docs/ADMINS.md

@@ -227,6 +227,7 @@ Everything else has a default (the vutuv.de production value):
 | `SAVED_SEARCHES_MAX_PER_MEMBER` | `10` | Most saved searches (with e-mail alerts) one member may store (anti-abuse). A member at the cap is asked to delete one first |
 | `GEO_COUNTRIES` | `DE,AT,CH` | Comma-separated ISO 3166-1 alpha-2 codes whose bundled GeoNames postal data is loaded for offline zip → coordinate resolution on job postings. To add a country, drop its GeoNames zip export (`download.geonames.org/export/zip/<CC>.zip` → extracted `<CC>.txt`, optionally gzipped to `<CC>.txt.gz`) into `priv/geo/` and add the code here. Fully offline — no outbound calls |
 | `IMAGE_MODERATION_ENABLED` | `true` | `false` turns AI image moderation off (images publish immediately, as before the feature). While enabled, **every** image — avatars, covers, post / job-posting / organization images and the automatic link and homepage screenshots — waits invisible to everyone but its owner until a local Ollama vision model approves it; an unsafe image is deleted on the spot and the owner notified. Fail-closed: with Ollama unreachable, new images queue up and are scanned automatically once it is back — nothing is ever auto-approved. Set `false` only on installations without Ollama |
+| `AI_CHECK_STALL_SECONDS` | `1800` | How long the scanner may be unreachable before a post waiting on one of its verdicts stops telling its author that a check is in progress and says plainly that the check cannot run. It changes **words, not the queue**: nothing is refused, nothing is released early, the retry goes on at its usual five-minute pace, and the post publishes itself the moment a verdict lands. Raise it on an installation whose GPU box is routinely away for an hour, so a normal absence is not called a stall; the chip in the app bar stops counting such a post as work in flight either way |
 | `IMAGE_PIXELATION_WINDOW_SECONDS` | `3600` | How long a picture waiting for that verdict shows readers a **pixelated preview** of itself — a separately stored file reduced to 64 cells on its long edge, not the picture behind a blur filter, so what reaches a reader carries none of the detail. It keeps a post card whole while the scan runs, and the real picture replaces it live the moment the verdict lands. Past this window the card falls back to a grey "being checked" tile, so a derivative of an unvetted picture never sits on a public page indefinitely. `0` switches the pixelated preview off entirely, which is the strictest posture |
 | `OLLAMA_URL` | `http://localhost:11434` | Base URL of the Ollama instance every AI feature talks to (image scan, translations, tag merge assist, employment-reference analysis). May be a **comma-separated list** (`http://gpu-box:11434,http://second-gpu:11434,http://localhost:11434`), which is read two ways at once. For a single call it is a **priority list**: every instance but the last is tried with a 30 s budget and skipped on any failure, the last one is the patient fallback (120 s, covers a CPU cold load). For calls that overlap it is also a **pool**: the second one starts on the least busy instance, so a second GPU takes work rather than waiting for the first to break. Verdicts are identical either way — the list only buys speed |
 | `OLLAMA_CONCURRENCY` | all `OLLAMA_URL` entries but the last | How many instances at the head of the list are treated as workers — which is both how deep the pool goes and how many calls a background sweep may have in flight. The default holds the **last** entry back as the fallback of record, because that is typically the web server's own CPU Ollama: `http://gpu-box:11434,http://localhost:11434` is one worker (unchanged from before this existed), `http://gpu1:11434,http://gpu2:11434,http://localhost:11434` is two. Set it to the number of entries when the list is all GPUs and nothing is held in reserve, or higher when one box can genuinely run two calls at once |

docs/architecture/attachments.md

@@ -158,6 +158,21 @@ the clip now one case of it. `Vutuv.Posts.Pending` owns the whole question:
   stops waiting, and the author is offered the text without the refused file
   or neither. Both are `phx-click` events, never links: each destroys state,
   and a state-destroying GET dies on a Back button or a link prefetch.
+* **A check that cannot run is said out loud** (#2149). An unreachable scanner
+  is retried for ever by design — nothing may be released without a verdict —
+  so the wait itself has no ceiling, and the *sentence* used not to either: on
+  an installation whose Ollama was down the card said "our AI is checking 1
+  picture" at ten minutes, at a day and at thirty days. `image_scans` now
+  records **since when** the scanner has been unreachable
+  (`service_failing_since`, stamped on the first service error and cleared the
+  moment Ollama answers at all — even to say it cannot judge that file), and
+  past `ImageScans.stall_after_seconds/0` (`AI_CHECK_STALL_SECONDS`, 1800 s)
+  the stage is `:stalled`: the card says the check cannot be reached, the
+  app-bar chip stops counting the row as work in flight, and the clip's own
+  "our AI is checking it" line steps aside. The queue is untouched, so an
+  outage is a delay and never a refused post — a blip is 15 times under the
+  ceiling and never named, and whenever the scanner returns the verdict lands
+  and the post publishes itself.
 * **Surviving a deploy.** The publish is claimed by a compare-and-set on
   `status`, and the claim writes `minted_post_id` — the id the post is about to
   get — so a slot killed between the insert and the bookkeeping is resumed by

docs/architecture/images.md

@@ -1541,7 +1541,12 @@ The moving parts (all under `Vutuv.Moderation`):
   said what the model actually saw). Two error classes: `{:service, _}`
   (Ollama down — retry forever, fail-closed) vs `{:image, _}` (this file
   can't be judged — capped, then rejected; an unverifiable image is never
-  released).
+  released). A service error also stamps `service_failing_since` on the scan
+  row, kept from the first failure of the run and cleared as soon as Ollama
+  answers at all; `ImageScans.stalled_subjects/1` reads it so whoever is
+  *waiting* on a verdict (a post with a file, #2149) can stop claiming a check
+  is in progress after `stall_after_seconds/0`. It bounds the sentence, never
+  the queue: nothing is released or refused when it passes.
 - `ImageScanWorker` — boot-resume + poll + nudge, mirroring
   `Vutuv.Posts.ScreenshotWorker`; hourly `repair_drift/0` re-enqueues any
   asset stranded in `pending`, and `ImageSubjects.settle_stranded_quarantine/0`

lib/vutuv/moderation/image_scan.ex

@@ -50,6 +50,12 @@ defmodule Vutuv.Moderation.ImageScan do
     field(:attempts, :integer, default: 0)
     field(:next_attempt_at, :utc_datetime)
     field(:last_error, :string)
+    # Since when the *scanner* — not this image — has been failing, unbroken.
+    # Set on the first service error and cleared the moment Ollama answers at
+    # all, even to say it cannot judge this file, so it measures one outage
+    # rather than a lifetime of them. `Vutuv.Posts.Pending` reads it to stop
+    # telling an author a check is in progress when it is not (issue #2149).
+    field(:service_failing_since, :utc_datetime)
     field(:category, :string)
     # The model's own one-line description of the image. For a rejection it is
     # the only surviving record of what was deleted (the files are gone), and

lib/vutuv/moderation/image_scans.ex

@@ -23,7 +23,11 @@ defmodule Vutuv.Moderation.ImageScans do
   scan (the backstop for a crash between verdict and application — and for
   any future upload path that forgets to enqueue, since the gallery tables
   default their `moderation` column to `pending`). When Ollama is down the
-  queue retries forever; nothing is ever auto-approved. With
+  queue retries forever; nothing is ever auto-approved. The row does record
+  **since when** it has been unreachable (`service_failing_since`), which is
+  what lets a member waiting on that verdict be told the check cannot run
+  instead of that it is running — see `stall_after_seconds/0` and issue #2149.
+  With
   `:moderate_images` off (tests, installations without Ollama) assets are
   created `approved` and this module is dormant.
   """
@@ -50,8 +54,12 @@ defmodule Vutuv.Moderation.ImageScans do
   @image_error_cap 5
   # Service failures (Ollama down/unreachable) retry forever at this pace.
   @service_retry_seconds 300
+  # How long such an outage may run before whoever waits on the verdict is told
+  # plainly that the check cannot run (issue #2149). See `stall_after_seconds/0`.
+  @default_stall_seconds 1800
 
   @kinds ImageScan.kinds()
+  @open_statuses ImageScan.open_statuses()
   # The member personally chose these images; a rejection deletes their
   # content, so they get the notice. Machine captures (link screenshots) are
   # our artifact of a third-party page — silently showing no preview is the
@@ -71,6 +79,47 @@ defmodule Vutuv.Moderation.ImageScans do
   @doc "Whether AI image moderation is enabled on this installation."
   def enabled?, do: Application.get_env(:vutuv, :moderate_images, true)
 
+  @doc """
+  How long the scanner may be unreachable before whoever is *waiting* on one of
+  its verdicts should stop being told a check is in progress (issue #2149).
+
+  Not a cap on the queue: a service outage still retries at
+  `@service_retry_seconds` for ever, and nothing is released or refused when
+  this passes. It is the point at which the wait stops being a wait and becomes
+  an open question for the member, and the config key is there because an
+  installation whose GPU box is regularly away for an hour should not call that
+  a stall.
+  """
+  def stall_after_seconds,
+    do: Application.get_env(:vutuv, :ai_check_stall_seconds, @default_stall_seconds)
+
+  @doc """
+  Of these scan subjects, the ones whose check cannot run: an **open** scan
+  that has been failing against the service — not against the picture — for
+  longer than `stall_after_seconds/0`, unbroken.
+
+  Answered for a whole batch at once, because the readers are page-shaped: one
+  waiting card per post, and a member may have several. A subject with no open
+  scan, or one whose last attempt reached Ollama at all, is simply not in the
+  set — `service_failing_since` is cleared the moment the scanner answers, even
+  to say it cannot judge that file, so a stamp this old means every retry since
+  failed to reach it.
+  """
+  def stalled_subjects([]), do: MapSet.new()
+
+  def stalled_subjects(subject_ids) when is_list(subject_ids) do
+    cutoff = DateTime.add(DateTime.utc_now(:second), -stall_after_seconds(), :second)
+
+    from(s in ImageScan,
+      where:
+        s.subject_id in ^subject_ids and s.status in ^@open_statuses and
+          not is_nil(s.service_failing_since) and s.service_failing_since < ^cutoff,
+      select: s.subject_id
+    )
+    |> Repo.all()
+    |> MapSet.new()
+  end
+
   @doc """
   The moderation state a freshly stored image starts in: `"pending"` (limbo)
   when moderation is enabled, `"approved"` otherwise.
@@ -125,6 +174,7 @@ defmodule Vutuv.Moderation.ImageScans do
               attempts: 0,
               next_attempt_at: nil,
               last_error: nil,
+              service_failing_since: nil,
               updated_at: now
             ]
           ],
@@ -179,8 +229,23 @@ defmodule Vutuv.Moderation.ImageScans do
     cutoff = NaiveDateTime.add(NaiveDateTime.utc_now(), -@stuck_after_seconds, :second)
 
     {count, _} =
-      from(s in ImageScan, where: s.status == "scanning" and s.updated_at < ^cutoff)
-      |> Repo.update_all(set: [status: "pending", updated_at: NaiveDateTime.utc_now(:second)])
+      from(s in ImageScan,
+        where: s.status == "scanning" and s.updated_at < ^cutoff,
+        update: [
+          set: [
+            status: "pending",
+            updated_at: ^NaiveDateTime.utc_now(:second),
+            # A claim this old means the scanner never answered — an Ollama
+            # that accepts the connection and then hangs leaves no service
+            # error to stamp, and whoever waits on this verdict would never
+            # learn the check is not running (issue #2149). `coalesce` keeps an
+            # outage already being measured.
+            service_failing_since:
+              fragment("coalesce(?, ?)", s.service_failing_since, ^DateTime.utc_now(:second))
+          ]
+        ]
+      )
+      |> Repo.update_all([])
 
     count
   end
@@ -397,7 +462,12 @@ defmodule Vutuv.Moderation.ImageScans do
     update_claimed(scan,
       status: "pending",
       next_attempt_at: retry_at,
-      last_error: error_string(reason)
+      last_error: error_string(reason),
+      # Since when this outage has been running, kept from the first failure of
+      # the run rather than rewritten each time: it is what lets a waiting post
+      # tell a two-minute blip from a host whose scanner has been gone for a
+      # month (issue #2149). Cleared the moment Ollama answers at all.
+      service_failing_since: scan.service_failing_since || DateTime.utc_now(:second)
     )
 
     :ok
@@ -433,7 +503,10 @@ defmodule Vutuv.Moderation.ImageScans do
         status: "pending",
         attempts: attempts,
         next_attempt_at: retry_at,
-        last_error: error_string(reason)
+        last_error: error_string(reason),
+        # The scanner answered — it just could not judge this file. Whatever
+        # outage was running is over, and the next one starts its own clock.
+        service_failing_since: nil
       )
 
       :ok
@@ -538,15 +611,15 @@ defmodule Vutuv.Moderation.ImageScans do
   should be. A reader's own translation request is never stood down.
   """
   def busy? do
-    Repo.exists?(from(s in ImageScan, where: s.status in ^ImageScan.open_statuses()))
+    Repo.exists?(from(s in ImageScan, where: s.status in ^@open_statuses))
   end
 
   @doc "Queue totals for the admin dashboard: %{pending: n, rejected_7d: n}."
   def counts do
     week_ago = DateTime.add(DateTime.utc_now(), -7 * 86_400, :second)
 
     pending =
-      Repo.aggregate(from(s in ImageScan, where: s.status in ^ImageScan.open_statuses()), :count)
+      Repo.aggregate(from(s in ImageScan, where: s.status in ^@open_statuses), :count)
 
     rejected =
       Repo.aggregate(