🔄️ Chat message pre-fetching by AndyScherzinger · Pull Request #6454 · nextcloud/talk-android

AndyScherzinger · 2026-08-04T18:51:07Z

New ChatMessageSyncer: fetch-and-persist core extracted from OfflineFirstChatRepository into a stateless singleton usable without an open chat; both paths share one write path.
Room-list prefetch: rooms with advanced lastActivity (matching iOS) and unread rooms without cached messages get a background catch-up - delta fetch, or initial fetch creating the first chat block for never-opened rooms (beyond iOS). Capped at 20 rooms / 3 concurrent; skipped in battery saver / restricted background data.
Push-triggered single-room catch-up after the notification is shown (best effort, never delays the notification).
Notification-safe fetching: markNotificationsAsRead=0, read marker untouched, gated on the chat-keep-notifications capability.
Chat open trusts the cache: zero network when cached messages reach lastMessage.id; proportional delta fetch otherwise; initial load never waits for the websocket.
Hardening: chat-block reconciliation on message expiry; empty /room responses no longer cascade-delete the cache; insurance fetches anchor on the DB.
Tests: syncer unit tests + Robolectric integration test; debug logging for logcat verification.

How to verify: check logcat -> expect "Catching up messages for N rooms", per-room "Background catch-up … fetched N message(s)", and the "Initial online request is skipped … until the conversation's last message" line on open. Most visible on cold opens from push notifications and throttled networks. (for logging, see commit 64a3c85). Alternatively, deploy it on the phone, wait for a while, put the phone in flight-mode, then open the chats. You should then see the unread messages.

Assumptions to be aware of

markNotificationsAsRead=0 server semantics - assumed the server keeps push notifications for background-fetched messages whenever chat-keep-notifications is announced; taken from iOS usage, not server code. If wrong, prefetching silently dismisses notifications. Most critical - please confirm.
Federated conversations use one consistent message-ID space - lastMessage.id from the room list is compared against local chat-block IDs; divergence via the federation proxy could wrongly skip the initial fetch (unread tail delayed to polling) or fetch redundantly (harmless). Needs a federated-room test.
The chat relay never replays a backlog on (re)connect - sole justification for the delta fetch at open on HPB; based on code comments/observed design, not signaling-server sources. Partial replay could leave gaps until the 2-min insurance request.
GET /chat long polling still works against HPB servers - the fallback when the websocket is slow/mis-detected; pre-existing behavior, unverified server-side.
lastActivity advances for new chat messages and its persisted copy is a valid cross-restart baseline; missed candidates degrade to pre-PR open behavior, non-message bumps cost one empty delta.
Message IDs strictly increase per conversation (non-contiguous is fine) - inherited chat-block premise, now leaned on harder for delta anchors.
Message expiration is roughly server-synchronized - block trimming tolerates clock skew (early: harmless re-fetch; late: briefly stale block, as before).
pushMessage.id is always a fetchable room token for TYPE_CHAT pushes (incl. federated/invite edge cases); wrong token costs one failed, caught request; the notification is unaffected.
NotificationWorker lifetime accommodates the synchronous catch-up - notification dispatched first, but ordering isn't formally guaranteed; an expedited separate work item is the fallback design.
Concurrent persists are race-safe - open-path delta, first poll, signaling and background catch-up may interleave; upserts + getConnectedChatBlocks merging + insurance requests are the intended self-healing, without formal proof. Transaction boundaries deserve review.
Capability cache freshness suffices for the gate - 12 h CapabilitiesWorker window; a server downgrade inside it exposes assumption 1's impact until refresh.
Chat API stays at v1 - CHAT_API_VERSION = 1 hardcoded in prefetch/push paths, mirroring ChatViewModel; compile-time visible, low risk.
20 rooms / 3 concurrent caps are server-friendly - no rate-limit guidance existed, iOS is uncapped; too aggressive risks throttling, too conservative leaves rooms un-prefetched until the next sync (logged). Tunable constants.

🏁 Checklist

⛑️ Tests (unit and/or integration) are included or not needed
🔖 Capability is checked or not needed
🔙 Backport requests are created or not needed: /backport to stable-xx.x
📅 Milestone is set
🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

🤖 AI (if applicable)

The content of this PR was partly or fully generated using AI

github-actions · 2026-08-05T09:34:44Z

APK file: https://github.com/nextcloud/talk-android/actions/runs/30940554955/artifacts/8925256420
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
(please click on link to get QR code displayed)

mahibi

Most of it works really good and it's really promising 👍

However there are some major problems with the chatrelay and insurance request (see my comments).
These are edge cases what won't happen too often, but they would create permanent gaps.
For now i just identified this by code reviewing (not yet reproduced by testing).
Fixing it should not be too hard, might be just reverting some lines..

I did not finish the code review yet, so there might be more on monday (wont finish it today).

github-actions · 2026-08-11T15:59:08Z

APK file: https://github.com/nextcloud/talk-android/actions/runs/31505646937/artifacts/9108620109
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
(please click on link to get QR code displayed)

mahibi · 2026-08-11T17:47:04Z

With the current changes that i made i think the great dangerous pitfalls are avoided.
However i would like to test this branch even more tomorrow..

github-actions · 2026-08-12T09:41:56Z

APK file: https://github.com/nextcloud/talk-android/actions/runs/31583130547/artifacts/9136342451
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
(please click on link to get QR code displayed)

AndyScherzinger · 2026-08-12T09:54:45Z

With the current changes that i made i think the great dangerous pitfalls are avoided.
However i would like to test this branch even more tomorrow.

@mahibi sounds good to me. I'd also like to further test and added some more commits to improve the unread marker behavior while opening talk rooms with a lot of unread messages - a thing I experienced i.e. opening rooms with more than 200 unread messages.

github-actions · 2026-08-12T11:15:43Z

APK file: https://github.com/nextcloud/talk-android/actions/runs/31590464667/artifacts/9139104323
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
(please click on link to get QR code displayed)

After displaying a message notification, trigger a single-room catch-up so the pushed message and any backlog are persisted to the local database while the app is backgrounded. Best effort only: failures never delay or suppress the notification. Skipped without the chat-keep-notifications capability or in battery saver. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Skip deleteLeftConversations when GET /room unexpectedly returns no conversations while some exist locally. A broken or partial server response would otherwise delete every local conversation and, via foreign key cascade, wipe the cached chat messages and chat blocks that the message prefetch relies on. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Trim chat block boundaries to the oldest/newest message that still exists after deleteExpiredMessages and delete blocks whose messages are all gone, so block boundaries never point to rows that no longer exist. The cleanup moved into ChatMessageSyncer so future background callers share it. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Replace the in-memory latestKnownMessageIdFromSync with the newest message id from the chat blocks. The field lived in the unscoped repository and was reset to zero on every chat open, so an insurance request or signaling-triggered refresh running before the first successful sync of the session queried with lastKnownMessageId=0. The database is always at least as fresh because messages and chat blocks are persisted together. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Add unit tests for ChatMessageSyncer: field map safety flags, offline and capability gates of catchUpRoom, delta fetch for rooms with a chat block, initial fetch with block creation for never-opened rooms, merging of connected chat blocks, the not-modified and constraint violation paths, and chat block reconciliation after message expiry. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Run at most one catch-up per room at a time: requests arriving while one runs only mark a rerun that the running catch-up executes after finishing, consecutive fetches are paced by a five second cooldown and a burst performs at most three fetches. A flood of push notifications for an active room now causes one or two delta fetches instead of one per push. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Reintroduce a sync-only anchor for the insurance request, now held per conversation in the singleton ChatMessageSyncer so it survives reopening a chat. It is updated exclusively from http pull results and seeded from the conversation's lastMessage on skip, never from signaling messages, so the insurance request keeps detecting messages that arrived between the last sync and signaling delivery. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

A single delta fetch is capped by the request limit and may only narrow a large backlog. Repeat the fetch until the server returns fewer messages than the limit, bounded by a maximum round count, and fall back to fetching the newest messages with includeLastKnown when the bound is hit — so chat blocks never claim ranges that were not fetched and the chat relay path cannot create permanent gaps. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Trigger the push catch-up from the notification data callback and pass the thread id parsed from the notification's objectId, so messages of a pushed thread land in the thread's chat block. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

...brought in via a second dev work strem not related to chat message fetching, hence removing Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

… rounds' closeBacklog's fallback fires when a backlog exceeds MAX_BACKLOG_ROUNDS worth of fetches and switches to pulling the newest messages directly. That fallback lands in its own, disconnected chat block (see updateBlocks) — it is explicitly not contiguous with whatever the backlog rounds already fetched. The returned SyncOutcome nevertheless combined both: oldestPersisted from the backlog rounds with the fallback's own newest id, and summed both counts. This reported a misleadingly contiguous oldest..newest span for two unrelated ranges with an unclosed gap in between — currently only surfacing in a log line, but a footgun for any future caller that assumes the range is coherent. Report the fallback's own oldest/newest/count instead, and fold the backlog rounds' totals into the existing warning log so that information isn't lost, just no longer misattributed to a single range. persistedNewMessages stays an aggregate (totalCount > 0 || fallbackOutcome.persistedNewMessages): it only answers "was anything new persisted in this call", which holds independently of whether the two ranges are contiguous. Reducing it to fallbackOutcome.persistedNewMessages would incorrectly report "nothing new" whenever the terminal fallback fetch itself finds nothing further, even though the backlog rounds already persisted real messages moments earlier in the same call — which would, for example, make fetchNewMessages()'s caller retry a sync that had already succeeded. Update the closeBacklog fallback test accordingly. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Marcel Hibbe <dev@mhibbe.de>

loadInitialMessages skipped the network fetch entirely whenever the local chat block already reached conversationModel.lastMessage.id, trusting that value as proof we were caught up with the server. That field is only as fresh as the last room list sync, though: a message sent while the app had the conversation list open but the chat closed (and no push-triggered catch-up ran, e.g. on flavors without FCM) never updates it. Opening the chat then wrongly concluded there was nothing to fetch, leaving the newest message missing until whatever live-update mechanism happened to be active caught up on its own. Always close the backlog from the newest locally known message instead of gating on the conversation's cached lastMessage. closeBacklog is a single cheap request when there is genuinely nothing new, so there is no upside to trusting a value that isn't guaranteed current. Drop ChatMessageSyncer.seedHttpSyncedMessageId with it: it existed only to seed the insurance anchor from that same stale field for the now-removed branch, and has no other caller. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Marcel Hibbe <dev@mhibbe.de>

…ions Turn the two-way choice in loadInitialMessages into a plain if/else on a single named condition, weLikelyOnlyHaveASmallBacklog, and extract each branch body into its own function: closeBacklogFromNewestOfflineMessage and fetchNewestMessagesForInitialLoad. No behavior change. closeBacklogFromNewestOfflineMessage's fallback-safety rationale and fetchNewestMessagesForInitialLoad's per-case log messages are now documented against the actual booleans they depend on, rather than being inferred from which when-branch happened to call them. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Marcel Hibbe <dev@mhibbe.de>

closeBacklog can fall back to fetching the newest messages instead of actually closing the gap once MAX_BACKLOG_ROUNDS is exceeded, leaving the remaining range genuinely open for later. The name promised full closure with no such caveat. Rename it and its OfflineFirstChatRepository wrapper (closeBacklogFromNewestOfflineMessage) to make the best-effort nature explicit, and update call sites, comments, and test names to match. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Marcel Hibbe <dev@mhibbe.de>

With prefetching, a chat usually opens with an already up-to-date cache, so every pull request sends the server's own lastCommonReadId back and is answered with 304 Not Modified. The X-Chat-Last-Common-Read header only exists on 200 responses, so _lastCommonReadFlow never emitted and the UI stayed at its initial 0 — every own message rendered with a single checkmark regardless of its actual read state. - seed _lastCommonReadFlow directly in loadInitialMessages: the conversation entity is kept fresh by the room list sync and the prefetch, so its lastCommonReadMessage is the correct initial value - give _lastCommonReadFlow replay = 1 so the seed survives subscriber timing between repository and view model - keep the previous value when a 200 response carries no X-Chat-Last-Common-Read header: overwriting with null dropped lastCommonReadId from all following field maps, and without that parameter the server never reports pure read-state changes Known limitation (follow-up): while a chat is open against a high performance backend, read-state changes still only arrive with the insurance requests (every 2 minutes), because signaling messages carry no read state and 304 responses carry no header. The room list sync refreshes the conversation's lastCommonReadMessage far more often, but ChatViewModel's conversationFlow is filtered with distinctUntilChangedBy { it.lastReadMessage }, so that fresher value never reaches the message list. Folding it in — e.g. maxOf(repository value, conversation.lastCommonReadMessage) in observeMessages — requires relaxing that filter and should be done as a separate change. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

… open Read-receipt checkmarks were only updated by the X-Chat-Last-Common-Read header of 200 chat pull responses. On a high performance backend those are rare while a chat is open: messages arrive via signaling (which carries no read state) and the insurance requests only run every two minutes, so a pure read-state change took up to two minutes to show — even though the room list sync keeps writing a fresher lastCommonReadMessage into the conversation row all along. - let conversationFlow re-emit when lastCommonReadMessage changes instead of filtering on lastReadMessage alone - fold the conversation's value into the checkmark input with maxOf(header value, conversation.lastCommonReadMessage): either side can be ahead of the other, and since lastCommonRead only moves forward server-side the maximum is always correct The extra re-emissions are harmless: the message list rebuild is debounced, observeConversation is idempotent, conversationAndUserFlow is consumed with take(1) and the pinned-message flow deduplicates downstream. Also removes a left-over debug println from conversationFlow. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

Fetching the pushed room's messages no longer runs synchronously inside NotificationWorker. It now enqueues a ChatMessageCatchUpWorker (network-constrained, exponential backoff) after the notification is displayed, so a slow or failing fetch can never delay the notification and transient failures are retried instead of lost. SyncOutcome gained a syncFailed flag so the worker can tell "no new messages" from a failed fetch. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

@transaction

The upsert -> getConnectedChatBlocks -> replaceConnectedChatBlocks sequence in ChatMessageSyncer.updateBlocks was not atomic: with the open-path delta, long poll/insurance, signaling and background catch-up all able to update the same conversation concurrently, two callers could each upsert a block and query connectivity before seeing the other's write, leaving overlapping blocks behind. The whole sequence now runs as a single Room @transaction in ChatBlocksDao.upsertAndMergeConnectedChatBlocks. getConnectedChatBlocks and deleteChatBlocks became suspend so no blocking DAO calls run inside the suspending transaction. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

github-actions · 2026-08-13T21:49:53Z

APK file: https://github.com/nextcloud/talk-android/actions/runs/31746876327/artifacts/9199528317
To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.
(please click on link to get QR code displayed)

mahibi · 2026-08-14T15:41:44Z

reviewed and tested a lot. It's ready to merge in my opinion.
Great improvements 👍 🚀

AndyScherzinger added enhancement New feature or request 3. to review Waiting for reviews feature: chat 🗨️ AI assisted labels Aug 4, 2026

AndyScherzinger changed the title ~~Feat/noid/msg pre fetching~~ Chat message pre-fetching Aug 4, 2026

mahibi self-requested a review August 5, 2026 08:49

AndyScherzinger force-pushed the feat/noid/msgPreFetching branch from 6492492 to 569655c Compare August 6, 2026 12:18

mahibi requested changes Aug 7, 2026

View reviewed changes

AndyScherzinger force-pushed the feat/noid/msgPreFetching branch 2 times, most recently from 3610e28 to c4e596c Compare August 9, 2026 20:04

AndyScherzinger changed the title ~~Chat message pre-fetching~~ 🔄️ Chat message pre-fetching Aug 9, 2026

AndyScherzinger force-pushed the feat/noid/msgPreFetching branch 3 times, most recently from 4705792 to 462364f Compare August 10, 2026 05:42

mahibi force-pushed the feat/noid/msgPreFetching branch from 207ea88 to f4d4177 Compare August 11, 2026 15:11

nextcloud deleted a comment from github-actions Bot Aug 11, 2026

AndyScherzinger force-pushed the feat/noid/msgPreFetching branch from ffa71ca to ac433db Compare August 12, 2026 11:07

AndyScherzinger and others added 20 commits August 13, 2026 23:42

feat: Add extra logging to spot pre-fetching in debug mode

acd87f7

Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

refactor(chat): restructure catch-up guards to satisfy ReturnCount

bac2d79

Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

chore: remove unrelated code change

a0f0d9c

...brought in via a second dev work strem not related to chat message fetching, hence removing Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>

AndyScherzinger force-pushed the feat/noid/msgPreFetching branch from 1a1f664 to 3f6a24d Compare August 13, 2026 21:42

AndyScherzinger marked this pull request as ready for review August 13, 2026 21:48

AndyScherzinger mentioned this pull request Aug 13, 2026

🔄️ Improve unread marker display #6484

Merged

6 tasks

mahibi approved these changes Aug 14, 2026

View reviewed changes

AndyScherzinger added this to the 25.0.0 milestone Aug 14, 2026

AndyScherzinger merged commit 5698dab into master Aug 14, 2026
17 of 20 checks passed

AndyScherzinger deleted the feat/noid/msgPreFetching branch August 14, 2026 15:44

AndyScherzinger mentioned this pull request Aug 22, 2026

fix(chat): don't freeze the author-info flag at first composition #6544

Open

8 tasks

Navigation Menu

Uh oh!

Uh oh!

🔄️ Chat message pre-fetching - #6454

🔄️ Chat message pre-fetching#6454
AndyScherzinger merged 27 commits into
masterfrom
feat/noid/msgPreFetching

AndyScherzinger commented Aug 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

mahibi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

mahibi commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

AndyScherzinger commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

mahibi commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

AndyScherzinger commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🏁 Checklist

🤖 AI (if applicable)

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

mahibi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

mahibi commented Aug 11, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

AndyScherzinger commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

mahibi commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AndyScherzinger commented Aug 4, 2026 •

edited

Loading