Tab switching stalls for 1-5 s on live SSH tabs

[wilsonglasser]

Split out from #87, where it was reported alongside the vertical tab strip work. It is not a side-dock problem, so it gets its own issue.

What was reported

@sosokun sees a 1-5 s freeze when switching between tabs, most visibly at 0:20-0:24 of the recording below, switching from pxm1 to sys-el9. Shorter stalls happen on other switches too.
https://github.com/user-attachments/assets/f7672947-7086-4ec0-9423-4ce35f72adb7

What we already know

Not specific to the side dock.
The reporter confirmed the same delays with every tab on the top strip, so the vertical strip work
(#87) is not the cause and this outlives it. All tabs are live SSH sessions, not local shells.
Reported on v0.10.0 nightly (cfaedc8).
Reporter environment:

OS	CachyOS Linux Rolling (x86_64), kernel 7.1.3-2-cachyos
Display	Wayland, KDE
Renderer	Vulkan, NVIDIA GeForce RTX 4080 SUPER

Leading theory (unconfirmed)

A terminal buffer reflow on reveal.
A background tab is not resized while it is hidden, so if the window geometry changed in the meantime, making it visible reflows its entire scrollback in one go, which can take a while on a large buffer. That would explain why the stall scales with the specific tab rather than being uniform. This is a theory, not a diagnosis. It has not been reproduced locally.

What would settle it

The Performance HUD (Settings > Advanced > Performance HUD) measures the terminal frame time, and the reveal-time re-layout is inside what it measures. So a screenshot of the HUD taken during or right after a slow switch splits the problem in two: a huge frame (hundreds of ms to seconds) exactly on the switch means the cost is the terminal re-layout, and the fix has a precise target; the HUD staying low while the app still hangs rules the terminal out entirely and moves the search elsewhere (compositor, surface reconfigure, the iced layout pass). Deliberately not guess-fixing this before the measurement exists: both branches lead to completely different code, and the wrong guess would add complexity to a path that is already correct.

Also useful, if easy to check

Does it happen with the window untouched since launch, or only after a resize / sidebar toggle / layout change? Is the slow tab always one with a lot of scrollback or a long-running session? Any tab, or consistently the same one?

[wilsonglasser]

Measurements, one bug found and fixed, and a narrower ask.

A real bug, fixed

view_content picked the connection-progress screen whenever a connect was in flight anywhere, without checking which tab it belonged to. So one connecting (or failed-and-not-dismissed) tab painted over every tab until the connect resolved. That reads as a tab-switching stall, and it lines up with your report: a dormant pinned tab dials on its first open, which is also why it was intermittent. Fixed on main by scoping the connect screen to its own tab.

What the terminal is NOT doing

Buffer reflow on reveal, the leading theory here, is real as a mechanism but far too cheap to matter:

History	Column change	Reflow cost
10.000 lines (the default)	-20 cols	18,7 ms
10.000 lines	-1 col	0,7 ms
50.000 lines	-20 cols	92 ms
100.000 lines	-20 cols	198 ms

A whole second would need roughly half a million lines of scrollback. Rows-only resizes and same-geometry switches cost nothing. I also rebuilt your setup locally (real sshd, four live sessions with 60.000 lines each, two pinned tabs restored from boot, dock on the left): every switch cost between 0,0 ms and 3,8 ms of UI-thread work, including the one where a dormant pinned tab dials.

What your video already told us

Your recording has the Performance HUD on. During the 0:20-0:24 freeze it reads peak T 0.0, slow 0, cache 87%, net rtt 0-1 ms. Comparing frames at 20 fps in three regions, normal switches resolve under 100 ms, and then between 21,05 s and 23,30 s not one pixel changes anywhere, not even the hover highlight under your cursor. So the app did not draw a slow frame, it did not draw at all. That leaves the gap between event loop and presentation, which is where your Wayland + KDE + NVIDIA + Vulkan setup differs from anything I can reproduce.

The one thing that would settle it

ICED_PRESENT_MODE=mailbox oryxis

Does the freeze survive? If it disappears the present mode is the cause and the fix is on our side. immediate and no_vsync are the other values worth trying. If it makes no difference: the OpenGL renderer (Settings > Advanced) and an X11 session would narrow it further.

The reflow result is measured and final, the fix above is a real bug that explains part of it, and the presentation theory is a hypothesis, not a diagnosis.

[sosokun]

Hi! Today I installed Oryxis: v0.11.0 (nightly, 50e1746).
Now the freezes have become even more noticeable and longer in time. Sometimes they take more than a minute. The ICED_PRESENT_MODE=mailbox parameter did not change anything.

Also I checked Oryxis: v0.11.0 (stable, ee153a5). There aren't freezes so long as in nightly, 50e1746.

We can negotiate about real-time testing on my env, if u want.

[wilsonglasser]

Thanks, the stable vs nightly comparison and the mailbox test were exactly what was needed. Mailbox changing nothing rules the present mode out as the whole story, and I audited every commit between your two builds (ee153a5 and 50e1746): nothing in that range explains a minute-long freeze by analysis, and none of it reproduces here on identical build flags.

So instead of another guess, the next nightly ships an instrument.

What I need from you (one run)

1. Install the next nightly (any build containing commit 2d80c1a).
2. Settings > Advanced > enable Debug logging.
3. Use the app normally until you hit a few freezes, ideally a long one.
4. Attach ~/.oryxis/oryxis-debug.log here.

With that toggle on, a watchdog thread observes the event loop from outside and classifies every freeze into the log: a message handler blocked (it names the message plus the 24 before it), the event loop no longer dispatching, the app processing messages but not drawing, or everything alive on our side, which means the frame dies after it leaves us. The first case plus a message name is essentially the diagnosis.

If you can catch a freeze live from a second terminal, that is gold:

gdb -p $(pidof oryxis) -batch -ex 'thread apply all bt' > oryxis-stacks.txt

(eu-stack -p <pid> works too if you have elfutils.)

Also in that nightly: the shell-integration mark buffer paid a hidden per-batch cost once it filled up, growing with session age, which matches "got worse over time". Fixed regardless, though I doubt it is your whole minute.

And yes to the real-time testing offer, gladly, if the log comes back ambiguous.

[sosokun]

Hi! I had a little rest.

There were 2 freezes (each for 5-7 seconds) in the attached files.

oryxis-stacks2.txt
oryxis-stacks.txt

debug log contains nothing:

Oryxis: v0.11.0 (nightly, a9f7d98)

The same with OpenGL and It's even worse with CPU

[wilsonglasser]

v0.12.0 has both fixes (the connect screen no longer paints over other tabs, and the queue no longer scans per mark). If it still stalls for you, Debug logging now ships a stall watchdog that names which layer stopped, that log is exactly what I need.