Nyxt freezes when typing very fast in prompt buffer

[Nselm]

Describe the bug
Nyxts's GUI freezes after typing fast in any prompt buffer with selection filters.

Precise recipe to reproduce the issue
When typing fast in the prompt buffer, nyxt just freezes.
A video I had running in the background continued to play, but

1. I could not do anything
2. When I switched first to another window, then back, it did not display nyxt anymore, but the previous window

Possibly relevant:

• I have a history of ~100 tabs
• I have ~2000 bookmarks
• I have greatly increased the key repeat rate and reduced the delay (not sure how much, but its probably around twice as fast a normal)
• My laptop is extremely weak: It has ~2 G RAM and the rest of the specs is similar

Information

• OS name+version:
  Arch linux

• Graphics card and driver:
  Stoney [Radeon R2/R3/R4/R5 Graphics], AMD

• Desktop environment / Window manager name+version:
  i3-gaps 4.20.1-2

• How you installed Nyxt (Guix pack, package manager, build from source):
  Nyxt package in aur

• Information from nyxt --system-information:

Nyxt version: 2.2.4
Renderer version: GI-GTK
Operating system kernel: Linux 5.16.12-arch1-1
Lisp implementation: SBCL 2.2.1 (Dynamic space size: 3221225472)
Features: (:WEBKIT2 :WEBKIT2-2.34 :WEBKIT2-PASTE-PLAINTEXT :WEBKIT2-TRACKING
 :WEBKIT2-MUTE :WEBKIT2-EMOJI :WEBKIT2-MEDIA :WEBKIT2-SANDBOXING :GTK-3-22
 :GTK-3-20 :GTK-3-18 :GTK-3-16 :GTK-3-14 :GTK-3-12 :GTK-3-10 :GTK-3-8 :GTK-3-6
 :GTK-3-4 :GTK :GDK-3-22 :GDK-3-20 :GDK-3-18 :GDK-3-16 :GDK-3-14 :GDK-3-12
 :GDK-3-10 :GDK-3-8 :GDK-3-6 :GDK-3-4 :CAIRO-1-10 :CAIRO-1-12 :GDK-PIXBUF
 :GLIB-2-30 :GLIB-2-32 :GLIB-2-34 :GLIB-2-36 :GLIB-2-38 :GLIB-2-40 :GLIB-2-42
 :GLIB-2-44 :GLIB-2-46 :GLIB-2-48 :GLIB-2-50 :GLIB-2-52 :GLIB-2-54 :GLIB-2-56
 :GLIB-2-58 :GLIB :NYXT-2 :FSET-EXT-STRINGS :CUSTOM-HASH-TABLE-NATIVE :SWANK
 :PLUMP-UTF-32 :GLOBAL-VARS :DECLARE-TYPES :PARENSCRIPT
 :SBCL+SAFE-STANDARD-READTABLE :NAMED-READTABLES :LPARALLEL :21BIT-CHARS
 :CHUNGA :CLOSER-MOP :CL-PPCRE-UNICODE :FLEXI-STREAMS :CL-UNICODE :CL-PPCRE
 :CL-JSON-DOUBLE-FLOAT-IS-SUBSUMED :CL-JSON-SINGLE-FLOAT-IS-SUBSUMED
 :BORDEAUX-THREADS :LPARALLEL.WITH-CLTL2 :LPARALLEL.WITH-CAS
 :LPARALLEL.WITH-STEALING-SCHEDULER :SPLIT-SEQUENCE CHIPZ-SYSTEM:GRAY-STREAMS
 CFFI-FEATURES:FLAT-NAMESPACE CFFI-FEATURES:X86-64 CFFI-FEATURES:UNIX :CFFI
 CFFI-SYS::FLAT-NAMESPACE ALEXANDRIA::SEQUENCE-EMPTYP :FAST-IO-SV :FAST-IO
 :SBCL-USES-SB-ROTATE-BYTE :CL-JSON-CLOS :CL-JSON :THREAD-SUPPORT :ASDF3.3
 :ASDF3.2 :ASDF3.1 :ASDF3 :ASDF2 :ASDF :OS-UNIX :NON-BASE-CHARS-EXIST-P
 :ASDF-UNICODE :X86-64 :GENCGC :64-BIT :ANSI-CL :COMMON-LISP :ELF
 :IEEE-FLOATING-POINT :LINUX :LITTLE-ENDIAN :PACKAGE-LOCAL-NICKNAMES
 :SB-CORE-COMPRESSION :SB-LDB :SB-PACKAGE-LOCKS :SB-THREAD :SB-UNICODE :SBCL
 :UNIX)

ASDF version: 3.3.1
ASDF registries: (NYXT-SOURCE-REGISTRY ENVIRONMENT-SOURCE-REGISTRY)
Critical dependencies: (/home/nselm/nyxt/src/nyxt/_build/cl-cffi-gtk/gtk/cl-cffi-gtk.asd
 /home/nselm/nyxt/src/nyxt/_build/cl-gobject-introspection/cl-gobject-introspection.asd
 /home/nselm/nyxt/src/nyxt/_build/cl-webkit/webkit2/cl-webkit2.asd)

[aartaka]

What do you mean by selection filters?

[Nselm]

That it has a lot of possible suggestions that it is trying to filter according to my input. (I mean the fuzzy matching the prompt buffer does against possible suggestions).
Things to note:

1. I have tried deleting all my bookmarks, but the issue still occurs.
2. I have tried this on a "real" PC with a little bit of RAM and running a non-modified version of linux mint, and everything works (so far) fine.
   As a result I think - if nobody else has this problem - that it is probably a fairly niche issue with my setup or my laptop - though I probably wouldnt know if it were not.

[Ambrevar]

@Nselm Can you try reproducding this on master? It may have been fixed already.

[jmercouris]

Indeed it is very possible it has been fixed on master. There was some work done recently for a "headless" implementation which fixes some very fast typing/concurrency issues that we potentially did not catch.

[Nselm]

I have tried out master and it seems to work.
However:

1. When typing very fast (esp. deleting, since in this case I just hold down the key) there are still noticeable lags
2. When holding down any key for a longer time, it still hangs.
   When I use nyxt normally, those problems do not occur, since I wont press the same key for longer than a few seconds,
   so for any practical purposes the problem seems to be fixed.
   For reference, the values xset -q gives me:
   auto repeat delay: 240 repeat rate: 50

[Ambrevar]

We've got an important bug here: I can indeed reproduce the freeze when I keep a key pressed or if I type very fast.

For future reference, to reproduce @Nselm settings, type

xset r rate 240 50

Then keep a key pressed in the set-url prompter.

I could not reproduce in execute-command though, so it seems that the race condition occurs on CPU-intensive filters.

[trev-dev]

Just my totally unhelpful 2 cents here but I believe this bug happens frequently enough for me that I do not have enough confidence in Nyxt to really use it. I can play with it for a while but the expectation is for it to ultimately crash. I can type slow, I can type fast, I can hold buttons down or I could not and it will still crash at the command prompt. This makes Nyxt an unproductive choice for me.

If there is a reasonable temporary work-around to be had I would implement it.

[Ambrevar]

I've been working on it, I'll release a fix before 3.0, don't worry :)

[aartaka]

So, is it fixed? Right now prompt buffer emits "Calculating..." message on new input. Is that the fix in action?

[Ambrevar]

No no no, nothing is fixed yet, it will come with the concurrency overhaul.

[timthelion]

I presume this freeze is the same issue? I see an error at the bottom of the screen as well as the obvious total freeze (see the overdraw from my launcher...).

[jmercouris]

Hello @timthelion you are using a rather old version of Nyxt, can you please try on latest 3.0.0 :-)?

[Ambrevar]

Note that the original post issue is not fixed on 3.0.0.

[odanoburu]

Can confirm I still get freezes fairly often. My machine is otherwise fast, RAM is far from exhausted, and I have been trying out Nyxt (3rd attempt) for just a day, so not a long history nor long list of bookmarks etc.

[aadcg]

@odanoburu do you have a detailed recipe to see it crashing? I still can't reproduce, even when following the guide above.

Also, how did you install Nyxt?

[odanoburu]

I installed using flatpak, and the guide above does cause the issue for me (I wasn't doing anything much different, just typing fast instead of holding down a key). But note that it does not crash, it just hangs for some time.

I'm compiling from source now to see if I can still reproduce it.

[aadcg]

Interesting! I can reproduce it on the Flatpak, but not when I run Nyxt installed via Guix. That gives us a new piece of information. I'll think about it.

The SBCL version differs (2.3.7 vs 2.3.10), but I doubt it plays a role. WebKitGTK differs too (2.40.5 vs 2.42.1). I've sent a patch to Guix that updates WebKitGTK, so soon I'll be able to test it.

[odanoburu]

Interestingly, I can reproduce the problem when compiling from source, although I didn't use Guix, just the makefile.

I get a warning on the minibuffer (and on the terminal output) that I don't get on the flatpak, but of course those are different versions:

<WARN> [15:54:35] Warning: Error while completing default search: The alien function &quot;SSL_get1_peer_certificate&quot; is undefined.

I'm using SBCL 2.3.6, webkitgtk 2.42.2.

[aadcg]

Interesting. Which command did you invoke to see this error message? set-url?

[odanoburu]

Indeed. My openssl version is 3.1.1, so it should have this function. Not sure of the version used in the flatpak though.

[odanoburu]

I think the hanging issue is related to garbage collection: increasing the the amount of memory that will be allocated before the next garbage collection is initiated (by setting sb-ext:bytes-consed-between-gcs) seems to delay the hanging — although of course when it does come it hangs even longer because there is more garbage to collect. I also tried decreasing this value so that the GC would run more often (but taking less time), but I not could find a value that would stop the hanging. SBCL's GC is not low-latency, after all. Although there are efforts towards a better GC:

https://applied-langua.ge/~hayley/swcl-gc.pdf (parallel GC in SBCL >2.3.8)
https://github.com/sbcl/sbcl/blob/master/doc/internals-notes/arena-allocation.txt (arena allocation reduces the need for garbage collection)
https://groups.google.com/g/sbcl-devel/c/VAA4SkQo3jM (work on low latency GC)

Not sure how much the nyxt team has looked into this, I can open a separate issue or discussion if you prefer. The simplest attempt at a solution (that does not require waiting for SBCL development) would be to trigger the GC manually very frequently, and running a full GC run when the there is no user input.

[aadcg]

@tomterl are the freezes related to the prompt buffer?

[tomterl]

@tomterl are the freezes related to the prompt buffer?

It would appear so, but it is hard to tell, as almost any interaction with Nyxt, other than scrolling, involves the prompt buffer - or am I understanding the term wrong? One freeze occured after executing make-window from the comand prompt (:)*, a couple more after editing the url both o and O and pressing the final return.

*: The new window was created, but blank and no further interaction was possible. I never had the patience to wait if the process would become reponsive again - I'll be more patient and see if an 'unfreeze' happens.

[odanoburu]

@aadcg This will come as no surprise to you, but I tested Nyxt on my old machine and indeed I cannot reproduce the freezing there. Nyxt is the only application that freezes so on my current machine though, so even if it's not related to Nyxt I still can't make it my daily driver, despite how much I'd like to do so. I tried increasing the delay between key repeats and decreasing their speed (I'm on Wayland so I couldn't run your command directly), but Nyxt still freezes. As I said before, holding the key down is just the easiest way of reproducing the issue, but if I just type at regular speed — and I don't don't type super fast — Nyxt will hang.

[tomterl]

Despite setting DEBUGINFOD_URLS="https://debuginfod.archlinux.org/" and waiting for my slow connection to fetch a metric ton of debug information, the traces I got from gdb are lacking a lot of details.

Nevertheless: https://paste.sr.ht/~tomterl/1d6be5cc3b86e6e67e9b890f11a5333ea9b100ab

Maybe something catches your eye - I'll try to get the correct symbols somehow, to make more details visible.

(Less detail missing?? https://paste.sr.ht/~tomterl/672e608a1098a8780bac1a2782a8b168a9caf7b4)

[aadcg]

It would appear so, but it is hard to tell, as almost any interaction with Nyxt, other than scrolling, involves the prompt buffer - or am I understanding the term wrong?

@tomterl all clear, thanks. I asked because some crashes are due to WebKitGTK struggling to handle heavy pages like YouTube. This issue is about crashes while using the prompt buffer (set URL, etc), so we're on the same page.

---

This will come as no surprise to you, but I tested Nyxt on my old machine and indeed I cannot reproduce the freezing there.

@odanoburu Right, no surprises here. Occasional crashes may happen due to WebKitGTK, but it is not acceptable if Nyxt crashes often when using the prompt buffer. I'd love to understand why, i.e. why you can't reproduce in your old machine. Btw, are you running Wayland in the old machine? Also, how did you install Nyxt? Let's try to narrow it down by answering the question "what differs between your new and older machine that could impact Nyxt's performance?".

[aadcg]

@tomterl thanks for the investigation! Nothing catches my eye. What executable did you use in the debugger - a local build or via Flatpak?

[tomterl]

@tomterl thanks for the investigation! Nothing catches my eye. What executable did you use in the debugger - a local build or via Flatpak?

A local build

[odanoburu]

Occasional crashes may happen due to WebKitGTK, but it is not acceptable if Nyxt crashes often when using the prompt buffer.

Yeah, unfortunately it freezes even when I'm trying to set-url for the first time, with no pages open and without loading any configuration.

I'd love to understand why, i.e. why you can't reproduce in your old machine.

It's pretty weird, isn't it? I'm running Wayland on both machines, yes. I installed Nyxt from the flatpak in the old machine, in the new one I tried both flatpak and source-built versions.

I guess my next step would be running Debian on my new machine — it currently runs Fedora. If that works, it's either Fedora-related or some customization I made (could it be using IBus as my IME?). I have recently installed Debian on the old machine and I haven't configured everything yet, so it's pretty close to out-of-the-box Debian.

[aadcg]

It's pretty weird, isn't it?

I can't possibly make sense of it, honestly. Could a program installed via Flatpak depend on the distro used? I doubt it. IBus or IME? I'd say it's unrelated too. What about differences in terms of hardware, especially the GPU?

[odanoburu]

Both have integrated Intel GPUs. The new machine sometimes has driver issues because the hardware is a bit too new (the camera still doesn't work reliably or fully under Linux), so it might be something like this, although I have never experienced any freezes with any other app.

[aadcg]

@odanoburu can you freeze Nyxt when invoking execute-command (Ctrl-space), or does it only happen on set-url?

[tomterl]

The only thing every platform/release-format I tried had in common is a tiling WM - Distros: Ubuntu 18.04 up to 22.04 (same machine, occasional LTS Updates), ubuntu and then manjaro (arch based) on another, and now ubuntu on my new workstation at work - ranging from nvidia GPU card, internal intel GPU;

[odanoburu]

@aadcg I can't freeze it by simply pressing a key like I can on set-url, no. Which is interesting…

[aadcg]

Then it's the network. Please give this suggestion a try.

[odanoburu]

That solved the freezing, but it is strange that it didn't happen on my old machine (using the same network). I used to get freezes all the time, not just when the network is acting up, although I guess what matters here is latency, not so much speed (maybe the servers being contacted are far from me). Or maybe the network driver of the new machine is bad.

Anyway, as you said on that issue, there should be a timeout for fetching the requests. And even if there isn't, the input shouldn't block because the requests should be asynchronous.

Thank you for the help and patience @aadcg, I can use Nyxt more extensively now!

[aadcg]

Good! We may not have solved all of the mysterious but we have made progress.

To close this issue means to fix precisely what you mention:

Anyway, as you said on that issue, there should be a timeout for fetching the requests. And even if there isn't, the input shouldn't block because the requests should be asynchronous.

[odanoburu]

Maybe one can reproduce the problem throttling one's network, using netem on Linux, there are some examples of how to use it on the man page. Out of curiosity, I'll try Nyxt without this patch whenever I'm on another network, and will also try to make more tests on my old machine to see if it ever reproduces this bug.

[odanoburu]

Simply doing

     (lambda (suggestions source input)
       (declare (ignore suggestions source))
       (input->queries input
-                      :check-dns-p t
+                      :check-dns-p nil
                       :engine-completion-p t)))
    (prompter:actions-on-return #'buffer-load*))
   (:export-class-name-p t)

seems to be enough to not freeze Nyxt on my machine.

[tomterl]

Setting :check-dns-p nil seems to be enough to prevent freezes on my end, too.