File picker is really slow on huge projects

[liquidev]

Summary

I just tried editing the Unreal Engine source code with Helix, and fing into an arbitrary file from my project. Unfortunately it seems like Helix indexes the entire source directory recursively from scratch every time you open the file picker, which results in unacceptably slow performance in a huge codebase like Unreal Engine's. (As of typing this I typed in the name of the file I'd like to open, and the file picker still hasn't indexed my file.)

This is also compounded by Helix indexing directories that are listed in the .ignore file. It seems like .ignore just results in files matching any of the globs being hidden from the view, rather than preventing recursion.

Reproduction Steps

1. Open a compiled Unreal Engine in Helix (since it has way more files than just the cloned source code)
2. f
3. Search for a file such as EdGraphPin.cpp

Helix log

Irrelevant since this is strictly a performance issue.

Platform

Windows

Terminal Emulator

Windows Terminal

Installation Method

winget

Helix Version

helix 24.3 (2cadec0)

[the-mikedavis]

The log isn't irrelevant, this line might be insightful:

helix/helix-term/src/ui/mod.rs

Line 218 in 31bcde3

log::debug!("file_picker init {:?}", Instant::now().duration_since(now));

Can you provide a simpler reproduction case - a different similarly large project? Cloning unreal engine is convoluted

[pascalkuthe]

Our transversal is currently not parallel because we sort entries alphabetically.

Windows has super slow IO. Anytime I see people having issues with IO speed like this it's Windows (if you are on WSL its even worse still) on Linux/macos it's never an issue for projects of practical size (I work on massive legacy codebases).

We can likely offer an option to disable the sorting and allow parallel transversal. It's not too useful since it's only a secondary tie breaker as soon as you enter a query.

Caching the filesystem does not make sense. For small directories it's not necessary and for large directories the memory consumption is prohibitive (you wouldn't want to cache $HOME or /).

And .ignore does prevent entering directories if a glob matches a directory

[liquidev]

With the amount of files in UE I suppose sorting entries above a certain number of results would not make too much sense anyways, so maybe it's a good solution. I wonder how much faster parallel directory traversal is on Windows, though.

And .ignore does prevent entering directories if a glob matches a directory

I do wonder then why sometimes the (running) text disappears for a while, and then resumes - could it be because it's hitting lots of ignored files within non-ignored subdirectories?

[kirawi]

I've created an issue to track sorting as a config option: #11021

[markstos]

Can you provide a simpler reproduction case - a different similarly large project?

The Linux kernel is often used when people need a huge project to test with.

I just tried untar'ing this tarball of 85,000 files and then opening hx in the directory:

https://git.kernel.org/torvalds/t/linux-6.10-rc2.tar.gz

It opened instantly. On Linux.

[RoloEdits]

@liquidev One thing you can try is the new dev drive (built on top of ReFS) windows 11 comes with (you didn't specify windows version, so assuming 11). I've noticed various I/O improvements myself. Your milage may vary.

link: https://learn.microsoft.com/en-us/windows/dev-drive/

[liquidev]

Unfortunately my work computer runs Windows 10 for now so I'm unable to use Dev Drive. I assume it would improve the performance somewhat, I wonder just how much exactly.

I tried opening the Unreal Engine Git repository on Linux though, and although it has way less files than a full Unreal Engine game, the entire directory was indexed in just over a second. So that's way better already, but unfortunately "just use Linux" is not a solution that's viable for work.

[markstos]

WSL2 is supported in Windows 10. I've heard it performs well. https://learn.microsoft.com/en-us/windows/wsl/compare-versions