perf: Multi-threaded directory walking with Async I/O prefetch queue

[kavix]

🎯 Goal

Replace serial filepath.Walk with a multi-threaded parallel directory scanner (using os.ReadDir concurrently per directory level) coupled with an async I/O read-ahead prefetch queue.

💡 Why

On modern NVMe drives and fast SSDs, serial directory traversal is IOPS-bound and underutilizes hardware parallelism. A multi-threaded walker traverses deeply nested directory trees 4–8x faster.

🛠️ Implementation Steps

1. Implement a lock-free parallel directory walker in internal/util/walker.go.
2. Use a worker pool of directory scanners that push discovered file jobs into a bounded channel.
3. Add double-buffered asynchronous file reading so I/O reading overlaps with SHA-256 hashing.
4. Benchmark directory walk times on large codebases (e.g. 50k+ files).