-
Notifications
You must be signed in to change notification settings - Fork 216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
duckdb cache miss #2870
Comments
Some small chance this helps with PRQL#2870 (which is becoming a bottleneck on iterations, I'm finding)
If we can't fix this, we could remove integration testing from the PR CI, and run it only on |
Is it possible to remove duckdb-rs from the default development dependencies? |
ref #2899 |
An update here — lots of work happened on this, but unresolved as of yet. |
This can be closed? |
Yes! |
(I'm not quite sure how this resolved, but thank you to everyone who spent time on it — it was quite an effort in the end. Very happy it's working now) |
What's up?
I spent some time trying to work out what was going on. I have a hypothesis — that
~/.cargo/registry/src
is required forduckdb
, but no other crates, and the cache action removes it.No need to read the full description below unless you're particularly interested; I'll post a reference to this on the rust cache repo.
Our build now takes 2-3x longer, because
duckdb-0.8.0
needs to be re-compiled, even when the build has successfully cached.I thought that the cache action might be deleting important files. So I tried reproducing the deletions it makes. It's a JS app, so I couldn't easily run it locally, so instead I ran a debug build of the job at https://github.com/PRQL/prql/actions/runs/5316798564/jobs/9627453030, and downloaded the file as
deletions.log
.I then ran this to delete the equivalent paths on my system.
...after doing this, the problem doesn't reproduce — it compiles quickly.
BUT! There is another path that does seem to matter though! Specifically removing
~/.cargo/registry/src
means we can reproduce the slow compilation:...so this is likely the culprit.
An aside — I confirmed that this ordering doesn't matter — running each one after
cargo clean
:cargo build -p prql-compiler && cargo clippy -p prql-compiler
takes 28scargo clippy -p prql-compiler && cargo build -p prql-compiler
also takes 28s,The text was updated successfully, but these errors were encountered: