Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate/mitigate acquisition lock timeouts #1600

Open
melange396 opened this issue Feb 18, 2025 · 0 comments
Open

Investigate/mitigate acquisition lock timeouts #1600

melange396 opened this issue Feb 18, 2025 · 0 comments
Labels
acquisition changes acquisition logic bug devops building, running, deploying, environment stuff, handy utils, repository-related, engineer QoL, etc mysql mysql database related

Comments

@melange396
Copy link
Collaborator

There appears to be a conflicting interaction between covidcast acquisition and the "coverage_crossref_updater" that causes acquisition to encounter a lock timeout (mysql.connector.errors.DatabaseError: 1205 (HY000): Lock wait timeout exceeded; try restarting transaction) which in turn causes failure on some input files. The failing files can be re-acquired, but this takes manual intervention and is suboptimal.

We have noticed a pattern in timing of errors for some patching job runs that leads to this hypothesis (see slack thread for more detail and links to logs). It makes sense because the updater has a long-running query that is reading from the same table that acquisition writes into.

Look into this to verify that it is indeed the case, and if so, figure out a way around it (which could just be "figure out how to prevent these jobs from running concurrently in our scheduler").

@melange396 melange396 added acquisition changes acquisition logic bug devops building, running, deploying, environment stuff, handy utils, repository-related, engineer QoL, etc mysql mysql database related labels Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
acquisition changes acquisition logic bug devops building, running, deploying, environment stuff, handy utils, repository-related, engineer QoL, etc mysql mysql database related
Projects
None yet
Development

No branches or pull requests

1 participant