feat: ADD rate-limiting functionality#1396
Conversation
|
Thanks for opening this pull request! Please check out our contributing guidelines! |
mattclegg
changed the title
|
|
||
| private static boolean isRateLimited(HttpClientErrorException exception) { | ||
| final boolean rateLimitStatus = | ||
| FORBIDDEN.equals(exception.getStatusCode()) |
There was a problem hiding this comment.
429 is only rate limit status code, isn't it? FORBIDDEN is slightly different
There was a problem hiding this comment.
Unfortunately GitHub API will return either; https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2026-03-10#exceeding-the-rate-limit
The x-ratelimit-remaining header is used to determine if it's rate limit related.
| } | ||
| log.warn("GitHub rate limit hit; waiting {}s until reset", waitFor.toSeconds()); | ||
| try { | ||
| Thread.sleep(waitFor.toMillis()); |
There was a problem hiding this comment.
Wouldn't it hang the complete system? also given waitUntilRateLimitResets will be called multiple times, multiple Threads will go on sleep mode. Ideally if the rate-limit got hit, then we should skip the remaining calls
There was a problem hiding this comment.
It wasn't hanging for me but just waiting until the limit was reset. waitFor is defined based on the value returned from the GitHub API.
For my use-case I wanted to ensure I had all the results and less concerned how long it took to wait for relevant rate limits to reset. If you think it might be useful to abort all requests once the limit is reached I can add that as an option?
2 participants
I was getting blocked when querying a large number of repositories (~2k).
Any feedback welcome.
Thanks