Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Receiving HTTP Error 429 #19

Open
tsepton opened this issue May 5, 2022 · 1 comment
Open

Receiving HTTP Error 429 #19

tsepton opened this issue May 5, 2022 · 1 comment

Comments

@tsepton
Copy link

tsepton commented May 5, 2022

When trying to execute the script for the third of fourth time in a row, I get an HTTP Error 429 from urllib.
Here is the full traceback :

Traceback (most recent call last):
  File "extract_occurrences.py", line 75, in <module>
    html = get_range(search_term, start_date, end_date)
  File "extract_occurrences.py", line 51, in get_range
    num_results, success = get_num_results(search_term, date, date)
  File "extract_occurrences.py", line 19, in get_num_results
    handler = opener.open(request)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

I didn't have a look a the code, or at the header of the request response, but putting a waiting time between each requests should fix this

@mason-lab
Copy link

I've also encountered this issue, was hoping someone has a code snippet I could insert to help fix this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants