Random RecursionError after a while

[seranpion]

Hello.

I have now twice ran into this critical exception while running the script:

An unexpected exception of type RecursionError occurred. Arguments:
('maximum recursion depth exceeded while calling a Python object',)

Nothing else to pinpoint the cause.

This happens after the script has run a long time.
I am running Python for windows version 3.9.150.1013 on windows 10.

Other python programs do not trigger this, so I suspect that the script itself causes it.
I will be running the script with the debug verbosity now, in case it happens again.

[rbignon]

Hello,

With only this log it is difficult to understand what occurs. Please update doctoshotgun to get the latest commit which will print the backtrace if you run it with -d.

[Carlotronics]

Hello,

I just encountered the same problem. Here are the last logs and stacktrace:

$ ./doctoshotgun.py --debug --pfizer --only-third --start-date 13/12/2021 --end-date 22/12/2021 fr 'city_with_accent,city,city_with_spaces_and_accents,city,city' <email> '<password>'
[...]
2021-12-13 15:58:52,759:INFO:browser::browsers.py:369:save_response Response saved to 9d351aa014634061a5eddd0f51095444
2021-12-13 15:58:52,760:DEBUG:browser::browsers.py:880:internal_callback Handle https://www.doctolib.fr/search_results/4904596.json?limit=4&ref_visit_motive_ids%5B%5D=8192&speciality_id=5494&search_result_format=json with CenterResultPage
2021-12-13 15:58:52,760:DEBUG:root::doctoshotgun.py:868:main Skipping city 'Paris' Pharmacie Mirabeau Lellouche
An unexpected exception of type RecursionError occurred. Arguments:
('maximum recursion depth exceeded while calling a Python object',)
Traceback (most recent call last):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 866, in main
    for center in docto.find_centers(cities, motives):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 357, in find_centers
    for center in self.find_centers(where, motives, next_page):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 357, in find_centers
    for center in self.find_centers(where, motives, next_page):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 357, in find_centers
    for center in self.find_centers(where, motives, next_page):
  [Previous line repeated 969 more times]
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 323, in find_centers
    self.centers.go(where=city, params={
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/url.py", line 98, in go
    r = self.browser.location(self.build(**kwargs), params=params, data=data, json=json, method=method, headers=headers or {})
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 911, in location
    response = self.open(*args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 898, in open
    return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 790, in open
    return super(DomainBrowser, self).open(req, *args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 531, in open
    response = self.session.send(preq,
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 76, in send
    resp = super().send(*args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 339, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "/usr/lib/python3.9/http/client.py", line 238, in parse_headers
    return email.parser.Parser(_class=_class).parsestr(hstring)
  File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 56, in parse
    feedparser.feed(data)
  File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
    self._call_parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 295, in _parsegen
    if self._cur.get_content_maintype() == 'message':
  File "/usr/lib/python3.9/email/message.py", line 594, in get_content_maintype
    ctype = self.get_content_type()
  File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/_policybase.py", line 316, in header_fetch_parse
    return self._sanitize_header(name, value)
  File "/usr/lib/python3.9/email/_policybase.py", line 287, in _sanitize_header
    if _has_surrogates(value):
  File "/usr/lib/python3.9/email/utils.py", line 57, in _has_surrogates
    s.encode()
RecursionError: maximum recursion depth exceeded while calling a Python object

2021-12-13 15:58:53,351:DEBUG:browser::browsers.py:1095:dump_state Stored cookies into storage

I started the script at ~14:06 CET so it ran near 2 hours before crashing. The current commit is 8c22878 (on which master currently points).

I am running Python 3.9.9 and installed following packages:

Babel==2.9.1
certifi==2021.10.8
charset-normalizer==2.0.9
cloudscraper==1.2.58
colorama==0.4.4
cssselect==1.1.0
html2text==2020.1.16
idna==3.3
lxml==4.6.5
Pillow==8.4.0
pyparsing==3.0.6
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
requests==2.26.0
requests-toolbelt==0.9.1
six==1.16.0
termcolor==1.1.0
Unidecode==1.3.2
urllib3==1.26.7
woob==3.0