UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 27673: invalid continuation byte

[sentry-io]

Sentry Issue: PERMA-B

This has been caught 92 times already including 36 times in the last 24 hours. Evidently, it's not uncommon for robots.txt not to be utf-8 encoded: let's see if we can be smarter about this, even if it doesn't cause entire captures to fail.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 27673: invalid continuation byte
  File "threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "perma/tasks.py", line 613, in robots_txt_thread
    content = str(robots_txt_response.content, 'utf-8')