SSLError: certificate verify failed

[wonderingwombat]

Hi @cebtenzzre,

Thanks for your response on my issue on the other repo. I've moved everything across to your fork but a different problem has cropped up. I get the following error for all the sites I try:

[wget] URL is https://66.media.tumblr.com/tumblr_DETAIL.jpg [wget] Error retrieving resource: urllib3 reached a retry limit. Caused by: WGHTTPSConnectionPool(host='66.media.tumblr.com', port=443): Max retries exceeded with url: /tumblr_DETAIL.jpg (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

I'm guessing this might not be related to your script at all. I would appreciate it if you could point me in the right direction to try and resolve this. I'm on OSX 10.14.6. I have tried running the wget command by itself and that seems to work fine.

Please let me know if you need more information.

[cebtenzzre]

Thanks for trying my fork! The relevant code was inspired by wget but it actually uses urllib3 under the hood. This error has the same meaining as the one in bbolli#225, see my comment for more info.

I just reviewed the code and pushed some changes that should either fix the problem or print a warning if something isn't working. If you still get those errors with the latest script, show me the lines it prints at the top that start with Warning: (if any), and give me the output of this one-liner so I can better understand how your Python is configured:

python -c 'from __future__ import print_function; import ssl, sys; print("Python " + sys.version + " on " + sys.platform, ssl.OPENSSL_VERSION, hex(ssl.OPENSSL_VERSION_NUMBER), getattr(ssl, "HAS_SNI", None), sep="\n")'

[wonderingwombat]

Thanks for taking the time to reply. Here is the result of the command:

Python2:

Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 12:54:16) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
OpenSSL 0.9.8zh 14 Jan 2016
0x9081dfL
True

Python3:

Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) 
[Clang 6.0 (clang-600.0.57)] on darwin
OpenSSL 1.1.1g  21 Apr 2020
0x1010107f
True

I was able to get your version running with the -S flag. I was wondering if there is any downside to using the -S flag and disabling SSL certificate verification?

Thanks again for your help.

[cebtenzzre]

Using -S to disable certificate validation means you could be vulnerable to a man-in-the-middle attack, even for HTTPS connections, so it's not a good long-term fix - even if you aren't necessarily sending or receiving sensitive information.

Which Python version have you been using to run the script? Your Python 2 install is linked to an old OpenSSL version that doesn't support TLS 1.2 and might have other issues, but your Python 3 install looks relatively up-to-date (though 3.8.8 would be better). This fork supports both versions, but Python 3 is recommended because Python 2 is no longer being updated.

[wonderingwombat]

I'm using python 3 to run the script since python 2 looks like a mess on OSX.

[cebtenzzre]

I looked into the specific exception you're getting, and it turns out it comes from urllib3's pyOpenSSL wrapper (urllib3.contrib.pyopenssl). I actually get the same exception if I inject pyOpenSSL on my system, so pyOpenSSL seems to have certificate verification issues - that's not a bug in tumblr-utils.

But pyOpenSSL shouldn't even be in use, since both of your Python versions report HAS_SNI as True. The standard ssl module would work just fine, so we should find out why pyOpenSSL is getting injected.

Run this to make sure you have urllib3 installed:

python3 -m pip install urllib3

Run this to find where urllib3.contrib.pyopenssl lives:

python3 -c 'from urllib3.contrib import pyopenssl; print(pyopenssl.__file__)'

It will print a path like /path/to/pyopenssl.py. Open that file in a text editor and add a raise ... line to inject_into_urllib3 so it looks like this:

def inject_into_urllib3():
    "Monkey-patch urllib3 with PyOpenSSL-backed SSL-support."

    raise RuntimeError('Somebody called me')
    _validate_dependencies_met()

Then save that file and try to backup some blog with tumblr_backup.py, and it will hopefully print a traceback and exit before you get any SSL errors.

[wonderingwombat]

So I added the raise line and when I try and backup a blog I get the following:

Traceback (most recent call last):
  File "tumblr_backup.py", line 96, in <module>
    import requests
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/__init__.py", line 96, in <module>
    pyopenssl.inject_into_urllib3()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 117, in inject_into_urllib3
    raise RuntimeError('Somebody called me')
RuntimeError: Somebody called me

I get the following response when I run the pip command:
Requirement already satisfied: urllib3 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (1.25.9)

[cebtenzzre]

Ah, apparently requests versions prior to v2.24.0 (released Jun 17, 2020) lack commit psf/requests@db47b9b and inject pyOpenSSL unconditionally. python3 -m pip install -U requests should update requests to a version that won't inject it on your platform. Then you can remove the raise RuntimeError... line from pyopenssl.py and (hopefully) use this fork's tumblr_backup.py without -S.

[wonderingwombat]

Awesome, that did the trick! Thanks so much for helping me resolve this.