🔖 Release 2.2.900

See CHANGES.rst for more details
jawah · Nov 2, 2023 · 59f89e9 · 59f89e9
1 parent 8b6d9c9
commit 59f89e9
Show file tree

Hide file tree

Showing 56 changed files with 2,015 additions and 498 deletions.
diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml
@@ -1,3 +1,2 @@
-tidelift: pypi/urllib3
-github: urllib3
-open_collective: urllib3
+github:
+  - Ousret
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
@@ -1,11 +1,8 @@
 blank_issues_enabled: false
 contact_links:
   - name: 📚 Documentation
-    url: https://urllib3.readthedocs.io
+    url: https://urllib3future.readthedocs.io
     about: Make sure you read the relevant docs
   - name: ❓ Ask on StackOverflow
     url: https://stackoverflow.com/questions/tagged/urllib3
     about: Ask questions about usage in StackOverflow
-  - name: 💬 Ask the Community
-    url: https://discord.gg/CHEgCZN
-    about: Join urllib3's Discord server
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,8 +1,8 @@
 <!---
-Thanks for your contribution! ♥️
+Hello!
 
-If this is your first PR to urllib3 please review the Contributing Guide:
-https://urllib3.readthedocs.io/en/latest/contributing.html
+If this is your first PR to urllib3.future please review the Contributing Guide:
+https://urllib3future.readthedocs.io/en/latest/contributing.html
 
 Adhering to the Contributing Guide means we can review, merge, and release your change faster! :)
 --->
diff --git a/.github/PULL_REQUEST_TEMPLATE/release.md b/.github/PULL_REQUEST_TEMPLATE/release.md
@@ -1,19 +1,8 @@
 * [ ]  See if all tests, including integration, pass
-* [ ]  Get the release pull request approved by a [CODEOWNER](https://github.com/urllib3/urllib3/blob/main/.github/CODEOWNERS)
+* [ ]  Get the release pull request approved by a [CODEOWNER](https://github.com/jawah/urllib3.future/blob/main/.github/CODEOWNERS)
 * [ ]  Squash merge the release pull request with message "`Release <VERSION>`"
-* [ ]  Tag with X.Y.Z, push tag on urllib3/urllib3 (not on your fork, update `<REMOTE>` accordingly)
-  * Notice that the `<VERSION>` shouldn't have a `v` prefix (Use `1.26.6` instead of `v.1.26.6`)
-  * ```
-    git tag -s -a '<VERSION>' -m 'Release: <VERSION>'
-    git push <REMOTE> --tags
-    ```
-* [ ]  Execute the `publish` GitHub workflow. This requires a review from a maintainer.
+* [ ]  Create a Github Release
 * [ ]  Ensure that all expected artifacts are added to the new GitHub release. Should
        be one `.whl`, one `.tar.gz`, and one `multiple.intoto.jsonl`. Update the GitHub
        release to have the content of the release's changelog.
-* [ ]  Announce on:
-  * [ ]  Twitter
-  * [ ]  Discord
-  * [ ]  OpenCollective
 * [ ]  Update Tidelift metadata
-* [ ]  If this was a 1.26.x release, add changelog to the `main` branch
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -47,7 +47,7 @@ jobs:
           - macos-latest
           - windows-latest
           - ubuntu-20.04  # OpenSSL 1.1.1
-          - ubuntu-22.04  # OpenSSL 3.0
+          - ubuntu-latest  # OpenSSL 3.0
         nox-session: ['']
         include:
           - experimental: false
@@ -89,7 +89,7 @@ jobs:
             os: ubuntu-22.04
 
     runs-on: ${{ matrix.os }}
-    name: ${{ fromJson('{"macos-latest":"macOS","windows-latest":"Windows","ubuntu-latest":"Ubuntu","ubuntu-20.04":"Ubuntu 20.04 (OpenSSL 1.1.1)","ubuntu-22.04":"Ubuntu 22.04 (OpenSSL 3.0)"}')[matrix.os] }} ${{ matrix.python-version }} ${{ matrix.nox-session }}
+    name: ${{ fromJson('{"macos-latest":"macOS","windows-latest":"Windows","ubuntu-latest":"Ubuntu","ubuntu-20.04":"Ubuntu 20.04 (OpenSSL 1.1.1)","ubuntu-latest":"Ubuntu Latest (OpenSSL 3+)"}')[matrix.os] }} ${{ matrix.python-version }} ${{ matrix.nox-session }}
     continue-on-error: ${{ matrix.experimental }}
     timeout-minutes: 40
     steps:
@@ -208,7 +208,7 @@ jobs:
                   - --log.level=INFO
             
               httpbin:
-                image: mccutchen/go-httpbin:v2.10.0
+                image: mccutchen/go-httpbin:v2.11.1
                 restart: unless-stopped
                 depends_on:
                   proxy:

diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
@@ -14,7 +14,7 @@ jobs:
       fail-fast: false
       matrix:
         downstream: [botocore, niquests]
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-latest
     timeout-minutes: 30
 
     steps:

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -17,7 +17,7 @@ repos:
       - id: isort
 
   - repo: https://github.com/PyCQA/flake8
-    rev: 6.0.0
+    rev: 6.1.0
     hooks:
       - id: flake8
         additional_dependencies: [flake8-2020]
diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,3 +1,67 @@
+2.2.900 (2023-11-01)
+====================
+
+- Added support for in-memory client (intermediary) certificate to be used with mTLS.
+  This feature compensate for the complete removal of ``pyOpenSSL``. Unfortunately it is only
+  available on Linux, OpenBSD, and FreeBSD. Using newly added ``cert_data`` and ``key_data`` arguments
+  in ``HTTPSConnection`` and ``HTTPSPoolConnection`` you will be capable of passing the certificate along with
+  its key without getting nowhere near your filesystem.
+  MacOS and Windows are not concerned by this feature when using HTTP/1.1, and HTTP/2 with TLS over TCP.
+- Removed remnant ``SSLTransport.makefile`` as it was built to circumvent a legacy constraint when urllib3 depended upon
+  ``http.client``.
+- Bumped minimum requirement for ``qh3`` to version 0.13.0 in order to support in-memory client certificate (mTLS).
+- Symbolic complete detachment from ``http.client``. Removed all references and imports to ``http.client``. Farewell!
+- Changed the default ciphers in default SSLContext for an **increased** security level.
+  *Rational:* Earlier in v2.1.901 we initialized the SSLContext ciphers with the value ``DEFAULT`` but after much
+  consideration, after we saw that the associated ciphers (e.g. ``DEFAULT`` from OpenSSL) includes some weak suites
+  we decided to inject a rather safer and limited cipher suite. It is based on https://ssl-config.mozilla.org
+  Starting now, urllib3.future will match Mozilla cipher recommendations (intermediary) and will regularly update the suite.
+- Added support for multiplexed connection. HTTP/2 and HTTP/3 can benefit from this.
+  urllib3.future no longer blocks when ``urlopen(...)`` is invoked using ``multiplexed=True``, and return
+  a ``ResponsePromise`` instead of a ``HTTPResponse``. You may dispatch as much requests as the protocol
+  permits you (concurrent stream) and then retrieve the response(s) using the ``get_response(...)``.
+  ``get_response(...)`` can take up to one kwarg to specify the target promise, if none specified, will retrieve
+  the first available response. ``multiplexed`` is set to False by default and will likely be the default for a long
+  time.
+  Here is an example::
+
+    from urllib3 import PoolManager
+
+    with PoolManager() as pm:
+        promise0 = pm.urlopen("GET", "https://pie.dev/delay/3", multiplexed=True)
+        # <ResponsePromise 'IOYTFooi0bCuaQ9mwl4HaA==' HTTP/2.0 Stream[1]>
+        promise1 = pm.urlopen("GET", "https://pie.dev/delay/1", multiplexed=True)
+        # <ResponsePromise 'U9xT9dPVGnozL4wzDbaA3w==' HTTP/2.0 Stream[3]>
+        response0 = pm.get_response()
+        # the second request arrived first
+        response0.json()["url"]  # https://pie.dev/delay/1
+        # the first arrived last
+        response1 = pm.get_response()
+        response1.json()["url"]  # https://pie.dev/delay/3
+
+  or you may do::
+
+    from urllib3 import PoolManager
+
+    with PoolManager() as pm:
+        promise0 = pm.urlopen("GET", "https://pie.dev/delay/3", multiplexed=True)
+        # <ResponsePromise 'IOYTFooi0bCuaQ9mwl4HaA==' HTTP/2.0 Stream[1]>
+        promise1 = pm.urlopen("GET", "https://pie.dev/delay/1", multiplexed=True)
+        # <ResponsePromise 'U9xT9dPVGnozL4wzDbaA3w==' HTTP/2.0 Stream[3]>
+        response0 = pm.get_response(promise=promise0)
+        # forcing retrieving promise0
+        response0.json()["url"]  # https://pie.dev/delay/3
+        # then pick first available
+        response1 = pm.get_response()
+        response1.json()["url"]  # https://pie.dev/delay/1
+
+  You may do multiplexing using ``PoolManager``, and ``HTTPSPoolConnection``. Connection upgrade
+  to HTTP/3 cannot be done until all in-flight requests are completed.
+  Be aware that a non-capable connection (e.g. HTTP/1.1) will just ignore the ``multiplexed=True`` setting
+  and act traditionally.
+- Connection are now released into their respective pool when the connection support multiplexing (HTTP/2, HTTP/3)
+  before the response has been consumed. This allows to have multiple response half-consumed from a single connection.
+
 2.1.903 (2023-10-23)
 ====================
 

diff --git a/README.md b/README.md
@@ -10,10 +10,9 @@
   <br><small>urllib3.future is as BoringSSL is to OpenSSL but to urllib3 (except support is available!)</small>
 </p>
 
-urllib3 is a powerful, *user-friendly* HTTP client for Python. urllib3.future goes beyond supported features while remaining
-mostly compatible.
-urllib3.future brings many critical features that are missing from the Python
-standard libraries:
+⚡ urllib3.future is a powerful, *user-friendly* HTTP client for Python. 
+⚡ urllib3.future goes beyond supported features while remaining compatible.
+⚡ urllib3.future brings many critical features that are missing from the Python standard libraries:
 
 - Thread safety.
 - Connection pooling.
@@ -22,10 +21,11 @@ standard libraries:
 - Helpers for retrying requests and dealing with HTTP redirects.
 - Support for gzip, deflate, brotli, and zstd encoding.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
+- Multiplexed connection.
 - Proxy support for HTTP and SOCKS.
 - 100% test coverage.
 
-urllib3 is powerful and easy to use:
+urllib3.future is powerful and easy to use:
 
 ```python
 >>> import urllib3
@@ -46,12 +46,11 @@ urllib3.future can be installed with [pip](https://pip.pypa.io):
 $ python -m pip install urllib3.future
 ```
 
-⚠️ Installing urllib3.future shadows the actual urllib3 package (_depending on installation order_) and you should
-carefully weigh the impacts. The semver will always be like _MAJOR.MINOR.9PP_ like 2.0.941, the patch node
-is always greater or equal to 900.
+⚠️ Installing urllib3.future shadows the actual urllib3 package (_depending on installation order_). 
+The semver will always be like _MAJOR.MINOR.9PP_ like 2.0.941, the patch node  is always greater or equal to 900.
 
 Support for bugs or improvements is served in this repository. We regularly sync this fork
-with the main branch of urllib3/urllib3.
+with the main branch of urllib3/urllib3 against bugfixes and security patches if applicable.
 
 ## Compatibility with downstream
 
@@ -64,15 +63,7 @@ python -m pip install requests
 python -m pip install urllib3.future
 ```
 
-| Package          | Is compatible? | Notes                                                                                                                                           |
-|------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| requests         | ✅              | Invalid chunked transmission may raises ConnectionError instead of ChunkedEncodingError. Use of Session() is required to enable HTTP/3 support. |
-| HTTPie           | ✅              | Require plugin `httpie-next` to be installed or wont be able to upgrade to HTTP/3 (QUIC/Alt-Svc Cache Layer)                                    |
-| pip              | 🛑             | Cannot use the fork because of vendored urllib3 v1.x                                                                                            |
-| openapigenerator | ✅              | Simply patch generated `setup.py` requirement and replace urllib3 to urllib3.future                                                             |
-
-Want to report an incompatibility? Open an issue in that repository.
-All projects that depends on listed *compatible* package should work as-is.
+We suggest using the package **Niquests** as replacement for **Requests**. It leverage urllib3.future capabilities.
 
 ## Documentation
 

diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -10,4 +10,3 @@ cryptography==39.0.2;implementation_name=="pypy" and implementation_version<"7.3
 cryptography==41.0.2;implementation_name!="pypy" or implementation_version>="7.3.10"
 backports.zoneinfo==0.2.1;python_version<"3.9"
 towncrier==21.9.0
-pytest-memray==1.4.0;python_version>="3.8" and python_version<"3.12" and sys_platform!="win32" and implementation_name=="cpython"
diff --git a/docs/advanced-usage.rst b/docs/advanced-usage.rst
@@ -688,3 +688,99 @@ It takes a ``set`` of ``HttpVersion`` like so:
 
     HTTP/3 require installing ``qh3`` package if not automatically available. Setting disabled_svn has no effect otherwise.
     Also, you cannot disable HTTP/1.1 at the current state of affairs.
+
+Multiplexed connection
+----------------------
+
+Since the version 2.2 you can emit multiple concurrent requests and retrieve the responses later.
+A new keyword argument is available in ``PoolManager``, ``HTTPPoolConnection`` through the following methods:
+
+- :meth:`~urllib3.PoolManager.request`
+- :meth:`~urllib3.PoolManager.urlopen`
+- :meth:`~urllib3.PoolManager.request_encode_url`
+- :meth:`~urllib3.PoolManager.request_encode_body`
+
+When you omit ``multiplexed=...`` it default to the old behavior of waiting upon the response and return a :class:`~urllib3.response.HTTPResponse`
+otherwise if you specify ``multiplexed=True`` it will return a :class:`~urllib3.backend.ResponsePromise` instead.
+
+Here is an example::
+
+    from urllib3 import PoolManager
+
+    with PoolManager() as pm:
+        promise0 = pm.urlopen("GET", "https://pie.dev/delay/3", multiplexed=True)
+        # <ResponsePromise 'IOYTFooi0bCuaQ9mwl4HaA==' HTTP/2.0 Stream[1]>
+        promise1 = pm.urlopen("GET", "https://pie.dev/delay/1", multiplexed=True)
+        # <ResponsePromise 'U9xT9dPVGnozL4wzDbaA3w==' HTTP/2.0 Stream[3]>
+        response0 = pm.get_response()
+        # the second request arrived first
+        response0.json()["url"]  # https://pie.dev/delay/1
+        # the first arrived last
+        response1 = pm.get_response()
+        response1.json()["url"]  # https://pie.dev/delay/3
+
+  or you may do::
+
+    from urllib3 import PoolManager
+
+    with PoolManager() as pm:
+        promise0 = pm.urlopen("GET", "https://pie.dev/delay/3", multiplexed=True)
+        # <ResponsePromise 'IOYTFooi0bCuaQ9mwl4HaA==' HTTP/2.0 Stream[1]>
+        promise1 = pm.urlopen("GET", "https://pie.dev/delay/1", multiplexed=True)
+        # <ResponsePromise 'U9xT9dPVGnozL4wzDbaA3w==' HTTP/2.0 Stream[3]>
+        response0 = pm.get_response(promise=promise0)
+        # forcing retrieving promise0
+        response0.json()["url"]  # https://pie.dev/delay/3
+        # then pick first available
+        response1 = pm.get_response()
+        response1.json()["url"]  # https://pie.dev/delay/1
+
+.. note:: You cannot expect the connection upgrade to HTTP/3 if all in-flight request aren't consumed.
+
+.. warning:: Using ``multiplexed=True`` if the target connection does not support it is ignored and assume you meant ``multiplexed=False``. It will raise a warning in a future version.
+
+Associate a promise to its response
+-----------------------------------
+
+When issuing concurrent request using ``multiplexed=True`` and want to retrieve
+the responses in whatever order they may come, you may want to clearly identify the originating promise.
+
+To identify with certainty::
+
+    from urllib3 import PoolManager
+
+    with PoolManager() as pm:
+        promise0 = pm.urlopen("GET", "https://pie.dev/delay/3", multiplexed=True)
+        # <ResponsePromise 'IOYTFooi0bCuaQ9mwl4HaA==' HTTP/2.0 Stream[1]>
+        promise1 = pm.urlopen("GET", "https://pie.dev/delay/1", multiplexed=True)
+        # <ResponsePromise 'U9xT9dPVGnozL4wzDbaA3w==' HTTP/2.0 Stream[3]>
+        response = pm.get_response()
+        # verify that response is linked to second promise
+        response.is_from_promise(promise0)
+        # True!
+        response.is_from_promise(promise1)
+        # False.
+
+In-memory client (mTLS) certificate
+-----------------------------------
+
+.. note:: Available since version 2.2
+
+Using newly added ``cert_data`` and ``key_data`` arguments in ``HTTPSConnection``, ``HTTPSPoolConnection`` and ``PoolManager``.
+you will be capable of passing the certificate along with its key without getting nowhere near your filesystem.
+
+.. warning:: When connected to a TLS over TCP, this is only supported with Linux, FreeBSD, and OpenBSD. When connected over QUIC (e.g. HTTP/3) it is broadly supported.
+
+This feature compensate for the complete removal of ``pyOpenSSL``.
+
+You may give your certificate to urllib3.future this way::
+
+    with HTTPSConnectionPool(
+        self.host,
+        self.port,
+        key_data=CLIENT_INTERMEDIATE_KEY,
+        cert_data=CLIENT_INTERMEDIATE_PEM,
+    ) as https_pool:
+        r = https_pool.request("GET", "/")
+
+.. note:: If your platform isn't served by this feature it will raise a warning and ignore the certificate.
diff --git a/docs/conf.py b/docs/conf.py
@@ -107,8 +107,14 @@
     ("py:class", "_TYPE_TIMEOUT"),
     ("py:class", "_TYPE_FIELD_VALUE"),
     ("py:class", "_TYPE_BODY"),
+    ("py:class", "_TYPE_TIMEOUT_INTERNAL"),
+    ("py:class", "_TYPE_FAILEDTELL"),
+    ("py:class", "_TYPE_FIELD_VALUE_TUPLE"),
+    ("py:class", "_TYPE_FIELDS"),
+    ("py:class", "_TYPE_ENCODE_URL_FIELDS"),
     ("py:class", "_HttplibHTTPResponse"),
     ("py:class", "_HttplibHTTPMessage"),
+    ("py:class", "Message"),
     ("py:class", "TracebackType"),
     ("py:class", "Literal"),
     ("py:class", "email.errors.MessageDefect"),
@@ -118,12 +124,15 @@
     ("py:class", "SSLTransportType"),
     ("py:class", "VerifyMode"),
     ("py:class", "_ssl._SSLContext"),
+    ("py:func", "ssl.wrap_socket"),
     ("py:class", "urllib3._collections.HTTPHeaderDict"),
     ("py:class", "urllib3._collections.RecentlyUsedContainer"),
     ("py:class", "urllib3._request_methods.RequestMethods"),
     ("py:class", "urllib3.contrib.socks._TYPE_SOCKS_OPTIONS"),
     ("py:class", "urllib3.util.timeout._TYPE_DEFAULT"),
-    ("py:class", "BaseHTTPConnection"),
+    ("py:class", "urllib3.util.request._TYPE_FAILEDTELL"),
+    ("py:class", "BaseHTTPResponse"),
+    ("py:class", "urllib3.response.BaseHTTPResponse"),
     ("py:class", "connection._TYPE_SOCKET_OPTIONS"),
     ("py:class", "urllib3.backend.httplib._PatchedHTTPConnection"),
     ("py:class", "urllib3.backend._base.LowLevelResponse"),

diff --git a/docs/index.rst b/docs/index.rst
@@ -12,18 +12,19 @@ urllib3
    contributing
    changelog
 
-urllib3 is a powerful, *user-friendly* HTTP client for Python.
+⚡ urllib3.future is a powerful, *user-friendly* HTTP client for Python.
+⚡ urllib3.future goes beyond supported features while remaining compatible.
 
-urllib3 brings many critical features that are missing from the Python
-standard libraries:
+⚡ urllib3.future brings many critical features that are missing from the Python standard libraries:
 
 - Thread safety.
 - Connection pooling.
-- Client-side TLS/SSL verification.
+- Client-side SSL/TLS verification.
 - File uploads with multipart encoding.
 - Helpers for retrying requests and dealing with HTTP redirects.
 - Support for gzip, deflate, brotli, and zstd encoding.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
+- Multiplexed connection.
 - Proxy support for HTTP and SOCKS.
 - 100% test coverage.
 
@@ -58,7 +59,7 @@ The :doc:`reference/index` documentation provides API-level documentation.
 License
 -------
 
-urllib3 is made available under the MIT License. For more details, see `LICENSE.txt <https://github.com/urllib3/urllib3/blob/master/LICENSE.txt>`_.
+urllib3.future is made available under the MIT License. For more details, see `LICENSE.txt <https://github.com/urllib3/urllib3/blob/master/LICENSE.txt>`_.
 
 Contributing
 ------------