🔖 Release 3.9.0 (#148)

**Added**
- Support for WebSocket over HTTP/1, HTTP/2 and HTTP/3. It brings a
unified API that makes you leverage our powerful features like Happy
Eyeballs, SOCKS/HTTP/HTTPS proxies, thread/task safety etc...
- Hook for catching early responses like "103 Early Hints".

**Fixed**
- Informational responses are fully supported over HTTP/1, HTTP/2 and
HTTP/3.

**Changed**
- urllib3-future lower bound version is raised to 2.10.900.

.github/workflows/run-tests.yml

@@ -20,7 +20,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "pypy-3.9", "pypy-3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "pypy-3.9", "pypy-3.10"]
         os: [ubuntu-latest, macOS-13, windows-latest]
         include:
           # pypy-3.7, pypy-3.8 may fail due to missing cryptography wheels. Adapting.

.pre-commit-config.yaml

@@ -23,9 +23,9 @@ repos:
     # Run the formatter.
     - id: ruff-format
 - repo: https://github.com/pre-commit/mirrors-mypy
-  rev: v1.10.0
+  rev: v1.11.2
   hooks:
   -   id: mypy
       args: [--check-untyped-defs]
       exclude: 'tests/|noxfile.py'
-      additional_dependencies: ['charset_normalizer', 'urllib3.future>=2.9.900', 'wassima>=1.0.1', 'idna', 'kiss_headers']
+      additional_dependencies: ['charset_normalizer', 'urllib3.future>=2.10.900', 'wassima>=1.0.1', 'idna', 'kiss_headers']

HISTORY.md

@@ -1,6 +1,20 @@
 Release History
 ===============
 
+3.9.0 (2024-10-08)
+------------------
+
+**Added**
+- Support for WebSocket over HTTP/1, HTTP/2 and HTTP/3. It brings a unified API that makes you leverage
+  our powerful features like Happy Eyeballs, SOCKS/HTTP/HTTPS proxies, thread/task safety etc...
+- Hook for catching early responses like "103 Early Hints".
+
+**Fixed**
+- Informational responses are fully supported over HTTP/1, HTTP/2 and HTTP/3.
+
+**Changed**
+- urllib3-future lower bound version is raised to 2.10.900.
+
 3.8.0 (2024-09-24)
 ------------------
 

README.md

@@ -17,7 +17,7 @@ Niquests, is the “**Safest**, **Fastest[^10]**, **Easiest**, and **Most advanc
 | `HTTP/1.1`                          |       ✅        |     ✅     |       ✅       | ✅             |
 | `HTTP/2`                            |       ✅        |     ❌     |     ✅[^7]     | ❌             |
 | `HTTP/3 over QUIC`                  |       ✅        |     ❌     |       ❌       | ❌             |
-| `Synchronous`                       |       ✅        |     ✅     |       ✅       | ❌             |
+| `Synchronous`                       |       ✅        |     ✅     |       ✅       | _N/A_[^1]     |
 | `Asynchronous`                      |       ✅        |     ❌     |       ✅       | ✅             |
 | `Thread Safe`                       |       ✅        |     ✅     |     ❌[^5]     | _N/A_[^1]     |
 | `Task Safe`                         |       ✅        | _N/A_[^2] |       ✅       | ✅             |
@@ -42,6 +42,9 @@ Niquests, is the “**Safest**, **Fastest[^10]**, **Easiest**, and **Most advanc
 | `HTTP/2 with prior knowledge (h2c)` |       ✅        |     ❌     |       ✅       | ❌             |
 | `Post-Quantum Security`             | _Limited_[^12] |     ❌     |       ❌       | ❌             |
 | `HTTP Trailers`                     |       ✅        |     ❌     |       ❌       | ❌             |
+| `Early Responses`                   |       ✅        |     ❌     |       ❌       | ❌             |
+| `WebSocket over HTTP/1`             |       ✅        |  ❌[^14]   |    ❌[^14]     | ❌[^14]        |
+| `WebSocket over HTTP/2 and HTTP/3`  |     ✅[^13]     |     ❌     |       ❌       | ❌             |
 </details>
 
 <details>
@@ -158,9 +161,11 @@ Niquests is ready for the demands of building scalable, robust and reliable HTTP
 - Streaming Downloads
 - HTTP/2 by default
 - HTTP/3 over QUIC
+- Early Responses
 - Happy Eyeballs
 - Multiplexed!
 - Thread-safe!
+- WebSocket!
 - Trailers!
 - DNSSEC!
 - Async!
@@ -191,15 +196,17 @@ You may also be interested in unlocking specific advantages _(like access to a p
 
 Niquests is a highly improved HTTP client that is based (forked) on Requests. The previous project original author is Kenneth Reitz and actually left the maintenance of Requests years ago.
 
-[^1]: aiohttp has no support for synchronous request.
+[^1]: aiohttp was conceived solely for an asynchronous context.
 [^2]: requests has no support for asynchronous request.
 [^3]: while the HTTP/2 connection object can handle concurrent requests, you cannot leverage its true potential.
 [^4]: loading client certificate without file can't be done.
 [^5]: httpx officially claim to be thread safe but recent tests demonstrate otherwise as of february 2024. https://github.com/jawah/niquests/issues/83#issuecomment-1956065258 https://github.com/encode/httpx/issues/3072 https://github.com/encode/httpx/issues/3002
 [^6]: they do not expose anything to control network aspects such as IPv4/IPv6 toggles, and timings (e.g. DNS response time, established delay, TLS handshake delay, etc...) and such.
-[^7]: while advertised as possible, they refuse to make it the default due to performance issues. as of february 2024 an extra is required to enable it manually.
+[^7]: while advertised as possible, they refuse to make it the default due to performance issues. as of October 2024 an extra is required to enable it manually.
 [^8]: they don't support HTTP/3 at all.
 [^9]: you must use a custom DNS resolver so that it can preemptively connect using HTTP/3 over QUIC when remote is compatible.
-[^10]: performance measured when leveraging a multiplexed connection with or without uses of any form of concurrency as of July 2024. The research compared `httpx`, `requests`, `aiohttp` against `niquests`. See https://github.com/Ousret/niquests-stats
+[^10]: performance measured when leveraging a multiplexed connection with or without uses of any form of concurrency as of October 2024. The research compared `httpx`, `requests`, `aiohttp` against `niquests`. See https://github.com/Ousret/niquests-stats
 [^11]: enabled when using a custom DNS resolver.
 [^12]: available only when using HTTP/3 over QUIC and that the remote server support also the same post-quantum key-exchange algorithm. Also, the `qh3` installed version must be >= 1.1.
+[^13]: most servers out there are not ready for this feature, but Niquests is already compliant and future-proof! Modern server like Caddy are still working on it, see https://github.com/caddyserver/caddy/pull/6567 for more.
+[^14]: they don't offer any built-in to speak with a WebSocket server.

docs/community/faq.rst

@@ -140,7 +140,7 @@ Ever encountered something along::
 Yes? Usually it means that you tried to load a certificate (CA or client cert) that is malformed.
 
 What does malformed means?
-~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Could be just a missing newline character *RC*, or wrong format like passing a DER file instead of a PEM
 encoded certificate.

docs/index.rst

@@ -86,9 +86,11 @@ Niquests is ready for today's web.
 - Streaming Downloads
 - HTTP/2 by default
 - HTTP/3 over QUIC
+- Early Responses
 - Happy Eyeballs
 - Multiplexed!
 - Thread-safe!
+- WebSocket!
 - Trailers!
 - DNSSEC!
 - Async!

docs/user/advanced.rst

@@ -453,6 +453,8 @@ the request process, or signal event handling.
 
 Available hooks:
 
+``early_response``:
+    An early response caught before receiving the final Response for a given Request. Like but not limited to 103 Early Hints.
 ``response``:
     The response generated from a Request.
 ``pre_send``:
@@ -706,6 +708,10 @@ Alternatively you can configure it once for an entire
 
     See `#2018 <https://github.com/psf/requests/issues/2018>`_ for details.
 
+.. note:: WebSocket are too concerned by that section. By default ``wss://...`` will pick the ``https`` proxy
+    and the ``ws://...`` the ``http`` entry. You are free to add a ``wss`` key in your proxies
+    to route them on another proxy.
+
 When the proxies configuration is not overridden per request as shown above,
 Niquests relies on the proxy configuration defined by standard
 environment variables ``http_proxy``, ``https_proxy``, ``no_proxy``,
@@ -1315,7 +1321,7 @@ by passing a custom ``QuicSharedCache`` instance like so::
 When the cache is full, the oldest entry is removed.
 
 Disable HTTP/1.1, HTTP/2, and/or HTTP/3
------------------------------
+---------------------------------------
 
 You can at your own discretion disable a protocol by passing ``disable_http2=True`` or
 ``disable_http3=True`` within your ``Session`` constructor.
@@ -1507,3 +1513,28 @@ For example, we retrieve our trailers this way::
 
 
 .. warning:: The ``trailers`` property is only filled when the response has been consumed entirely. The server only send them after finishing sending the body. By default, ``trailers`` is an empty CaseInsensibleDict.
+
+Early Response
+--------------
+
+A server may send one or several (informational) response before the final response. Before this, those responses were
+silently ignored or worst, misinterpreted.
+
+Most notably, the status https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/103 is one of the most useful use case out there.
+
+To catch response like those::
+
+    from niquests import Session
+
+    def early_response_hook(early_response):
+        print(early_response)  # <Response HTTP/2 [103]>
+        print(early_response.headers)  # {'origin-trial': ..., 'link': '</hinted.png>; rel=preload; as=image'}
+
+    with Session() as s:
+        resp = s.get("https://early-hints.fastlylabs.com/", hooks={"early_response": early_response_hook})
+
+        print(resp)  # <Response HTTP/2 [200]>
+
+Isn't it easy and pleasant to write ?
+
+.. warning:: Some servers choose to enable it in HTTP/2, and HTTP/3 but not in HTTP/1.1 for security concerns. But rest assured that Niquests support this no matter the protocol.

docs/user/quickstart.rst

@@ -1012,6 +1012,114 @@ OCSP requests (certificate revocation checks) will follow given ``happy_eyeballs
 
 .. warning:: This feature is disabled by default and we are actually planning to make it enabled as the default in a future major.
 
+WebSockets
+----------
+
+.. note:: Available since version 3.9+ and requires to install an extra. ``pip install niquests[ws]``.
+
+It is undeniable that WebSockets are a vital part of the web ecosystem along with HTTP. We noticed that
+most users met frictions when trying to deal with a WebSocket server for the first time, that is why
+we decided to expand Niquests capabilities to automatically handle WebSockets for you.
+
+Synchronous
+~~~~~~~~~~~
+
+In the following example, we will explore how to interact with a basic, but well known echo server::
+
+    from niquests import Session
+
+    with Session() as s:
+        resp = s.get(
+            "wss://echo.websocket.org",
+        )
+
+        print(resp.status_code)  # it says "101", for "Switching Protocol"
+
+        print(resp.extension.next_payload())  # unpack the next message from server
+
+        resp.extension.send_payload("Hello World")  # automatically sends a text message to the server
+
+        print(resp.extension.next_payload() == "Hello World")  # output True!
+
+        resp.extension.close()  # don't forget this call to release the connection!
+
+.. warning:: Without the extra installed, you will get an exception that indicate that the scheme is unsupported.
+
+.. note:: Historically, Requests only accepted http:// and https:// as schemes. But now, you may use wss:// for WebSocket Secure or ws:// for WebSocket over PlainText.
+
+.. warning:: Be careful when accessing ``resp.extension``, if anything goes wrong in the "establishment" phase, meaning the server denies us the WebSocket upgrade, it will be worth ``None``.
+
+WebSocket and HTTP/2+
+~~~~~~~~~~~~~~~~~~~~~
+
+By default, Niquests negotiate WebSocket over HTTP/1.1 but it is well capable of doing so over HTTP/2 and HTTP/3 following RFC8441.
+But rare are the servers capable of bootstrapping WebSocket over a multiplexed connection. There's a little tweak to the URL
+so that it can infer your desire to use a modern protocol, like so ``wss+rfc8441://echo.websocket.org``.
+
+Asynchronous
+~~~~~~~~~~~~
+
+Of course, as per our feature coverage, this is doable both in synchronous and asynchronous contexts.
+Like so::
+
+    from niquests import AsyncSession
+    import asyncio
+
+    async def main() -> None:
+        async with AsyncSession() as s:
+            resp = await s.get("wss://echo.websocket.org")
+
+            # ...
+
+            print(await resp.extension.next_payload())  # unpack the next message from server
+
+            await resp.extension.send_payload("Hello World")  # automatically sends a text message to the server
+
+            print((await resp.extension.next_payload()) == "Hello World")  # output True!
+
+            await resp.extension.close()
+
+
+Ping and Pong
+~~~~~~~~~~~~~
+
+Ping sent by a server are automatically handled/answered by Niquests each time to read from the socket with `next_payload()`.
+However, we do not send automatically Ping TO the server.
+
+In order to do so::
+
+    from niquests import Session
+
+    with Session() as s:
+        resp = s.get(
+            "wss://echo.websocket.org",
+        )
+
+        print(resp.extension.ping())  # send a ping to the websocket server, notify it that you're still there!
+
+You can use the elementary methods provided by Niquests to construct your own logic.
+
+Binary and Text Messages
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+You may use ``next_payload()`` and ``send_payload(...)`` with str or bytes.
+
+If ``next_payload()`` output bytes, then it is a BinaryMessage.
+If ``next_payload()`` output str, then it is a TextMessage.
+
+The same apply to ``send_payload(...)``, if passed item is str, then we send a TextMessage.
+Otherwise, it will be a BinaryMessage.
+
+.. warning:: Niquests does not buffer "incomplete" message (e.g. end marker for a message). It returns every chunk received as is.
+
+.. note:: If ``next_payload()`` returns ``None``, that means that the remote choose to close the connection.
+
+Others
+~~~~~~
+
+Every other features still applies with WebSocket, like proxies, happy eyeballs, thread/task safety, etc...
+See relevant docs for more.
+
 -----------------------
 
 Ready for more? Check out the :ref:`advanced <advanced>` section.

noxfile.py

@@ -8,7 +8,7 @@
 
 def tests_impl(
     session: nox.Session,
-    extras: str = "socks",
+    extras: str = "socks,ws",
     cohabitation: bool | None = False,
 ) -> None:
     # Install deps and the package itself.

pyproject.toml

@@ -42,7 +42,7 @@ dynamic = ["version"]
 dependencies = [
     "charset_normalizer>=2,<4",
     "idna>=2.5,<4",
-    "urllib3.future>=2.9.900,<3",
+    "urllib3.future>=2.10.900,<3",
     "wassima>=1.0.1,<2",
     "kiss_headers>=2,<4",
 ]
@@ -57,6 +57,9 @@ http3 = [
 ocsp = [
     "urllib3.future[qh3]",
 ]
+ws = [
+    "urllib3.future[ws]",
+]
 speedups = [
     "orjson>=3,<4",
     "urllib3.future[zstd,brotli]",

requirements-dev.txt

@@ -1,7 +1,7 @@
 -e .[socks]
 pytest>=2.8.0,<=7.4.4
 pytest-cov
-pytest-httpbin==2.0.0
+pytest-httpbin>=2,<3
 pytest-asyncio>=0.21.1,<1.0
 httpbin==0.10.2
 trustme

src/niquests/__version__.py

@@ -9,9 +9,9 @@
 __url__: str = "https://niquests.readthedocs.io"
 
 __version__: str
-__version__ = "3.8.0"
+__version__ = "3.9.0"
 
-__build__: int = 0x030800
+__build__: int = 0x030900
 __author__: str = "Kenneth Reitz"
 __author_email__: str = "[email protected]"
 __license__: str = "Apache-2.0"