QUIC implementation in Py-libp2p

[seetadev]

Dear Py-libp2p team members,

Following the initial review and interoperability testing phase, I’d like to outline the next phase of the QUIC implementation and development work. This phase will be key to strengthening our Python-based QUIC transport for libp2p and aligning it better with broader ecosystem efforts.

After completing the MVP and interoperability testing, our focus should be on implementing QUIC more robustly by leveraging one of the following lower-level approaches:

• MSQUIC-based approach (either through a direct wrapper or via an intermediate like Zig or C bindings)
• NAPI/Rust library-based approach (Rust bindings exposed to Python, possibly via PyO3 or other wrappers)
• C library-based approach (direct CPython bindings if necessary)

Depending on your preference and what you find easier to work with, we can decide the exact pathway — all three options are valid and have their own trade-offs in terms of performance, maintenance, and compatibility.

Additionally, it’s worth keeping an eye on some related efforts across the libp2p ecosystem, which could significantly inform and refine our own design:

• zig-msquic + zig-libp2p:
  There’s work happening to wrap ms-quic using Zig’s FFI capabilities for use within Zig-libp2p: please visit https://github.com/MarcoPolo/zig-libp2p
  ➔ This could offer valuable insights into performance optimization and interoperability techniques, and potentially inspire native bindings strategies for Python.
  ➔ Aleksey (@flcl42) is also exploring MSQUIC integration in dotnet-libp2p, providing another parallel track we can learn from: please visit https://github.com/NethermindEth/dotnet-libp2p/tree/main and Quic on Windows should use OpenSSL version NethermindEth/dotnet-libp2p#45

• Cayman’s js-libp2p-quic (Rust-backed for JS stack):
  Cayman (@wemeetagain) has built a Rust-based QUIC transport for js-libp2p : please visit https://github.com/ChainSafe/js-libp2p-quic
  ➔ This can be very useful to validate handshake compatibility and muxing behavior across different runtimes, which will become crucial once we do transport interop with js-libp2p.
  ➔ Great work by Cayman that we can learn from as we aim for smooth cross-language interoperability.

• Waku team's ngtcp2 bindings in Nim:
  Our collaborators at Waku are wrapping ngtcp2 (another QUIC implementation) in Nim using OpenSSL.
  ➔ See here: Waku Nim-libp2p QUIC Transport
  ➔ Their approach will be a wonderful reference, especially if we consider evaluating other QUIC backend libraries for our use cases or future interop scenarios.
  ➔ Hats off to @richard-ramos and @vladopajic for their great work here.

Immediate next steps for you:

1. Study the specs on QUIC and try learning from the current MVP PR and get hands-on familiarity.
2. Attempt interoperability testing with other libp2p nodes (Go, Rust, JS) as described.
3. Based on the outcome, start planning the deeper integration using either MSQUIC, a Rust binding, or C library binding — whichever you are most comfortable with.
4. Keep tracking related ecosystem projects as they might offer shortcuts, better techniques, or libraries we can use instead of building everything from scratch.

[AkMo3]

I had been spending some thoughts regarding what approach we should take regarding implementation of QUIC.

Questions and Considerations:

1. What implementation other libp2p projects are using?

Libp2p Implementation	QUIC Framework
go-libp2p	quic-go
rust-libp2p	quinn
js-libp2p	quinn (via Rust/NAPI bindings)

2. Possible implementation plans:

   1. Python Native Implementation using aioquic - https://github.com/aiortc/aioquic (Python Implementation conforming with RFC 9000 (QUIC v1) and RFC 9369 (QUIC v2))
   2. Use FFI to generate python bindings for quinn - https://github.com/quinn-rs/quinn (Rust implementation, can be used inside python project using PyO3 which will generate rust binding similar to js-libp2p)

3. Interoperability result:

   1. The https://interop.seemann.io/ provides great interop status of various QUIC client and server implementation. Here is a summarised versioned filtered according to our needs.

   2. Client (aioquic vs quinn):
      

   3. Server (aioquic vs quinn):
      

   4. Summary:

      1. Server Performance:

         • quinn: The quinn server column shows a nearly perfect record. It successfully handles connections and all test cases (like Handshake, Data Transfer, Connection Migration, and Retry) from almost every client implementation in the matrix, including aioquic, quic-go, mvfst (Meta), quiche (Cloudflare), and msquic (Microsoft).
         • aioquic: The aioquic server column also shows a very strong performance, with successful tests against the vast majority of clients. It demonstrates full compatibility with all major libp2p-relevant clients like quic-go and quinn.

      2. Client Performance:

         • quinn: As a client, quinn successfully connects to and passes all test cases against nearly every server in the matrix. Its row is overwhelmingly green, showing it can reliably talk to servers from Google, Meta, Microsoft, and others.
         • aioquic: Similarly, the aioquic client has a very high success rate. It passes the test suites against all major server implementations, including quinn, proving its robustness as a client.

      3. Direct aioquic vs. quinn Interaction:

         • The test cell for aioquic (client) connecting to quinn (server) is green.
         • The test cell for quinn (client) connecting to aioquic (server) is green.

         In conclusion, the live data shows no meaningful difference in standards compliance or interoperability between aioquic and quinn.

4. Performance:

   • aioquic in itself doesn't provide any benchmarks, but I found a interesting graph in one of the reasearch study

 As expected, `aioquic` was least performant since it is the only interpreted and not compiled implementation. I similarly expect `quinn` to be more performant one here.

6. Maintainability:

   1. Using quinn with PyO3 will require some significant changes:
   • Development Environment:

   - Rust toolchain: rustc, cargo, rustfmt, clippy installation
   - maturin: Python-Rust build tool (pip install maturin)
   - Platform linkers: MSVC on Windows, appropriate cross-compilation toolchains
   - Python dev headers: For all target Python versions (python3-dev packages)
   

   • Build System:

   - Multi-stage builds: Rust compilation → Python extension building
   - Cross-ecosystem dependencies: Rust crates + Python packages coordination
   - Platform-specific compilation: Different flags for Linux/macOS/Windows
   - Build time impact: 3-5x longer than pure Python builds
   

   • Package Distribution:

   - Wheel matrix: Multiple platforms × Python versions (12+ wheel variants typical)
   - Binary size: ~10-50MB wheels vs ~1MB pure Python
   - Build infrastructure: High-CPU runners for compilation
   - Storage costs: Significantly larger artifact storage requirements
   

   2. Project Activity: The quinn implementation is better maintained and undergoing active development, whereas the aioquic project development is quite slow paced.

   3. Production Ready: The quinn implementation is fairly production ready, while aioquic is used by projects like mitmproxy, I am yet to see a fairly large production grade project yet to use it.

7. My thoughts:

It is a very tough choice, using quinn will bring significant performance and better interoperability with other libp2p projects using quinn, but at the significant cost of harder development lifecycle.

Are the performance benefits worth the cost, I would beg to differ at this point.

Rather I would like to go first with the python native approach resulting in a much faster implementation of QUIC, and it has all the features 0-RTT, HTTP/3 support etc. If and when we see a requirement of bringing high-performance QUIC transport, then we can come back here, curse me and start development of a new transport.

Would love to hear your thoughts on this @seetadev @pacrob

[seetadev]

@AkMo3 : Thank you so much for the incredibly thorough and thoughtful breakdown — this is excellent work, and I really appreciate the depth of analysis you’ve brought in here. You've laid out the trade-offs between the two implementation options for QUIC with great clarity, especially factoring in performance, interoperability, and maintainability.

The proposed path — starting with aioquic as the initial QUIC transport layer — makes a lot of sense in our current context.

On the same note, thanks for also sharing your feedback on the existing efforts on QUIC transport layer within Py-libp2p.

Here's why I strongly agree with your approach:

1. Rapid Prototyping and Implementation Velocity:
   Choosing aioquic allows us to move faster with a native Python implementation that keeps the development process lightweight and aligned with the existing ecosystem. This is especially important as we continue to evolve the py-libp2p stack — early integration, testing, and iteration matter more than premature optimization.

2. Interoperability Looks Strong:
   The test results from https://interop.seemann.io/ are very reassuring — the direct aioquic ↔ quinn compatibility greenlights interoperability with rust-libp2p and, by extension, other language implementations like js-libp2p and go-libp2p.

3. Clear Migration Path:
   Your suggestion to revisit a quinn + PyO3 approach only when performance demands it is a balanced and forward-looking idea. It leaves the door open for optimization while allowing us to validate the protocol integration first.

4. Lower Overhead for New Contributors:
   Keeping the stack Python-native at this stage lowers the entry barrier, especially for contributors who may not be familiar with the complexities of a Rust-based toolchain or FFI builds. That supports broader community involvement and makes mentoring and onboarding easier.

5. Community Alignment:
   While quinn is being used more broadly across the libp2p ecosystem, aioquic is still a highly standards-compliant implementation with growing adoption. And as you noted, its features like 0-RTT and HTTP/3 are all there.

Overall, I’m fully aligned with your proposal to go ahead with aioquic as our initial QUIC transport layer for py-libp2p. It strikes a great balance between pragmatism and future extensibility.

Please go ahead and start shaping the transport layer around aioquic.

Let’s keep tracking performance metrics, and if any bottlenecks arise during usage or scaling, we can revisit and prioritize optimization.

Thanks again for the clarity and leadership here — delighted to see this move forward.

[pacrob]

Thanks for your thorough review, @AkMo3. I agree with your and @seetadev's view.

[seetadev]

@pacrob : Thank you Paul for your kind feedback and pointers. Appreciate it.

[seetadev]

@AkMo3 , @lla-dane , @guha-rahul : We can generate pytest + trio-based test templates for QUIC support in py-libp2p using pytest-trio. Wish to share some of the test cases we should cover in the QUIC PR. This will also be helpful in transport interop efforts with other libp2p modules:

1. Basic QUIC Handshake and Connection

• Test successful QUIC connection establishment between two nodes

• Test failure cases:

  • Invalid certificate or key
  • Mismatched TLS versions
  • Wrong peer ID

2. Identity and Peer ID Validation

• Validate that the remote peer’s identity matches the expected PeerID
• Test that QUIC multiaddrs (/ip4/.../udp/.../quic-v1) correctly parse and resolve
• Validate rejection of unknown or tampered certificates

3. Stream Multiplexing over QUIC

• Open multiple simultaneous streams over a single QUIC connection

• Test stream-level operations:

  • Send and receive data
  • Close stream gracefully
  • Abort stream and catch error

• Validate backpressure or flow control behavior (e.g. limit stream window)

4. Interoperability

• Optional: Simulate or test against a mock go-libp2p QUIC node (if available)
• Ensure that standard QUIC features match behavior in go-libp2p

5. Error Handling and Recovery

• Test for proper fallback or error raising when:

  • Connection is refused
  • QUIC transport is disabled
  • Unreachable peer

6. Integration with Transport Manager

• Verify QUIC transport integrates with Swarm and Host layers
• Ensure that QUIC transport coexists with TCP and WebSocket transports if enabled

7. Reconnect and Resilience

• Drop a connection and test automatic reconnection over QUIC
• Reuse multiaddrs from peerstore with /quic-v1 suffix

8. Unit Tests for Helpers and Internals

• Certificate/key generation utilities
• TLS config creation and handshake logic
• Stream buffer handling and framing logic (e.g. chunked I/O)

9. Code Quality & Linting

• Run black, ruff, or any configured linters/formatters
• Ensure type hints are validated via mypy
• Check for asyncio compliance — no blocking calls

10. Coverage

• Aim for ≥ 90% test coverage on the new quic/ module

• Cover edge cases like:

  • PeerID mismatch
  • Early connection close
  • QUIC packet loss simulation (if feasible)

[acul71]

@seetadev @pacrob @AkMo3

QUIC Interoperability Analysis Report

Repository: acul71/test-plans-fork - quic branch

Executive Summary

This report analyzes QUIC interoperability test results for Python libp2p v0.2.9 against various other libp2p implementations. The tests reveal a critical interoperability issue where Python libp2p fails to interoperate with Go and Rust implementations when acting as a listener, but succeeds when acting as a dialer.

QUIC Libraries by Implementation

Implementation	QUIC Library	Version	Notes
Python libp2p v0.2.9	aioquic	Latest	Python QUIC implementation with TLS 1.3
Go libp2p v0.40-0.42	quic-go	v0.49.0	Go QUIC implementation with TLS 1.3
Rust libp2p v0.53-0.54	quinn	v0.10.2	Rust QUIC implementation with rustls
Java libp2p v0.9	Netty QUIC	v0.0.71.Final	Java QUIC implementation with BoringSSL
JVM libp2p v1.2	Netty QUIC	v0.0.71.Final	JVM QUIC implementation with BoringSSL

Test Results Overview

Test Configuration	Outcome	Role	Notes
jvm-v1.2 x python-v0.2.9	✅ SUCCESS	Python as listener	JVM dialer → Python listener
java-v0.9 x python-v0.2.9	✅ SUCCESS	Python as listener	Java dialer → Python listener
python-v0.2.9 x rust-v0.54	❌ FAILURE	Python as dialer	Python dialer → Rust listener
python-v0.2.9 x python-v0.2.9	✅ SUCCESS	Both roles	Python ↔ Python
python-v0.2.9 x jvm-v1.2	✅ SUCCESS	Python as dialer	Python dialer → JVM listener
python-v0.2.9 x java-v0.9	✅ SUCCESS	Python as dialer	Python dialer → Java listener
python-v0.2.9 x rust-v0.53	❌ FAILURE	Python as dialer	Python dialer → Rust listener
python-v0.2.9 x go-v0.42	✅ SUCCESS	Python as dialer	Python dialer → Go listener
python-v0.2.9 x go-v0.41	✅ SUCCESS	Python as dialer	Python dialer → Go listener
python-v0.2.9 x go-v0.40	✅ SUCCESS	Python as dialer	Python dialer → Go listener
go-v0.40 x python-v0.2.9	❌ FAILURE	Python as listener	Go dialer → Python listener
go-v0.41 x python-v0.2.9	❌ FAILURE	Python as listener	Go dialer → Python listener
go-v0.42 x python-v0.2.9	❌ FAILURE	Python as listener	Go dialer → Python listener
rust-v0.53 x python-v0.2.9	❌ FAILURE	Python as listener	Rust dialer → Python listener
rust-v0.54 x python-v0.2.9	❌ FAILURE	Python as listener	Rust dialer → Python listener

Key Findings

✅ Successful Interoperability

• Python ↔ Java/JVM: Full bidirectional interoperability
• Python → Go: Python dialer works with Go listeners
• Python ↔ Python: Self-interoperability works perfectly

❌ Failed Interoperability

• Go → Python: All Go versions fail when dialing Python listeners
• Rust → Python: All Rust versions fail when dialing Python listeners
• Python → Rust: Python dialer fails with Rust listeners

Root Cause Analysis

Primary Issue: "Unhandled event type: ProtocolNegotiated"

The most consistent error across all failed tests is:

Unhandled event type: ProtocolNegotiated

This error occurs in the Python listener when receiving QUIC protocol negotiation events from Go/Rust dialers.

Secondary Issues

1. "No CRYPTO frame found in datagram!" - QUIC handshake issues
2. "Connection already started" - Connection state management problems
3. "Connection close received (code 0x12E, reason invalid peer certificate)" - Certificate validation failures

Error Pattern Analysis

Go → Python Failures

dialer-1  | 2025-09-30 06:18:55,067 [INFO] [quic] [f0f7ed4124907509] Connection close received (code 0x12E, reason invalid peer certificate: Other(OtherError(UnknownIssuer)))
listener-1 | Unhandled event type: ProtocolNegotiated
listener-1 | No CRYPTO frame found in datagram!
listener-1 | Connection already started

Rust → Python Failures

listener-1 | Unhandled event type: ProtocolNegotiated
listener-1 | 2025-09-30 06:18:54,570 [INFO] [quic] [b7287c7ab0959936969fcc76a77599beadcff7d4] Connection close received (code 0xC, reason )

Python QUIC Configuration Analysis

Current Python QUIC Configuration

# Enhanced QUIC transport configuration
quic_config = QUICTransportConfig(
    # Disable certificate verification for interoperability
    verify_mode=ssl.CERT_NONE,
    # Increase timeouts for interoperability testing
    idle_timeout=60.0,
    connection_timeout=30.0,
    # Increase stream timeouts
    STREAM_OPEN_TIMEOUT=15.0,
    STREAM_ACCEPT_TIMEOUT=30.0,
    STREAM_READ_TIMEOUT=30.0,
    STREAM_WRITE_TIMEOUT=30.0,
    # Increase connection handshake timeout
    CONNECTION_HANDSHAKE_TIMEOUT=60.0,
    # Enable both QUIC versions for maximum compatibility
    enable_draft29=True,
    enable_v1=True,
    # Enhanced QUIC settings for interoperability
    max_idle_timeout=60.0,
    keep_alive_interval=30.0,
    disable_active_migration=True,
    enable_0rtt=True,
    # Protocol negotiation settings
    enable_protocol_negotiation=True,
    alpn_protocols=["libp2p"],
    # Connection management
    max_concurrent_streams=200,
    initial_max_data=104857600,
    initial_max_stream_data_bidirectional_local=1048576,
    initial_max_stream_data_bidirectional_remote=1048576,
    initial_max_streams_bidirectional=100,
    initial_max_streams_unidirectional=100
)

Host Configuration for QUIC

# For QUIC transport, use it as a secure transport (like Java implementation)
# QUIC has built-in security and multiplexing, so no separate security/muxer needed
if enable_quic:
    self.host = new_host(
        key_pair=key_pair,
        muxer_opt=muxer_options,
        listen_addrs=[listen_addr],
        enable_quic=enable_quic,
        quic_transport_opt=quic_config
    )
else:
    self.host = new_host(
        key_pair=key_pair,
        sec_opt=security_options,
        muxer_opt=muxer_options,
        listen_addrs=[listen_addr],
        enable_quic=enable_quic,
        quic_transport_opt=quic_config
    )

Hypotheses for Test Failures

1. Protocol Negotiation Event Handling

Hypothesis: Python libp2p's QUIC implementation doesn't properly handle ProtocolNegotiated events that Go/Rust implementations send during the QUIC handshake process.

Evidence: Consistent "Unhandled event type: ProtocolNegotiated" errors across all Go/Rust → Python tests.

Root Cause: The Python libp2p QUIC transport layer lacks proper event handlers for protocol negotiation events that occur after the QUIC handshake is complete.

2. Certificate Validation Mismatch

Hypothesis: Different certificate validation approaches between implementations cause connection failures.

Evidence:

• Go → Python: "invalid peer certificate: Other(OtherError(UnknownIssuer))"
• Rust → Python: "Connection close received (code 0x12E, reason invalid peer certificate)"

Root Cause: Python libp2p's certificate validation logic is incompatible with Go/Rust certificate handling.

3. QUIC Stream Multiplexing Issues

Hypothesis: Python libp2p's QUIC stream handling is incompatible with Go/Rust stream multiplexing approaches.

Evidence: "No CRYPTO frame found in datagram!" errors suggest QUIC stream processing issues.

Root Cause: Different approaches to QUIC stream creation and management between implementations.

4. Connection State Management

Hypothesis: Python libp2p's connection state tracking conflicts with Go/Rust connection lifecycle management.

Evidence: "Connection already started" errors indicate duplicate connection attempts.

Root Cause: Different connection state machines between implementations.

Comparison with Working Implementations

Java/JVM Success Pattern

// JVM treats QUIC as a secure transport
if (params.transport == QUIC_V1) {
    it.secureTransports.add(QuicTransport::ECDSA)
} else {
    it.transports.add(::TcpTransport)
}

Key Insight: Java/JVM implementations treat QUIC as a complete secure transport with built-in security and multiplexing, not as a regular transport requiring additional layers.

Python Implementation Pattern

# Python libp2p QUIC implementation
if enable_quic:
    self.host = new_host(
        key_pair=key_pair,
        muxer_opt=muxer_options,  # QUIC has built-in multiplexing
        listen_addrs=[listen_addr],
        enable_quic=enable_quic,
        quic_transport_opt=quic_config
    )

Key Insight: Python libp2p treats QUIC as a complete secure transport with built-in security and multiplexing (QUICConnection implements both IRawConnection and IMuxedConn), similar to Java/JVM implementations. The interoperability issue is in protocol negotiation event handling, not architecture.

QUIC Interoperability Roadmap

Current Status: Partial QUIC Interoperability

• ✅ Python ↔ Java/JVM: Full bidirectional QUIC interoperability
• ✅ Python → Go: Python dialer works with Go QUIC listeners
• ❌ Go → Python: Go dialer fails with Python QUIC listeners
• ❌ Rust → Python: Rust dialer fails with Python QUIC listeners
• ❌ Python → Rust: Python dialer fails with Rust QUIC listeners

Required Fixes for Full QUIC Interoperability

1. Protocol Negotiation Event Handling

• Issue: Python libp2p QUIC doesn't handle ProtocolNegotiated events from Go/Rust
• Fix: Implement proper event handlers for QUIC protocol negotiation events
• Priority: Critical

2. Certificate Validation Compatibility

• Issue: Certificate validation mismatch between Python and Go/Rust
• Fix: Align certificate validation logic with Go/Rust approaches
• Priority: High

3. Connection State Management

• Issue: "Connection already started" errors indicate state conflicts
• Fix: Improve connection lifecycle management compatibility
• Priority: High

4. QUIC Stream Processing

• Issue: "No CRYPTO frame found in datagram!" errors
• Fix: Enhance QUIC stream handling for Go/Rust compatibility
• Priority: Medium

Conclusion

The test results reveal partial QUIC interoperability for Python libp2p. While Python successfully interoperates with Java/JVM implementations and works as a dialer with Go implementations, it fails when acting as a listener for Go/Rust dialers.

Path to Full QUIC Interoperability: The identified issues (protocol negotiation events, certificate validation, connection state management, and QUIC stream processing) need to be addressed in Python libp2p's QUIC implementation to achieve complete cross-implementation QUIC interoperability.

[AkMo3]

Appreciate for the report, I will be taking notes from this, in the mean time can you also share the source code for these test runs. Also, we have included a echo interop test using QUIC between py-libp2p and nim-libp2p already, which you can use to get better understanding for what might be actually causing the interop issue.