quic_recvmsg(): Transport endpoint is not connected on client side

[Powerjonny]

By using the libquic user space implementation, echoing more than 65535 Byte between client and server results in an error. After the server has called quic_recvmsg twice to get the whole data request, sending data back to client results in quic_recvmsg(): Transport endpoint is not connected. Monitoring the data exchange with Wireshark shows that the client is sending additional data (maybe event or notification data, see 2025.01.28__Issue_QUIC_wireshark.zip) . Furthermore, printing the flags parameter from quic_recvmsg indicates something (flags is equal to 128 instead 640). Is there something that has to be taken into account (e.g. setting transport parameters)?

[moritzbuhl]

To better understand and reproduce the problem, could you share a minimal code snippet that demonstrates how you're sending and receiving data using libquic?
It would be helpful if you could provide a SSLKEYLOGFILE to decrypt the QUIC traffic in the pcap.
What is the MTU of the network interface you're using for this?
The UDP datagrams of length 65515 bytes in the pcap is the maximum UDP size, which could be contributing to the problem.

128 or 0x80 should mean MSG_EOR.
640 or 0x280 should be MSG_EOR and MSG_FIN or MSG_STREAM_FIN, so without any code it doesn't tell that anything is wrong.

[Powerjonny]

Hi! Thank you for your reply!

The data is sent over the loopback interface, which has a MTU of 65536 byte. A new recording with ssl key log file can be found here:

2025.01.28__Issue_QUIC_wireshark.zip

. The last interactions within the pcap file represent the error-related case with n = 65794 B.

The usage of libquic can be demostrated as follows:

Client:

/*
  * Sending request of size n
  *
  * @sockfd: 
  * @msg->enc = uint8_t*
  * @msg->size = n, with n > 65536 
  */
sid = QUIC_STREAM_TYPE_UNI_MASK;
if((ret = quic_sendmsg(sockfd, msg->enc, msg->size, QUIC_STREAM_TYPE_UNI_MASK, MSG_STREAM_NEW | MSG_STREAM_FIN)) < 0)
{
     printf("quic_sendmsg(): %s\n", strerror(errno));
     goto err; /*error handling*/
}

...

/*
  * Receiving response from server
  *
  * working for e.g. n = 33028, but not for n = 65794
  */
flags = 0;
sid = 0;
if((ret = quic_recvmsg(sockfd, buf, sizeof(buf), &sid, &flags)) < 0)
{
       printf("quic_recvmsg(): %s (%d)\n", strerror(errno), errno);
       goto err; /*error handling*/
}

Server:

/*
 * Waiting for client request data
 *
 * @buf is a static buffer (uint8_t) of size 70000
 * @connfd: is the target QUIC socket
 *
 * NOTE: If client sends e.g. 65794 B, @flags is set to 128 (decimal) by quic_recvmsg.
 */
flags = 0;
if((ret = quic_recvmsg(connfd, buf, sizeof(buf), &sid, &flags)) < 0)
{
       printf("quic_recvmsg(): %s\n", strerror(errno));
       goto end; /*error handling*/
}

...

/*
 * Checking if data is missing -> read further data from socket (in my case: only called for n = 65794)
 *
 * @is_missing: true, if not all expected bytes were received from the first quic_recvmsg() call
 * @buf is a static buffer (uint8_t) of size 70000
 */
if(is_missing)
{
    flags = 0;
    sid = 0;
    old = quic_recvmsg(connfd, buf+ret, sizeof(buf)-ret, &sid, &flags);
}

...

/*
  * Send response to client
  *
  * @msg->enc and msg->size identical to the client's values
  */
sid = QUIC_STREAM_TYPE_SERVER_MASK;
flags = MSG_STREAM_NEW | MSG_STREAM_FIN;
if((ret = quic_sendmsg(connfd, msg->enc, msg->size, sid, flags)) < 0)
{
     printf("quic_sendmsg(): %s\n", strerror(errno));
}

[moritzbuhl]

The attachment doesn't work for me.

[Powerjonny]

The attachment doesn't work for me.

Now, it should work to download the zip archive.

[moritzbuhl]

I can reproduce your problem with the following source code file:
issue.c
I put it in quic/tests and do cc -o issue ../libquic/*.c issue.c -lgnutls.
I enabled events and see the following in the logs:

server hs done
server recving ...
ret=65472 flags=128 sid=2
server recving again...
ret=4528 flags=640 sid=2
server sending...
ret=70000

client hs done
client sending ...
ret=70000
client recving ...
ret=25 flags=32896 sid=0
QUIC_EVENT_STREAM_UPDATE
client recving ...
ret=9 flags=32896 sid=0
QUIC_EVENT_STREAM_MAX_STREAM
client recving ...
ret=10 flags=32896 sid=0
QUIC_EVENT_CONNECTION_CLOSE errcode=5 frame=17 phrase=
client recving ...
quic_recvmsg(): Transport endpoint is not connected (107)
do_client error

errcode 5 indicates the internal QUIC_TRANSPORT_ERROR_STREAM_STATE. Which I narrow down to quic_frame_max_stream_data_process with some printk calls.

This is also visible in your pcap:

QUIC IETF
    QUIC Connection information
    [Packet Length: 35]
    QUIC Short Header DCID=4937d784ff53cf32 PKN=4
    MAX_STREAM_DATA id=2
        Frame Type: MAX_STREAM_DATA (0x0000000000000011)
        Stream ID: 2
        Maximum Stream Data: 1118576

and then

QUIC IETF
    QUIC Connection information
    [Packet Length: 33]
    QUIC Short Header DCID=6f02b2863b4a3538 PKN=5
    CONNECTION_CLOSE (Transport) Error code: STREAM_STATE_ERROR
        Frame Type: CONNECTION_CLOSE (Transport) (0x000000000000001c)
        Error code: STREAM_STATE_ERROR (5)
        Frame Type: 17
        Reason phrase Length: 0
        Reason phrase: 

To my understanding SID 2 no longer exists for the QUIC implementation as it set FIN and all data arrived at user space.
So I suspect that generating the MSD frame is wrong.

[moritzbuhl]

Please update your module, I think @lxin fixed this problem. Please report back.

[Powerjonny]

Yes, now it works without problems! Thanks a lot for your help!!

[Powerjonny]

Hi again!
This time, I have tested the same code/scenario over a wireless network with a MTU of 1500. Unfortunately, the described error above occurs again (loopback works still fine). However, the error quic_recvmsg(): Transport endpoint is not connected is already triggered on client side after the server is echoing 2054 B (1284 B is still working, but double this value -> larger than MTU does not work). Here is the corresponding Wireshark recording and the ssl key log file:

2025.02.07__libquic_n2054.zip

[lxin]

@Powerjonny Thanks for opening this issue.

I don't see the ssl key log file from the attachment, but only the pcapng file:

• 2025.02.07__libquic-n2054.pcapng

Can you run your test again the latest repo? and if the issue still exists, please attach both pcapng and ssl key log file.

BTW, you can get the ssl key log file by running: export SSLKEYLOGFILE=/tmp/sslkeys.log before starting your quic tests. The keys will be saved in /tmp/sslkeys.log during your testing. The file /tmp/sslkeys.log is the ssl key log we needed.

Thanks.