P999 may not be accurate if Never Send is high.

[neolinsu]

Hi all,

I find the synthetic in Caladan endures a high Never-Send rate (above 1%) when clients issue requests with a relatively high rate which is close to server’s capacity.
This is especially problematic under a Poisson distribution: when two adjacent requests are generated within a short time window (i.e., a bursty period), the latter one is more likely to be droped due to the Never-Send logic (see code). We have profiled Caladan's client logic, and find that the scheduling often causes the request to be delayed (which has already violated the Poisson distribution) and finally be dropped.

We further designed an experiment to confirm this. We modify the Caladan client with the scheduling policy disabled: specifically, workers are bound to different cores, which can execute send, do_softirq (directpath), handle_timeout, and recv in cycles without yielding.
We equip Caladan server with 4 kthreads and launch 16 client workers (each of which owns a TCP connection) to generate requests w/ a Poisson distribution and vary the request rate (last for 32 seconds). The following table shows the experiment result:

Client Type	Thoughtput(pps)	P50 (us)	P90 (us)	P999 (us)	Never Send
synthetic	0.75M	13.4	23.1	40.0	1.39%
client w/o sched	0.75M	9.809091	18.500909	50.903636	0.000442%
synthetic	0.8M	13.6	22.6	37.9	1.4486%
client w/o sched	0.8M	9.49	16.81	584.52	0.000430%
synthetic	1M	13.6	21.9	38.6	1.6726%
client w/o sched	1M	9.64	17.59	2841.83	0.000694%
synthetic	1.1M	13.5	21.1	55.5	1.7345%
client w/o sched	1.1M	10.6	21.9	5177.75	0.000781%

[joshuafried]

Thanks! Indeed, with insufficient resources the client itself can become the bottleneck. We typically run the load generator with spinning kthreads - see here and many cores. When one client is insufficient to generate load, we typically use multiple machines. What are the details of your machine, and what configurations are you using for your client?

[neolinsu]

The cpu is Intel Xeon 2.20GHz with 20 Hyper-Threadings (10 Phy cores), which are set to performance mode. The network is 100Gb RDMA.
The configuration I use for both two clients is the same as (replace some useless info):

host_addr 10.100.100.103
host_netmask 255.255.255.0
host_gateway 10.100.100.1
runtime_kthreads 16
runtime_guaranteed_kthreads 16
runtime_spinning_kthreads 16
host_mac X
disable_watchdog true
runtime_qdelay_us 10
runtime_priority lc
static_arp 10.100.100.102 X
static_arp 10.100.100.103 X
enable_directpath fs
directpath_pci X

I also notice that even client stresses at a low throughput (like 0.75 M) where resources should be sufficient, Never Send rate is still above 1%.

[joshuafried]

Can you post the output of a client here (and the parameters used to launch it)? Looking at some recent runs I see that even at 1MPPS my never sent rate is < .1%

[neolinsu]

Here is example to run 0.8M throughput.

synthetic --config synthetic.config 10.100.100.102:5190  --output=buckets --protocol memcached --mode runtime-client --threads 16 --runtime 32 --barrier-peers 1 --barrier-leader node151 --distribution=exponential --mpps=0.8 --samples=1 --transport tcp --nvalues=3200000

And synthetic's result is:

Distribution, Target, Actual, Dropped, Never Sent, Median, 90th, 99th, 99.9th, 99.99th, Start
exponential, 788411, 788411, 0, 326090, 13.6, 22.6, 33.2, 37.3, 37.9, 0, 8510673237596225

[joshuafried]

Hm, that is quite high. Can you post a log with many samples at lower loads (change the above command to --samples 20). Also can you try reducing the number of kthreads to 8 and see if that has any impact?

[neolinsu]

Can you post the output of a client here (and the parameters used to launch it)? Looking at some recent runs I see that even at 1MPPS my never sent rate is < .1%

I run server with

runtime_kthreads 4
runtime_guaranteed_kthreads 0
runtime_spinning_kthreads 0

It makes the cores mwait.

Would you plz share your configuration of server?

[joshuafried]

The server had 20 kthreads (20 guaranteed, 0 spinning). Does varying the server configuration impact the client behavior here?

[neolinsu]

The server had 20 kthreads (20 guaranteed, 0 spinning). Does varying the server configuration impact the client behavior here?

Yes. I think 20 kthreads can handle 1M pps.

You can try my configuration.

[neolinsu]

The point here is not how many guaranteed kthreads the Caladan server used. Instead, given the number of guaranteed kthreads (say 4 cores), we send client requests at a rate that is close (but lower) to the maximum capacity that the Caladan server can handle (say 1 Mpps), in this setup, no matter how many physical cores are used at client machines (even one core per connection), Never-Send rate is always high. As a result, the generated requests exhibit a distribution which is less bursty as expected.

W/ our modified clients (disable scheduling and let the softirq process one packet at a time), Never-Send rate is low. At this time, the generated requests follow a distribution that is more consistent to a Poisson distribution, but Caladan’s P999 latency becomes much higher.

[joshuafried]

Does this behavior change if you use many connections to the server? Say 100?

[joshuafried]

I'm trying to understand where the source of the delay is coming from that is causing so many never-sent packets. Please correct me if I am wrong in understanding the scenario here: the server machine is being tested at a load point close to its peak throughput. The client process/machine is not at full utilization and is not a bottleneck. Does this seem correct?

[neolinsu]

I'm trying to understand where the source of the delay is coming from that is causing so many never-sent packets. Please correct me if I am wrong in understanding the scenario here: the server machine is being tested at a load point close to its peak throughput. The client process/machine is not at full utilization and is not a bottleneck. Does this seem correct?

Yes, this is correct

[neolinsu]

Does this behavior change if you use many connections to the server? Say 100?

It seems the Never Send rate becomes higher as # of connections grows.