gRPC listchannels & listnodes performance

[michael1011]

Getting all nodes or channels CLN has gossip for via the gRPC interface is really slow. Calling listchannels via gRPC takes a minute while the CPU usage of the gRPC plugin spikes to 100% on one core

It is definitely not a problem with my CLN installation or my machine because the RPC socket socket is quick:

time lightning-cli listchannels | jq .channels > /dev/null

real    0m4.508s
user    0m2.125s
sys     0m0.273s

getinfo output

{
   "id": "02ead25af6e0271c5167d6fd05545d2d538995ce3e16ad780a1010fc4e91522202",
   "alias": "LOUDGOPHER",
   "color": "02ead2",
   "num_peers": 0,
   "num_pending_channels": 0,
   "num_active_channels": 0,
   "num_inactive_channels": 0,
   "address": [],
   "binding": [
      {
         "type": "ipv4",
         "address": "0.0.0.0",
         "port": 9745
      }
   ],
   "version": "v24.11-28-gdac505d",
   "blockheight": 878642,
   "network": "bitcoin",
   "fees_collected_msat": 0,
   "lightning-dir": "/home/michael/.lightning/bitcoin",
   "our_features": {
      "init": "08a0880a8a59a1",
      "node": "88a0880a8a59a1",
      "channel": "",
      "invoice": "02000002024100"
   }
}

[daywalker90]

While it is true that our custom rust decoder could be more optimized i can't reproduce these horrible stats:

time lightning-cli listchannels | jq .channels > /dev/null

real    0m2.721s
user    0m2.409s
sys     0m0.140s

python tests/test_grpc_speed.py

Grpc listchannels took 3.61s
channelcount: 78944
Grpc listnodes took 0.46s
nodecount: 15578

Did you compile the rust binaries in release mode? Check the size of cln-grpc if its around 20MB it's in release mode if it's ~200MB it's in debug mode. What is the hardware?

[daywalker90]

For reference:

import time
from pathlib import Path

import grpc
from pyln import grpc as clnpb

max_message_size = 100 * 1024 * 1024  # 100 MB

p = Path("/home/bitcoin/.lightning/bitcoin")
cert_path = p / "client.pem"
key_path = p / "client-key.pem"
ca_cert_path = p / "ca.pem"
creds = grpc.ssl_channel_credentials(
    root_certificates=ca_cert_path.open("rb").read(),
    private_key=key_path.open("rb").read(),
    certificate_chain=cert_path.open("rb").read(),
)

channel = grpc.secure_channel(
    "127.0.0.1:9736",
    creds,
    options=(
        ("grpc.ssl_target_name_override", "cln"),
        ("grpc.max_receive_message_length", max_message_size),
    ),
)
stub = clnpb.NodeStub(channel)
start_time = time.time()
response = stub.ListChannels(clnpb.ListchannelsRequest())
print(f"Grpc listchannels took {round(time.time() - start_time, 2)}s")
print(f"channelcount: {len(response.channels)}")

start_time = time.time()
response = stub.ListNodes(clnpb.ListnodesRequest())
print(f"Grpc listnodes took {round(time.time() - start_time, 2)}s")
print(f"nodecount: {len(response.nodes)}")

This is what i created inside the cln repo's tests folder.

[vincenzopalazzo]

lets assume that the encoding of grpc is fast enough because they have some users! What is the performance of the cln-rpc when calling listchannels compared with the lightning-cli?

I am worried about this part of the code https://github.com/ElementsProject/lightning/blob/master/cln-rpc/src/lib.rs#L314 that it is doing unnecessary conversion from serde_json::Value.

However, performance of encoding and decoding in rust depends also from CPU so probably this can be another reason that @daywalker90 is getting another type of performance result

[daywalker90]

i rewrote that script in rust so i can use call_typed:

raw listchannels (lightning-cli listchannels, no json decoding) took 1.15s
raw listnodes (lightning-cli listnodes, no json decoding) took 0.27s

Grpc listchannels took 3.87s
channelcount: 80275
Grpc listnodes took 0.61s
nodecount: 16945

rpc (.call_typed) listchannels took 3.57s
channelcount: 80275
rpc (.call_typed) listnodes took 3.98s
nodecount: 16945

For comparison in debug mode (pointless to benchmark in debug mode but i suspect this is what happened to OP, well maybe not looking at the grpc results lol):

raw listchannels (lightning-cli listchannels, no json decoding) took 1.02s
raw listnodes (lightning-cli listnodes, no json decoding) took 0.23s

Grpc listchannels took 4.44s
channelcount: 80275
Grpc listnodes took 0.67s
nodecount: 16945

rpc (.call_typed) listchannels took 65.69s
channelcount: 80275
rpc (.call_typed) listnodes took 70.52s
nodecount: 16945

I'm using a 10 year old intel cpu: i7-6700k