Query execution time increases when the number of concurrent users increases

[vishalratnam-cliq-scm]

We're trying to load test (using k6 testing tool) nebula graph DB using the nebula-go client.
We have created 1 global connection pool when the application starts with 1 host address pointing to our cluster (nebula cluster contains 3 graphd services, 3 metad services and 3 storaged services).
We create a session for each incoming request on our API, that executes a query and returns the data. After the API is done, we close the session.
The response time when we were trying to do the same with 1 virtual user on k6, the query execution time is 20ms. When we hit the same API with 10 virtual users with 10 concurrent requests from each user, the query execution time increases to ~100ms.

Is there any reason for this increase in execution time? Do we need to create a new pool for each request? Can't we use the same pool of connections when the load increases?

[xiajingchun]

Session is also expensive to create. It's highly recommened that you keep one session for each virtual user. Give it a try.
Here's a k6 plugin we use to test Nebula, which might interest you as well: https://github.com/vesoft-inc/k6-plugin

[akhilravuri-tul-scm]

how do you change connPool.MaxConnPoolSize = 1000000 when NewConnectionPool, it would initialize the connection pool. so just config the MaxConnPoolSize in poolConfig

poolConfig := nebula_sdk.GetDefaultConf()
poolConfig.MaxConnPoolSize = 1000000
connPool, err := nebula_sdk.NewConnectionPool(hostList, poolConfig, nebula_sdk.DefaultLogger{})

Yes..we use poolConfig.MaxConnPoolSize = 1000000 only.

[vishalratnam-cliq-scm]

Hi @HarrisChu,
The output of netstat |grep 9669|wc -l changes as we increase the number of virtual users from K6.
We set the poolConfig.MaxConnPoolSize to 100000. After increasing the number of virtual users from 10 to 250, the output of netstat |grep 9669|wc -l increased, but was never equal to the number of virtual users. Please check the below stats:

Single Pod:

1 VUs - 1 Connection
10 VUs - 8 Connections
50 VUs - 40 Connections
100 VUs - 81 Connections and 94, 97
150 VUs - 97 Connections
200 VUs - 166 Connections
250 VUs - 189 Connections, Errors at this point: Error: dial tcp 15.206.141.229:80: socket: too many open files

2 Pods:

1 VUs -  1 on 1 pod
10 VUs - 4 on each pod
50 VUs - 20 on each pod
100 VUs - 40 on pod 1, 41 on pod2
150 VUs - 50 on pod 1, 52 on pod2
200 VUs - 50 on pod 1, 52 on pod2
250 VUs - 65 on each pod, Errors at this point: Error: dial tcp 15.206.141.229:80: socket: too many open files

[xiajingchun]

could you check the result of
ulimit -n
more details at: https://docs.nebula-graph.io/3.1.0/5.configurations-and-logs/1.configurations/6.kernel-config/#ulimit_-n

[akhilravuri-tul-scm]

@xiajingchun
sh-4.4# ulimit -n
1048576

[akhilravuri-tul-scm]

@xiajingchun why current nebula-go client connection pool does not have MaxWaitTime(time to wait till a session is available) kind of option.

Because if we use 1k we are getting too many open files and if we use 100(MaxConPool) and 1k(Vus) we are getting No valid connection in the idle queue and connection number has reached the pool capacity

[HarrisChu]

I am not sure, but suppose topology as below:

1. why 150 VUs using 97 Connections
   maybe the http server is slow. i.e. when http server send 98th query, the 1st query is finished and then return the connection to pool. So no need to create a new connection.

2. too many open files
   when raise the error? Immediately or after several seconds?
   I'm not sure if the http server is in the same host with nebula. maybe you can verify

netstat |grep {your http server port}|wc -l
# or
sudo lsof -p {graph pid}|wc -l
sudo lsof -p {storage pid}|wc -l
sudo lsof -p {http server pid}|wc -l

[akhilravuri-tul-scm]

@HarrisChu error is raised immediately if concurrent connection count is more than 65.

http server and nebula are on different host(node in cluster)