HGETALL on 2,7 million recrods (9 fields) took 40 seconds

[pnthai88]

I'm using pipeline and concurrent.futures.ThreadPoolExecutor . Any idea to get data faster?
Whole cluster in Docker,

While doing hgetall on mass, CPU does not consume much

Any idea?
Thanks.

[Grokzen]

@pnthai88 This is not a issue but a question so moved it to discussions section.

Basically i do not know of any other way of getting data faster then what you are doing, due to the huge amount of data you will hit some basic limitations just for the simple fact that converting 1000 values will take X ammount of time no matter what you do and allocating memory and converting things in python will just take time and with 2.7 mil records that will take time.

However, i would not recommend that you use Threads for this kind of work, depending on what type of work you do with the data itself you should really look at multiprocessing instead of maximize the number of cores you can run in parallel to make fetching data possibly faster, threads are notorious for begin slow.

In general whenever i see a thread like this or a question about threads or multiprocesses or similar topic i always recommend this talk (David Beazley - Python concurrency From the Ground Up: LIVE! - PyCon 2015) https://www.youtube.com/watch?v=MCs5OvhV9S4 as it is possibly one of the best talks ever and it deals with threads in a deeper way then anything out there.

[pnthai88]

@Grokzen
You're right. HGetall is slow, i replaced it with compressed json and result is better.

My production does not use that much data, just doing some insane work if use redis as primary database, what i'm doing are:
1/ store a table as user defined columns.
2/ implement its key by zadd so everytime people want to get all columns without doing command keys or scan_iter (your scan_iter is the most beautiful thing)
3/ fetch whole table and search what user want.

Yea , i tried with process pool but seem not compatible with flask. Still working on it :)

Thanks for your advice, have a good day!

[pnthai88]

Well, first i tried with thread pool and i think i hit the limit of pipeline at 40k ops/sec.

Then I tried with process pool with 8 workers and easy reach 160k ops/sec.

I use another computer to spawn another process pool and easy hit 160k ops/sec in both clients total 320k ops/sec.

I do some benchmark on cluster and got 10 million ops/sec

I think , i should use something like multiple connection or different pipeline to improve its speed.

[Grokzen]

@pnthai88 Any mechanism where you can utilize more cpu cores in true parallel and not the more fake one with threads will increase your performance. A very common solution in a web scenario is that you can simulate to run any number of web servers to stress test it instead of just checking with a single webserver you can test it with 10 and then the threadpool to simualte flask inside it.