We use SysBench to perform load testing.
https://github.com/akopytov/sysbench
Before starting the connectors, create the topic with max partitions
rpk topic create SERVER5432.sbtest.sbtest1 -p 6
To Run SysBench tests, run the debezium-connector-setup-sysbench.sh script in deploy folder to
create the MySQL debezium connector. The connector is setup to read from the sbtest table required by SysBench
For sink, the default sink-connector-setup-schema-registry.sh script can be executed to create
the ClickHouse Sink Connector.
| SysBench Test | Status |
|---|---|
| oltp_insert | Pass |
| oltp_read_write | Pass |
| oltp_update_index | Pass |
| oltp_update_non_index | Pass |
| oltp_delete | Pass |
| bulk_insert | Pass |
sysbench/run_sysbench_insert_load_test.sh script executes the oltp_insert lua script in Sysbench.
\
select database, table, event_type, partition_id, count() c, round(avg(rows)) from system.part_log where event_date >= today() and event_type = 'NewPart' group by database, table, event_type, partition_id order by c desc\
select sum(tmp.rows), tmp.event_time from ( select rows,event_time ,event_type from system.part_log pl where database='test' and table='sbtest1' and event_type='NewPart' order by event_time desc) tmp group by tmp.event_time
With the SysBench insert tests(6 Kafka partitions), the following are the numbers we observed on 8-core i7 64 GB RAM instance.
ClickHouse Rows Insertion Rate: 9k/second, 2.1 MB/second.
select database, table, event_type, partition_id, count() c, round(avg(rows)) from system.part_log where event_date >= today() and event_type = 'NewPart' group by database, table, event_type, partition_id order by c desc
Target: 5 threads , 600k/second
show binary logs;
show binlog events in mysql-bin.000003`