Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: CN crashed by oom when running Insert on dup with 1000000 basic data and 1000 thread for about 20 minutes #20645

Open
1 task done
aressu1985 opened this issue Dec 8, 2024 · 7 comments
Assignees
Labels
kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone

Comments

@aressu1985
Copy link
Contributor

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

2.0-dev

Commit ID

6c37ff0

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
3*PROXY: 3C 7G
- OS type:
- Others:

Actual Behavior

sql:
insert into sbtest6 values(?,3214,'83868641912-28773972837-60736120486-75162659906-27563526494-20381887404-41576422241-93426793964-56405065102-33518432330','67847967377-48000963322-62604785301-91415491898-96926520291') on duplicate key update c='aaa';

ddl:
mysql> show create table sbtest1;
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| sbtest1 | CREATE TABLE sbtest1 (
id int NOT NULL,
k int NOT NULL DEFAULT 0,
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k_1 (k)
) |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

conflict rate is 100%

heap :
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22OeX%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bpod%20%3D%20%5C%22nightly-regression-dis-tp-cn-788hd%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D,%22profileTypeId%22:%22memory:inuse_space:bytes:space:bytes%22%7D%5D,%22range%22:%7B%22from%22:%221733644908363%22,%22to%22:%221733645073929%22%7D%7D%7D&schemaVersion=1&orgId=1

image

Expected Behavior

No response

Steps to Reproduce

1. git clone mo-sysbench and mo-load
2. prepare data:
   cd mo-sysbench 
   sysbench  --mysql-host=xxxxx --mysql-port=6001 --mysql-user=dump --mysql-password=111 oltp_common.lua --mysql-db=sysbench_db --tables=10 --table_size=1000000 --threads=100 --time=30 --report-interval=10    cleanup;
sysbench  --mysql-host=xxxxxx --mysql-port=6001 --mysql-user=dump --mysql-password=111 oltp_common.lua --mysql-db=sysbench_db --tables=10 --table_size=1000000 --threads=100 --time=30 --report-interval=10  --range_selects=off --point_selects=1 --auto-inc=off  prepare;

3. run test
   cd mo-load
   ./start.sh -c cases/sysbench/simple_insert_duplicate_10_1000000_prepare -h xxxxxx -t 1000 -d 50 -g -b sysbench_db

Additional information

No response

@heni02
Copy link
Contributor

heni02 commented Dec 9, 2024

单机insert duplicate并发1000,主键冲突100%测试30分钟左右oom
2.0-dev:
oom时间:Dec 9 10:54:53
image

Dec 9 10:53 heap/malloc/cpu文件:
hn_download (1).zip

复现步骤同上

@heni02 heni02 added this to the 2.1.0 milestone Dec 10, 2024
@heni02 heni02 added the severity/s0 Extreme impact: Cause the application to break down and seriously affect the use label Dec 10, 2024
@aunjgr
Copy link
Contributor

aunjgr commented Dec 10, 2024

这问题是一直都有的,还是最近才出现的?

@aunjgr
Copy link
Contributor

aunjgr commented Dec 17, 2024

not working on it today

@aunjgr aunjgr assigned ouyuanning and unassigned aunjgr Dec 21, 2024
@ouyuanning
Copy link
Contributor

刚接手。明天看

@ouyuanning
Copy link
Contributor

ouyuanning commented Dec 23, 2024

从profile看。heap占用如下:
image

堆外占用如下:
image

@ouyuanning
Copy link
Contributor

我在该commit id和最新的main下,好像是会panic的
image

@ouyuanning
Copy link
Contributor

panic的先在这里处理 #20879

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Development

No branches or pull requests

5 participants