Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](split)Fixed the bug that batch mode split could not query data in multiple be scenarios. #46218 #46226

Merged
merged 1 commit into from
Jan 1, 2025

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #46218

…in multiple be scenarios. (#46218)

### What problem does this PR solve?
Problem Summary:
In multiple be scenarios, batch mode split sometimes could not query
data.

The reason is that the estimated `numApproximateSplits()` may be
relatively small, and the value after dividing the current number of be
may be 0. As a result, the split will not be distributed to be, and the
query result will be empty.
We need to take the max of the value after division and 1.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 31, 2024
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41110 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94f7b7f2c9608e2ceab0fa2251ebaadc3d3980e1, data reload: false

------ Round 1 ----------------------------------
q1	17593	7379	7214	7214
q2	2057	170	184	170
q3	10616	1138	1210	1138
q4	10335	723	694	694
q5	7821	2917	2906	2906
q6	234	145	143	143
q7	986	636	620	620
q8	9718	1960	2079	1960
q9	6732	6509	6542	6509
q10	8289	2282	2291	2282
q11	467	271	256	256
q12	409	213	209	209
q13	17790	2989	3041	2989
q14	236	215	214	214
q15	565	546	528	528
q16	699	609	634	609
q17	979	664	556	556
q18	7368	6692	6688	6688
q19	1393	1101	1071	1071
q20	468	201	205	201
q21	4428	3227	3173	3173
q22	1101	990	980	980
Total cold run time: 110284 ms
Total hot run time: 41110 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7369	7213	7256	7213
q2	327	236	239	236
q3	3042	2907	2955	2907
q4	2092	1758	1750	1750
q5	5700	5732	5812	5732
q6	223	147	141	141
q7	2297	1850	1799	1799
q8	3424	3509	3413	3413
q9	8977	8889	8914	8889
q10	3602	3605	3562	3562
q11	596	489	513	489
q12	834	609	591	591
q13	5492	3211	3177	3177
q14	295	275	274	274
q15	578	542	535	535
q16	707	665	681	665
q17	1836	1634	1596	1596
q18	8191	7815	7623	7623
q19	1677	1585	1566	1566
q20	2095	1867	1883	1867
q21	5653	5453	5560	5453
q22	1169	1040	1031	1031
Total cold run time: 66176 ms
Total hot run time: 60509 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197311 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94f7b7f2c9608e2ceab0fa2251ebaadc3d3980e1, data reload: false

query1	1255	918	922	918
query2	6228	2177	2147	2147
query3	10975	4530	4480	4480
query4	66531	29345	23545	23545
query5	5042	450	453	450
query6	413	176	170	170
query7	5634	311	321	311
query8	314	227	229	227
query9	9348	2717	2707	2707
query10	468	260	255	255
query11	17295	15289	15850	15289
query12	157	99	99	99
query13	1679	451	428	428
query14	10488	6837	7118	6837
query15	217	185	183	183
query16	6701	496	533	496
query17	1060	579	584	579
query18	1675	331	334	331
query19	236	174	170	170
query20	119	109	113	109
query21	57	43	45	43
query22	4856	4499	4362	4362
query23	34784	34491	34746	34491
query24	6509	2897	2893	2893
query25	529	408	435	408
query26	669	168	170	168
query27	1887	301	304	301
query28	4517	2506	2439	2439
query29	692	439	400	400
query30	222	158	153	153
query31	991	797	799	797
query32	64	51	52	51
query33	385	275	272	272
query34	895	483	489	483
query35	828	719	728	719
query36	1054	941	915	915
query37	114	68	74	68
query38	4019	3794	3774	3774
query39	1472	1429	1428	1428
query40	139	77	79	77
query41	50	46	46	46
query42	106	97	93	93
query43	551	520	508	508
query44	1165	805	806	805
query45	181	161	164	161
query46	1097	714	710	710
query47	1977	1835	1840	1835
query48	470	373	377	373
query49	723	375	372	372
query50	825	418	401	401
query51	7190	7217	7045	7045
query52	102	88	87	87
query53	247	176	179	176
query54	554	445	452	445
query55	75	76	74	74
query56	251	231	246	231
query57	1250	1132	1099	1099
query58	209	199	200	199
query59	3247	3216	3326	3216
query60	276	260	249	249
query61	108	105	103	103
query62	777	679	672	672
query63	214	183	188	183
query64	1420	667	640	640
query65	3285	3191	3239	3191
query66	706	322	303	303
query67	15887	15754	15686	15686
query68	3733	593	583	583
query69	428	260	265	260
query70	1203	1155	1114	1114
query71	341	253	255	253
query72	6300	4040	4000	4000
query73	747	343	347	343
query74	10082	9014	9059	9014
query75	3355	2637	2598	2598
query76	1829	1008	1131	1008
query77	492	274	267	267
query78	10574	9555	9656	9555
query79	1383	595	589	589
query80	862	428	414	414
query81	511	245	240	240
query82	1265	121	118	118
query83	239	140	142	140
query84	276	78	76	76
query85	882	308	298	298
query86	339	306	280	280
query87	4495	4487	4168	4168
query88	3709	2473	2414	2414
query89	408	289	285	285
query90	1992	185	180	180
query91	174	146	149	146
query92	67	47	49	47
query93	1754	550	551	550
query94	770	280	310	280
query95	346	259	257	257
query96	612	288	293	288
query97	3326	3207	3173	3173
query98	214	200	195	195
query99	1596	1302	1321	1302
Total cold run time: 317765 ms
Total hot run time: 197311 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94f7b7f2c9608e2ceab0fa2251ebaadc3d3980e1, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.09	0.10
query5	0.53	0.51	0.51
query6	1.15	0.73	0.73
query7	0.02	0.01	0.02
query8	0.03	0.03	0.05
query9	0.58	0.51	0.52
query10	0.55	0.55	0.57
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.61	0.59	0.59
query14	2.94	3.06	3.12
query15	0.90	0.83	0.83
query16	0.38	0.37	0.38
query17	0.99	1.01	1.01
query18	0.25	0.22	0.20
query19	2.00	1.90	2.07
query20	0.01	0.01	0.01
query21	15.36	0.60	0.60
query22	2.74	2.80	1.39
query23	16.92	0.94	0.94
query24	3.37	1.48	0.70
query25	0.25	0.12	0.05
query26	0.39	0.14	0.13
query27	0.05	0.05	0.05
query28	10.47	1.11	1.08
query29	12.56	3.20	3.20
query30	0.24	0.06	0.06
query31	2.84	0.39	0.38
query32	3.26	0.46	0.47
query33	2.97	3.03	3.08
query34	17.05	4.51	4.52
query35	4.57	4.57	4.55
query36	0.67	0.49	0.47
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.34 s
Total hot run time: 32.64 s

@morningman morningman merged commit d856a28 into branch-3.0 Jan 1, 2025
21 checks passed
@github-actions github-actions bot deleted the auto-pick-46218-branch-3.0 branch January 1, 2025 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants