Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](azure) Add FE azure object storage UT and fix log #46201

Merged
merged 2 commits into from
Jan 2, 2025

Conversation

gavinchou
Copy link
Contributor

What problem does this PR solve?

Add an UT for azure blob storage in FE, this is a test framework, more UTs should be added to cover all functionalities.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 31, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32580 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be025045117f498b415faee3985f4b6ab280b2a8, data reload: false

------ Round 1 ----------------------------------
q1	17588	6183	6066	6066
q2	2054	298	167	167
q3	10538	1235	761	761
q4	10239	875	444	444
q5	8223	2133	1988	1988
q6	202	184	152	152
q7	894	753	615	615
q8	9223	1350	1184	1184
q9	5190	4849	4967	4849
q10	6774	2307	1873	1873
q11	493	292	268	268
q12	358	389	237	237
q13	17786	3606	2985	2985
q14	229	231	223	223
q15	547	502	507	502
q16	607	626	572	572
q17	588	857	322	322
q18	6801	6460	6264	6264
q19	2575	983	585	585
q20	303	315	185	185
q21	2869	2220	2025	2025
q22	362	338	313	313
Total cold run time: 104443 ms
Total hot run time: 32580 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6389	6243	6215	6215
q2	239	326	229	229
q3	2261	2641	2340	2340
q4	1445	1842	1405	1405
q5	4374	4727	4924	4727
q6	186	177	146	146
q7	2089	1970	1833	1833
q8	2707	2844	2677	2677
q9	7354	7322	7288	7288
q10	3090	3358	2725	2725
q11	574	517	514	514
q12	715	762	602	602
q13	3418	3765	3132	3132
q14	301	304	284	284
q15	571	509	513	509
q16	638	691	642	642
q17	1233	1729	1251	1251
q18	7641	7538	7382	7382
q19	895	1100	1168	1100
q20	2031	2055	1951	1951
q21	5783	5226	4868	4868
q22	618	633	611	611
Total cold run time: 54552 ms
Total hot run time: 52431 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196985 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be025045117f498b415faee3985f4b6ab280b2a8, data reload: false

query1	1302	947	909	909
query2	6456	2259	2371	2259
query3	10999	4605	4764	4605
query4	33065	23599	23581	23581
query5	3811	604	460	460
query6	286	199	206	199
query7	4003	473	295	295
query8	305	240	233	233
query9	9379	2694	2666	2666
query10	442	292	256	256
query11	18028	15843	15114	15114
query12	155	107	102	102
query13	1567	522	392	392
query14	9194	6769	7024	6769
query15	272	231	203	203
query16	7922	643	456	456
query17	1626	769	601	601
query18	2094	396	315	315
query19	196	194	167	167
query20	125	116	118	116
query21	205	128	112	112
query22	4759	4718	4515	4515
query23	34637	33744	33750	33744
query24	7291	2303	2344	2303
query25	511	450	406	406
query26	765	278	157	157
query27	2594	467	324	324
query28	5988	2464	2451	2451
query29	698	570	424	424
query30	210	189	150	150
query31	960	920	861	861
query32	92	59	56	56
query33	486	347	295	295
query34	765	855	513	513
query35	819	837	853	837
query36	1040	1062	994	994
query37	121	100	76	76
query38	4352	4296	4180	4180
query39	1534	1474	1468	1468
query40	208	119	103	103
query41	47	46	46	46
query42	123	105	103	103
query43	520	539	522	522
query44	1314	817	824	817
query45	189	200	177	177
query46	886	1075	654	654
query47	2020	2013	1981	1981
query48	394	420	325	325
query49	737	485	385	385
query50	644	679	389	389
query51	7286	7285	7256	7256
query52	108	103	92	92
query53	225	254	187	187
query54	492	509	397	397
query55	84	76	80	76
query56	265	263	254	254
query57	1250	1282	1191	1191
query58	248	232	241	232
query59	3278	3245	3250	3245
query60	285	258	255	255
query61	114	109	109	109
query62	866	820	767	767
query63	226	190	190	190
query64	3651	1018	739	739
query65	3346	3245	3321	3245
query66	847	436	313	313
query67	16727	15915	15623	15623
query68	9120	736	520	520
query69	485	300	263	263
query70	1243	1076	1129	1076
query71	435	284	269	269
query72	6423	3995	3815	3815
query73	671	746	357	357
query74	9790	9094	9068	9068
query75	3901	3142	2637	2637
query76	3601	1187	761	761
query77	764	349	281	281
query78	10546	10215	9489	9489
query79	6335	866	576	576
query80	716	523	424	424
query81	497	273	232	232
query82	660	151	119	119
query83	193	268	149	149
query84	282	92	68	68
query85	810	352	333	333
query86	348	324	307	307
query87	4580	4493	4601	4493
query88	5039	2196	2207	2196
query89	440	325	291	291
query90	1946	187	186	186
query91	132	142	103	103
query92	64	51	49	49
query93	3013	831	520	520
query94	658	396	295	295
query95	387	265	252	252
query96	496	606	276	276
query97	2728	2842	2704	2704
query98	224	198	214	198
query99	1700	1561	1441	1441
Total cold run time: 303661 ms
Total hot run time: 196985 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be025045117f498b415faee3985f4b6ab280b2a8, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.06
query4	1.63	0.11	0.10
query5	0.39	0.42	0.43
query6	1.14	0.66	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.52
query10	0.55	0.57	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.61	0.60
query14	2.73	2.82	2.75
query15	0.89	0.83	0.83
query16	0.39	0.39	0.39
query17	1.06	1.04	1.04
query18	0.22	0.20	0.20
query19	1.87	1.85	2.03
query20	0.02	0.01	0.01
query21	15.36	0.92	0.58
query22	0.76	0.76	0.61
query23	15.44	1.46	0.55
query24	2.91	1.96	0.84
query25	0.22	0.19	0.12
query26	0.19	0.15	0.14
query27	0.06	0.05	0.04
query28	14.16	1.49	1.04
query29	12.54	3.91	3.26
query30	0.25	0.09	0.06
query31	2.84	0.58	0.38
query32	3.23	0.54	0.45
query33	3.16	3.14	3.11
query34	16.63	5.10	4.50
query35	4.46	4.51	4.48
query36	0.65	0.49	0.52
query37	0.10	0.06	0.05
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.1 s
Total hot run time: 31.21 s

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32567 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be025045117f498b415faee3985f4b6ab280b2a8, data reload: false

------ Round 1 ----------------------------------
q1	17569	6102	6032	6032
q2	2045	284	161	161
q3	10459	1189	736	736
q4	10229	843	436	436
q5	8033	2177	1962	1962
q6	206	183	146	146
q7	877	766	596	596
q8	9227	1333	1125	1125
q9	5219	4894	4870	4870
q10	6762	2290	1872	1872
q11	500	275	252	252
q12	340	359	231	231
q13	17777	3558	3036	3036
q14	234	232	219	219
q15	563	518	507	507
q16	616	613	582	582
q17	568	839	322	322
q18	6855	6525	6498	6498
q19	1602	961	535	535
q20	300	311	189	189
q21	2808	2188	1962	1962
q22	355	331	298	298
Total cold run time: 103144 ms
Total hot run time: 32567 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6229	6227	6183	6183
q2	231	330	228	228
q3	2280	2685	2339	2339
q4	1434	1901	1399	1399
q5	4341	4719	4747	4719
q6	188	178	141	141
q7	2075	1915	1830	1830
q8	2619	2856	2690	2690
q9	7290	7307	7316	7307
q10	3105	3381	2899	2899
q11	573	509	491	491
q12	656	744	617	617
q13	3388	3717	3063	3063
q14	276	311	292	292
q15	572	516	504	504
q16	670	683	657	657
q17	1340	1746	1274	1274
q18	7589	7460	7421	7421
q19	836	1158	1088	1088
q20	2000	2076	1925	1925
q21	5745	5329	4940	4940
q22	581	581	570	570
Total cold run time: 54018 ms
Total hot run time: 52577 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197070 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be025045117f498b415faee3985f4b6ab280b2a8, data reload: false

query1	1284	939	917	917
query2	6420	2409	2424	2409
query3	11126	4818	5143	4818
query4	33597	23721	23301	23301
query5	3688	634	457	457
query6	275	206	182	182
query7	4001	485	316	316
query8	309	272	256	256
query9	9419	2732	2726	2726
query10	476	311	248	248
query11	17933	15371	15178	15178
query12	171	108	106	106
query13	1583	535	410	410
query14	10185	6813	7348	6813
query15	239	209	192	192
query16	7986	642	452	452
query17	1538	792	586	586
query18	2065	414	309	309
query19	197	181	164	164
query20	121	118	115	115
query21	204	146	106	106
query22	4653	4524	4432	4432
query23	34832	33818	33670	33670
query24	6833	2287	2277	2277
query25	480	464	410	410
query26	852	272	158	158
query27	2539	455	328	328
query28	6011	2463	2459	2459
query29	646	564	436	436
query30	208	180	152	152
query31	979	923	860	860
query32	68	56	58	56
query33	476	366	302	302
query34	766	858	529	529
query35	849	885	778	778
query36	1034	1083	994	994
query37	115	101	83	83
query38	4374	4357	4081	4081
query39	1660	1480	1460	1460
query40	213	116	104	104
query41	49	49	46	46
query42	116	100	99	99
query43	532	555	503	503
query44	1346	830	825	825
query45	191	184	206	184
query46	875	1071	649	649
query47	2026	2011	1952	1952
query48	395	411	334	334
query49	706	471	387	387
query50	631	659	412	412
query51	7252	7303	7216	7216
query52	106	103	103	103
query53	229	254	185	185
query54	491	506	407	407
query55	93	81	85	81
query56	255	260	253	253
query57	1269	1244	1197	1197
query58	243	228	233	228
query59	3384	3486	3384	3384
query60	276	261	259	259
query61	112	109	116	109
query62	841	837	742	742
query63	235	188	200	188
query64	3404	1042	663	663
query65	3325	3284	3238	3238
query66	753	406	386	386
query67	16588	15789	15526	15526
query68	9889	757	513	513
query69	478	293	244	244
query70	1195	1185	1125	1125
query71	438	285	256	256
query72	6258	3918	3946	3918
query73	658	748	350	350
query74	10012	9002	9173	9002
query75	4617	3164	2656	2656
query76	5458	1183	767	767
query77	1029	366	277	277
query78	11035	10240	9542	9542
query79	3736	903	581	581
query80	735	515	491	491
query81	478	266	225	225
query82	305	150	123	123
query83	194	167	152	152
query84	286	84	67	67
query85	745	388	301	301
query86	358	303	304	303
query87	4409	4490	4589	4490
query88	3238	2251	2211	2211
query89	433	348	295	295
query90	2137	187	183	183
query91	134	128	113	113
query92	61	54	51	51
query93	2219	838	524	524
query94	676	383	282	282
query95	330	259	249	249
query96	491	606	279	279
query97	2762	2815	2671	2671
query98	256	199	191	191
query99	1633	1605	1452	1452
Total cold run time: 302811 ms
Total hot run time: 197070 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.95 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be025045117f498b415faee3985f4b6ab280b2a8, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.10	0.11
query5	0.44	0.42	0.43
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.59	0.49	0.50
query10	0.56	0.55	0.56
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.61	0.59
query14	2.74	2.84	2.75
query15	0.90	0.83	0.83
query16	0.39	0.39	0.38
query17	1.08	1.04	1.06
query18	0.22	0.21	0.21
query19	1.92	1.89	2.03
query20	0.01	0.01	0.02
query21	15.36	0.98	0.58
query22	0.77	0.75	0.66
query23	15.69	1.42	0.59
query24	3.61	2.30	0.34
query25	0.16	0.08	0.23
query26	0.30	0.15	0.14
query27	0.04	0.05	0.05
query28	13.38	1.47	1.04
query29	12.59	3.99	3.29
query30	0.25	0.08	0.06
query31	2.83	0.61	0.38
query32	3.23	0.53	0.48
query33	3.13	3.12	3.12
query34	16.79	5.13	4.53
query35	4.54	4.53	4.55
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.73 s
Total hot run time: 30.95 s

Copy link
Contributor

github-actions bot commented Jan 1, 2025

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
@dataroaring dataroaring merged commit 902151f into apache:master Jan 2, 2025
27 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 2, 2025
Add an UT for azure blob storage in FE, this is a test framework, more
UTs should be added to cover all functionalities.
gavinchou added a commit that referenced this pull request Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants