Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[upgrade](thirdparty) upgrade librdkafka from 1.8.2 to 2.0.2 #28210

Merged
merged 2 commits into from
Dec 12, 2023
Merged

[upgrade](thirdparty) upgrade librdkafka from 1.8.2 to 2.0.2 #28210

merged 2 commits into from
Dec 12, 2023

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Dec 10, 2023

Proposed changes

Some error happen when using routine load

[INTERNAL_ERROR]Message at offset XXX might be too large to fetch, try increasing receive.message.max.bytes

Refer to confluentinc/librdkafka#2993, we should upgrade librdkafka version to avoid this bug.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@sollhui
Copy link
Contributor Author

sollhui commented Dec 10, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit de86ce147904c764f3d8a262c26386cf7095ec9e, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4707	4472	4482	4472
q2	372	141	157	141
q3	1454	1244	1231	1231
q4	1105	884	907	884
q5	3156	3168	3132	3132
q6	253	128	128	128
q7	1008	481	479	479
q8	2183	2208	2188	2188
q9	6663	6664	6672	6664
q10	3211	3268	3255	3255
q11	326	201	192	192
q12	356	206	208	206
q13	4568	3805	3780	3780
q14	239	223	216	216
q15	565	517	527	517
q16	443	389	392	389
q17	997	570	538	538
q18	7426	7064	7079	7064
q19	1556	1265	1458	1265
q20	513	314	311	311
q21	3080	2696	2715	2696
q22	359	284	293	284
Total cold run time: 44540 ms
Total hot run time: 40032 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4425	4426	4402	4402
q2	268	166	173	166
q3	3569	3566	3544	3544
q4	2400	2388	2392	2388
q5	5775	5757	5757	5757
q6	242	119	123	119
q7	2425	1869	1892	1869
q8	3527	3523	3541	3523
q9	9046	9055	9040	9040
q10	3904	3967	3987	3967
q11	496	392	390	390
q12	779	599	597	597
q13	4304	3592	3583	3583
q14	284	255	263	255
q15	567	523	520	520
q16	496	470	461	461
q17	1881	1842	1872	1842
q18	8795	8400	8411	8400
q19	1731	1765	1738	1738
q20	2269	1939	1939	1939
q21	6608	6255	6213	6213
q22	502	425	430	425
Total cold run time: 64293 ms
Total hot run time: 61138 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.12 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17224444680 Bytes

dataroaring
dataroaring previously approved these changes Dec 10, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 10, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

TangSiyang2001
TangSiyang2001 approved these changes Dec 11, 2023
View reviewed changes
Copy link
Collaborator

@TangSiyang2001 TangSiyang2001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wm1581066 wm1581066 added the usercase Important user case type label label Dec 11, 2023
@sollhui
Copy link
Contributor Author

sollhui commented Dec 11, 2023

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 11, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.18 seconds
stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17224131442 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c6c3a9d144dda9004d22c7a0c50332a8790fa01a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4672	4426	4482	4426
q2	368	160	138	138
q3	1468	1244	1198	1198
q4	1110	895	872	872
q5	3156	3141	3164	3141
q6	250	128	130	128
q7	1023	493	478	478
q8	2226	2223	2179	2179
q9	7279	6692	6659	6659
q10	3224	3256	3277	3256
q11	329	199	198	198
q12	345	214	205	205
q13	5553	3815	3812	3812
q14	237	208	223	208
q15	562	519	519	519
q16	442	399	399	399
q17	1009	596	548	548
q18	7853	7078	8037	7078
q19	1521	1292	1477	1292
q20	814	302	368	302
q21	3138	2750	2692	2692
q22	352	286	287	286
Total cold run time: 46931 ms
Total hot run time: 40014 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4411	4419	4365	4365
q2	272	163	169	163
q3	3555	3562	3558	3558
q4	2396	2376	2389	2376
q5	5771	5747	5735	5735
q6	238	122	121	121
q7	2415	1880	1865	1865
q8	3520	3524	3537	3524
q9	9125	9074	9100	9074
q10	3932	3983	3980	3980
q11	501	369	393	369
q12	775	591	602	591
q13	4312	3580	3593	3580
q14	292	249	264	249
q15	561	524	524	524
q16	507	454	476	454
q17	1887	1880	1864	1864
q18	8666	8269	8493	8269
q19	1724	1764	1746	1746
q20	2273	1961	1942	1942
q21	6548	6232	6174	6174
q22	503	444	429	429
Total cold run time: 64184 ms
Total hot run time: 60952 ms

morningman
morningman approved these changes Dec 11, 2023
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 11, 2023
freemandealer
freemandealer approved these changes Dec 11, 2023
View reviewed changes
Copy link
Contributor

@freemandealer freemandealer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sollhui
Copy link
Contributor Author

sollhui commented Dec 11, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c6c3a9d144dda9004d22c7a0c50332a8790fa01a, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4706	4459	4475	4459
q2	370	154	156	154
q3	1454	1271	1195	1195
q4	1115	911	885	885
q5	3143	3164	3128	3128
q6	253	128	129	128
q7	991	489	476	476
q8	2210	2223	2185	2185
q9	6670	6701	6690	6690
q10	3190	3253	3276	3253
q11	325	207	213	207
q12	346	211	209	209
q13	4755	3892	3832	3832
q14	236	217	219	217
q15	570	525	523	523
q16	447	395	397	395
q17	1001	541	552	541
q18	8359	7568	7095	7095
q19	1526	1358	1439	1358
q20	521	342	359	342
q21	3077	2624	2652	2624
q22	343	275	285	275
Total cold run time: 45608 ms
Total hot run time: 40171 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4443	4378	4387	4378
q2	268	165	182	165
q3	3574	3564	3569	3564
q4	2376	2379	2374	2374
q5	5752	5750	5752	5750
q6	240	122	121	121
q7	2401	1887	1855	1855
q8	3522	3531	3538	3531
q9	9092	9050	9039	9039
q10	3906	3983	4004	3983
q11	501	388	382	382
q12	770	590	596	590
q13	4305	3593	3542	3542
q14	279	263	268	263
q15	569	520	516	516
q16	499	475	490	475
q17	1886	1857	1880	1857
q18	8755	8655	8240	8240
q19	1789	1757	1776	1757
q20	2256	1950	1938	1938
q21	6531	6222	6167	6167
q22	516	417	429	417
Total cold run time: 64230 ms
Total hot run time: 60904 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.13 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17219474263 Bytes

@morningman morningman merged commit 8a7b0e5 into apache:master Dec 12, 2023
@sollhui sollhui deleted the update_librdkafka branch December 12, 2023 07:06
xzj7019 pushed a commit to xzj7019/doris that referenced this pull request Dec 13, 2023
@sollhui
[upgrade](thirdparty) upgrade librdkafka from 1.8.2 to 2.0.2 (apache#…
594b392 
…28210)

Some error happen when using routine load
```
[INTERNAL_ERROR]Message at offset XXX might be too large to fetch, try increasing receive.message.max.bytes
```
Refer to confluentinc/librdkafka#2993, we should upgrade librdkafka version to avoid this bug.
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@sollhui
[upgrade](thirdparty) upgrade librdkafka from 1.8.2 to 2.0.2 (apache#…
750cd12 
…28210)

Some error happen when using routine load
```
[INTERNAL_ERROR]Message at offset XXX might be too large to fetch, try increasing receive.message.max.bytes
```
Refer to confluentinc/librdkafka#2993, we should upgrade librdkafka version to avoid this bug.
@sollhui sollhui mentioned this pull request Dec 17, 2023
Merged
@sollhui sollhui mentioned this pull request Dec 27, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 28, 2023
@sollhui
[upgrade](thirdparty) upgrade librdkafka from 1.8.2 to 1.9.2(#28210) (#…
47271f0 
…29160)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@sollhui @HappenLee
[upgrade](thirdparty) upgrade librdkafka from 1.8.2 to 2.0.2 (apache#…
1a0b713 
…28210)

Some error happen when using routine load
```
[INTERNAL_ERROR]Message at offset XXX might be too large to fetch, try increasing receive.message.max.bytes
```
Refer to confluentinc/librdkafka#2993, we should upgrade librdkafka version to avoid this bug.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@freemandealer freemandealer freemandealer approved these changes

@TangSiyang2001 TangSiyang2001 TangSiyang2001 approved these changes

@dataroaring dataroaring dataroaring left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@sollhui @doris-robot @morningman @freemandealer @TangSiyang2001 @dataroaring @xiaokang @wm1581066