[Umbrella] Flink Support (Help Wanted)

[jerqi]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the proposal

Flink Support is complex topic. We set up an issue here. In Flink Community,
Flink Shuffle will have much change because of https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage.
We should discuss actively with Flink community.
Their discussion thread
https://lists.apache.org/thread/nwrqd5jtqwks89tbxpcrgto6r2bhdhno

Task list

xx

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[slfan1989]

This document can also be used as a reference.

FLIP-31: Pluggable Shuffle Service

FLIP-148: Introduce Sort-Based Blocking Shuffle to Flink

FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework

FLIP-199: Change some default config values of blocking shuffle for better usability

FLIP-209: Support to run multiple shuffle plugins in one session cluster

FLIP-301: Hybrid Shuffle supports Remote Storage

Vedio: Pluggable Shuffle Service and Unaligned Checkpoints

flink-remote-shuffle

https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks

[slfan1989]

@jerqi I will follow up the issue with chenglong(Big Data Development Engineer In Lixiang Car).

[jerqi]

Thanks @slfan1989

[chenglong-bigdata]

with me

[jerqi]

with me

Thanks for your help.

[pegasas]

Hi, @jerqi and the team,

I am interested in this project due to investigating Flink batch shuffle service in my previous job.

Would I still be able to join this project?

[jerqi]

Hi, @jerqi and the team,

I am interested in this project due to investigating Flink batch shuffle service in my previous job.

Would I still be able to join this project?

You're welcome. Currently, we have a regular meeting. If you have interest and can Speak Chinese, you can join the Wechat Group, the meeting link is the Wechat Group.

[zuston]

cc @slfan1989 Do you already have a draft implementation? I remember you mentioned that in the online meeting

[slfan1989]

cc @slfan1989 Do you already have a draft implementation? I remember you mentioned that in the online meeting

@zuston My draft version code has been basically completed (but some details still need to be optimized), I will submit PR to the community as soon as possible.