Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#2261]: feat(Coordinator): Introduce banned id manager and checker #2255

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[#2261]: feat(Coordinator): Introduce banned id manager and checker #2255

wants to merge 4 commits into from

Conversation

maobaolong
Copy link
Member

@maobaolong maobaolong commented Nov 18, 2024 •
edited
Loading

What changes were proposed in this pull request?

Introduce banned id manager and checker

Why are the changes needed?

Provide a common way to banned an app to fallback to SORT shuffle manager.

The app id can be extracted from a specific spark conf.

Fix: #2261

Does this PR introduce any user-facing change?

$  curl -X POST http://localhost:19998/banned/reload \
-H "Content-Type: application/json" \
-d '{"version": "v1.0", "ids": ["123", "456", "789"]}'

{"code":0,"data":"success","errMsg":"success"}%                                                                                                                                                                                                                                                                                                                           

$  curl http://localhost:19998/banned/version

{"code":0,"data":"v1.0","errMsg":"success"}%

How was this patch tested?

New added UT.

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 18, 2024 •
edited
Loading

Test Results

 2 971 files  +15   2 971 suites  +15   6h 28m 13s ⏱️ ±0s
 1 094 tests + 2   1 092 ✅ + 2   2 💤 ±0  0 ❌ ±0 
13 715 runs  +30  13 685 ✅ +30  30 💤 ±0  0 ❌ ±0 

Results for commit ea1f3b0. ± Comparison against base commit 99eaa9d.

♻️ This comment has been updated with latest results.

@maobaolong maobaolong force-pushed the implBannedUsIdChecker branch from 031dac0 to 572c564 Compare November 18, 2024 14:35
@maobaolong maobaolong force-pushed the implBannedUsIdChecker branch from 572c564 to c72a5c9 Compare November 19, 2024 01:16
@zuston zuston self-requested a review November 19, 2024 02:04
@zuston
Copy link
Member

zuston commented Nov 19, 2024

Thanks for proposing this, I think I will review this in next days.

@maobaolong maobaolong mentioned this pull request Nov 19, 2024
Merged
@jerqi
Copy link
Contributor

jerqi commented Nov 25, 2024

You would better create an issue for a feature or a bug.

@maobaolong maobaolong changed the title [MINOR]: feat(Coordinator): Introduce banned id manager and checker [#2261]: feat(Coordinator): Introduce banned id manager and checker Nov 25, 2024
@maobaolong
Copy link
Member Author

@jerqi Thank you for remind me this, I created an issue mark it as a feature.

@maobaolong
Copy link
Member Author

@zuston Would you like to take a look this week? Thank you.

@zuston
Copy link
Member

zuston commented Dec 4, 2024

You would better create an issue for a feature or a bug.

Sorry for the late reply, I'm struggle with the busy work. I will review this tonight.

@zuston
Copy link
Member

zuston commented Dec 4, 2024

I do a quick review about this issue, and I hope I have gotten your motivation that you want to remote control the spark app to determinze whether to use uniffle or sort based shuffle manager. If so, I think the current code base have provided these ability to implement custom plugin to control this.

For the client side, you could implement the custom shuffle manager to extend the DelegationShuffleManager to inject the extra info into the access request's extraProperties protobuf var.

For the coordinator side, you also could implement custom checker to retrieve above extra properties to control app pass or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zuston zuston Awaiting requested review from zuston

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] Introduce a common BannedManager to ban the found bad application
3 participants
@maobaolong @zuston @jerqi