Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot set both parallelism and namespaceParallelism #10985

Open
2 of 3 tasks
yuzhouliu9 opened this issue Apr 25, 2023 · 7 comments · May be fixed by #14039
Open
2 of 3 tasks

Cannot set both parallelism and namespaceParallelism #10985

yuzhouliu9 opened this issue Apr 25, 2023 · 7 comments · May be fixed by #14039
Assignees
@Joibel
Labels
area/docs Incorrect, missing, or mistakes in docs type/bug

Comments

@yuzhouliu9
Copy link

yuzhouliu9 commented Apr 25, 2023 •
edited
Loading

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

In the workflow-controller-configmap, setting:

  parallelism: "10"
  namespaceParallelism: "5"

Expectation
The global number of running workflows to be limited at 10, and the running workflows per namespace limited to 5.

Actual
However, in testing, argo-workflows limits the global number of workflows to 5.

Having each setting individually works as expected, but not when they are together. This is a cluster installation.

Version

v3.4.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Simple workflow that sleeps 5 mins.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: parallelism-test-
spec:
  entrypoint: test
  podGC:
    strategy: OnWorkflowCompletion
  ttlStrategy:
    secondsAfterCompletion: 60
  templates:
  - name: test
    steps:
    - - name: test-1
        template: sleep
  - name: sleep
    container:
      image: alpine:latest
      command: [sleep, "300"]

Script that submits the above workflow 10 times to a namespace

#!/bin/bash

for i in {1..10}
do
    argo submit workflow.yaml -n $1 &
done

Submit to default and devops namespace

./submit.sh default
./submit.sh devops

Screenshots from UI, notice the number of running workflows for each namespace, and stuck waiting.

Screenshot 2023-04-26 at 4 28 52 PM

Screenshot 2023-04-26 at 4 27 26 PM

Logs from the workflow controller

controller_logs.txt

Logs from in your workflow's wait container

I don't think this is relevant.
@tico24
Copy link
Member

tico24 commented Apr 26, 2023 •
edited
Loading

This issue is reproducible. I can provide logs if needed.

Yes please. That's why they are requested in the template.

It's actually hard to understand your issue based on your example. I would fully expect your example workflow to only run a maximum of 5 times in parallel.

@tico24 tico24 added the problem/more information needed Not enough information has been provide to diagnose this issue. label Apr 26, 2023
@caelan-io caelan-io added the area/docs Incorrect, missing, or mistakes in docs label Apr 26, 2023
@yuzhouliu9
Copy link
Author

Apologies for the confusion. Updated description with better example, and the bash scripts I used to reproduce. Logs and screenshots attached.

@terrytangyuan
Copy link
Member

This looks like a bug where one of the configurations is overridden by another. Would anyone like to submit a PR?

@tico24 tico24 removed the problem/more information needed Not enough information has been provide to diagnose this issue. label Apr 27, 2023
@Guillermogsjc
Copy link

it is possible to know which overrides which in the configuration? namespaceParallelism > parallelism ?

@stale
Copy link

stale bot commented Jun 18, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

@stale stale bot added problem/stale This has not had a response in some time and removed problem/stale This has not had a response in some time labels Jun 18, 2023
@yuzhouliu9
Copy link
Author

it is possible to know which overrides which in the configuration? namespaceParallelism > parallelism ?

global parallelism overrides namespaceParallelism but uses the value from namespaceParallelism. Please read the original issue description to see the behavior. It's straight forward to reproduce all outlined.

@stale
Copy link

stale bot commented Sep 17, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the problem/stale This has not had a response in some time label Sep 17, 2023
@terrytangyuan terrytangyuan removed the problem/stale This has not had a response in some time label Sep 20, 2023
@Joibel Joibel self-assigned this Nov 29, 2024
@isubasinghe isubasinghe linked a pull request Dec 30, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Joibel Joibel

Labels
area/docs Incorrect, missing, or mistakes in docs type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: ensure namespace parallelism and parallelism work together. Fixes #10985 pipekit/argo-workflows
6 participants
@Joibel @terrytangyuan @Guillermogsjc @tico24 @caelan-io @yuzhouliu9