Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(config): make dask autoscaler configurable #604

Merged
merged 1 commit into from
Nov 11, 2024

Conversation

Alputer
Copy link
Member

@Alputer Alputer commented Sep 20, 2024

@Alputer Alputer self-assigned this Sep 20, 2024
Copy link

codecov bot commented Sep 20, 2024

Codecov Report

Attention: Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.61%. Comparing base (1515865) to head (db9ac05).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
reana_workflow_controller/consumer.py 33.33% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #604      +/-   ##
==========================================
+ Coverage   76.46%   76.61%   +0.14%     
==========================================
  Files          17       17              
  Lines        1831     1847      +16     
==========================================
+ Hits         1400     1415      +15     
- Misses        431      432       +1     
Files with missing lines Coverage Δ
reana_workflow_controller/config.py 93.95% <100.00%> (+0.08%) ⬆️
reana_workflow_controller/dask.py 76.36% <100.00%> (+1.85%) ⬆️
reana_workflow_controller/consumer.py 55.21% <33.33%> (-0.07%) ⬇️

@Alputer Alputer changed the title Dask autoscaler config feat(dask): dask autoscaler config (#604) Sep 20, 2024
@Alputer Alputer force-pushed the dask-autoscaler-config branch from ac5573d to f075666 Compare October 31, 2024 16:27
Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Oct 31, 2024
@Alputer Alputer force-pushed the dask-autoscaler-config branch from f075666 to 281a6fb Compare October 31, 2024 16:36
@Alputer Alputer changed the title feat(dask): dask autoscaler config (#604) feat(dask): make dask autoscaler configurable Oct 31, 2024
Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Nov 1, 2024
@Alputer Alputer force-pushed the dask-autoscaler-config branch from 281a6fb to 4921c62 Compare November 1, 2024 09:26
Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Nov 4, 2024
@Alputer Alputer force-pushed the dask-autoscaler-config branch from 4921c62 to 7b6a37e Compare November 4, 2024 10:09
@@ -290,6 +290,9 @@ def _parse_interactive_sessions_environments(env_var):
DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
"""Whether Dask is enabled in the cluster or not"""

DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
"""Whether dask autoscaler is enabled in the cluster or not"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cosmetics: s/dask/Dask/g in comments.

@@ -290,6 +290,9 @@ def _parse_interactive_sessions_environments(env_var):
DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
"""Whether Dask is enabled in the cluster or not"""

DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding "true" vs "True", how things work if people use uppercase or lowercase Helm values?

We may want to be ready to support two scenarios, so do some lower() gymnastics here, as we do elsewhere, for example:

os.getenv("K8S_USE_SECURITY_CONTEXT", "True").lower() == "true"

P.S. We could amend other occurrences of this boolean handling, see:

$ grep -Ei '"(true|false)' */*/config.py
reana-commons/reana_commons/config.py:    os.getenv("K8S_USE_SECURITY_CONTEXT", "True").lower() == "true"
reana-db/reana_db/config.py:    os.getenv("SQLALCHEMY_POOL_PRE_PING", "False").lower() == "true"
reana-db/reana_db/config.py:    os.getenv("REANA_PERIODIC_RESOURCE_QUOTA_UPDATE_POLICY", "false")
reana-job-controller/reana_job_controller/config.py:    strtobool(os.getenv("REANA_DB_CLOSE_POOL_CONNECTIONS", "false"))
reana-server/reana_server/config.py:DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
reana-server/reana_server/config.py:DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
reana-server/reana_server/config.py:    os.getenv("REANA_USER_EMAIL_CONFIRMATION", "true")
reana-workflow-controller/reana_workflow_controller/config.py:    {"name": "WDB_NO_BROWSER_AUTO_OPEN", "value": "True"},
reana-workflow-controller/reana_workflow_controller/config.py:    os.getenv("REANA_OPENSEARCH_ENABLED", "false").lower() == "true"
reana-workflow-controller/reana_workflow_controller/config.py:    os.getenv("REANA_OPENSEARCH_USE_SSL", "false").lower() == "true"
reana-workflow-controller/reana_workflow_controller/config.py:DASK_ENABLED = strtobool(os.getenv("DASK_ENABLED", "true"))
reana-workflow-controller/reana_workflow_controller/config.py:DASK_AUTOSCALER_ENABLED = strtobool(os.getenv("DASK_AUTOSCALER_ENABLED", "true"))
reana-workflow-engine-cwl/reana_workflow_engine_cwl/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-cwl/reana_workflow_engine_cwl/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))
reana-workflow-engine-serial/reana_workflow_engine_serial/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-serial/reana_workflow_engine_serial/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))
reana-workflow-engine-snakemake/reana_workflow_engine_snakemake/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-snakemake/reana_workflow_engine_snakemake/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))
reana-workflow-engine-yadage/reana_workflow_engine_yadage/config.py:MOUNT_CVMFS = os.getenv("REANA_MOUNT_CVMFS", "false")
reana-workflow-engine-yadage/reana_workflow_engine_yadage/config.py:WORKFLOW_KERBEROS = bool(strtobool(os.getenv("REANA_WORKFLOW_KERBEROS", "false")))

Alputer added a commit to Alputer/reana-workflow-controller that referenced this pull request Nov 11, 2024
@Alputer Alputer force-pushed the dask-autoscaler-config branch from 7b6a37e to e88d26f Compare November 11, 2024 09:16
@Alputer Alputer force-pushed the dask-autoscaler-config branch from e88d26f to db9ac05 Compare November 11, 2024 11:29
@Alputer Alputer changed the title feat(dask): make dask autoscaler configurable feat(config): make dask autoscaler configurable Nov 11, 2024
@tiborsimko tiborsimko merged commit db9ac05 into reanahub:master Nov 11, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

feat(dask): make dask autoscaler optional
2 participants