[fine_tuning] Add Cluster scale up for Ilab testing #595

kpouget · 2024-11-22T08:22:59Z

No description provided.

openshift-ci · 2024-11-22T08:24:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from kpouget. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kpouget · 2024-11-22T10:21:03Z

/test rhoai-light fine_tuning

topsail-bot · 2024-11-22T10:33:25Z

Jenkins Job #1694

🔴 Test of 'rhoai test test_ci' failed after 00 hours 05 minutes 11 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ilab

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 394, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 313, in _run_test_and_visualize
    prepare_finetuning.scale_down()
AttributeError: module 'prepare_finetuning' has no attribute 'scale_down'

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-22T10:51:38Z

Jenkins Job #1695

🔴 Test of 'rhoai test test_ci' failed after 00 hours 04 minutes 21 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: ilab

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 394, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 313, in _run_test_and_visualize
    prepare_finetuning.cluster_scale_down()
TypeError: cluster_scale_down() missing 1 required positional argument: 'to_zero'

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-22T11:17:37Z

Jenkins Job #1696

🔴 Test of 'rhoai test test_ci' failed after 00 hours 00 minutes 04 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: fms
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: fms

• Link to the Rebuild page.

Failure indicator:

/logs/artifacts/000_test_ci/FAILURE | Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 394, in test
    failed = _run_test_and_visualize()
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 264, in _run_test_and_visualize
    raise RuntimeError(msg)
RuntimeError: FMS or Quality or Ray or Ilab testing must be enabled. Found 0 enabled. Cannot proceed.

[Test ran on the internal Perflab CI]

topsail-bot · 2024-11-22T11:36:41Z

Jenkins Job #1697

🟢 Test of 'rhoai test test_ci' succeeded after 00 hours 04 minutes 39 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: fms
PR_POSITIONAL_ARG_0: fine_tuning-perf-ci
PR_POSITIONAL_ARG_1: fms

• Link to the Rebuild page.

[Test ran on the internal Perflab CI]

…an emphemeral PVC as output storage instead of the `emptyDir`, which may fill up the node's ephemeral storage

…ning

…S-specific default response template

…er_instructlab' preset

…preload command

… on error

kpouget · 2024-11-22T14:14:07Z

/test rhoai-light fine_tuning ilab gpu

topsail-bot · 2024-11-22T15:07:24Z

🔴 Test of 'rhoai test prepare_ci' failed after 00 hours 16 minutes 34 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test prepare_ci
PR_POSITIONAL_ARGS: ilab gpu
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab
PR_POSITIONAL_ARG_2: gpu

Failure indicator: Empty. (See run.log)

…ills_data.jsonl dataset

…ive 'unlimited' time for the job to complete

…uctlab cluster

…nnecessary ${}

…the source if it already exists

kpouget · 2024-11-23T21:25:16Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-23T21:31:49Z

🔴 Test of 'rhoai test prepare_ci' failed after 00 hours 00 minutes 06 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test prepare_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator:

/logs/artifacts/000__cluster__wait_fully_awake/FAILURE | [000__cluster__wait_fully_awake] ./run_toolbox.py cluster wait_fully_awake --> 2

kpouget · 2024-11-23T21:33:05Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-23T22:31:08Z

🔴 Test of 'rhoai test test_ci' failed after 00 hours 23 minutes 19 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator:

/logs/artifacts/004__ilab_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1} --> 2
/logs/artifacts/004__ilab_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/004__ilab_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 143, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

kpouget · 2024-11-24T07:31:06Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-24T08:45:08Z

🔴 Test of 'rhoai test test_ci' failed after 00 hours 31 minutes 35 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator:

/logs/artifacts/004__ilab_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1} --> 2
/logs/artifacts/004__ilab_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/004__ilab_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 143, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

kpouget · 2024-11-24T09:32:42Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-24T10:33:17Z

🔴 Test of 'rhoai test test_ci' failed after 00 hours 30 minutes 55 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator:

/logs/artifacts/004__ilab_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1} --> 2
/logs/artifacts/004__ilab_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/004__ilab_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 143, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

kpouget · 2024-11-24T16:07:22Z

/test rhoai-light fine_tuning

topsail-bot · 2024-11-24T16:40:22Z

🔴 Test of 'rhoai test prepare_ci' failed after 00 hours 13 minutes 56 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test prepare_ci
PR_POSITIONAL_ARGS: ''
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light

Failure indicator:

/logs/artifacts/002__prepare2/001__prepare_data/000__cluster__preload_image_fine-tuning-image/FAILURE | [000__cluster__preload_image_fine-tuning-image] ./run_toolbox.py from_config cluster preload_image --prefix=sutest --extra={} --> 2
/logs/artifacts/002__prepare2/001__prepare_data/002__cluster__preload_image_fine-tuning-image/FAILURE | [002__cluster__preload_image_fine-tuning-image] ./run_toolbox.py from_config cluster preload_image --prefix=sutest --extra={} --> 2
/logs/artifacts/002__prepare2/001__prepare_data/003__cluster__preload_image_fine-tuning-image/FAILURE | [003__cluster__preload_image_fine-tuning-image] ./run_toolbox.py from_config cluster preload_image --prefix=sutest --extra={} --> 2

kpouget · 2024-11-24T17:31:38Z

/test rhoai-light fine_tuning

topsail-bot · 2024-11-24T18:41:33Z

🟢 Test of 'rhoai test test_ci' succeeded after 00 hours 36 minutes 10 seconds. 🟢

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ''
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light

kpouget · 2024-11-24T19:12:50Z

/test rhoai-light fine_tuning ilab

kpouget · 2024-11-24T21:36:42Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-24T22:36:20Z

🔴 Test of 'rhoai test test_ci' failed after 00 hours 05 minutes 11 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator:

/logs/artifacts/004__ilab_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1} --> 2
/logs/artifacts/004__ilab_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/004__ilab_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 143, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

kpouget · 2024-11-25T09:02:54Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-25T09:26:21Z

🔴 Test of 'rhoai test prepare_ci' failed after 00 hours 00 minutes 03 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test prepare_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator: Empty. (See run.log)

openshift-ci · 2024-11-25T09:29:19Z

@kpouget: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/rhoai-light	`db4e8b1`	link	true	`/test rhoai-light`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

kpouget · 2024-11-25T09:30:21Z

/test rhoai-light fine_tuning ilab

topsail-bot · 2024-11-25T10:46:38Z

🔴 Test of 'rhoai test test_ci' failed after 00 hours 14 minutes 35 seconds. 🔴

• Link to the test results.

• Link to the reports index.

Test configuration:

# RHOAI: run fine_tuning test test_ci
PR_POSITIONAL_ARGS: ilab
PR_POSITIONAL_ARG_0: fine_tuning-rhoai-light
PR_POSITIONAL_ARG_1: ilab

Failure indicator:

/logs/artifacts/004__ilab_fine_tuning/000__fine_tuning__run_fine_tuning_job/FAILURE | [000__fine_tuning__run_fine_tuning_job] ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra={'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1} --> 2
/logs/artifacts/004__ilab_fine_tuning/FAILURE | CalledProcessError: Command 'set -o errexit;set -o pipefail;set -o nounset;set -o errtrace;ARTIFACT_DIR="/logs/artifacts/004__ilab_fine_tuning" ./run_toolbox.py from_config fine_tuning run_fine_tuning_job --extra="{'dataset_name': 'ilab_skills_data.jsonl', 'dataset_replication': 1, 'gpu': 1, 'model_name': 'granite-3b-code-instruct', 'name': 'ilab', 'pod_count': 1}"' returned non-zero exit status 2.
Traceback (most recent call last):
  File "/opt/topsail/src/projects/fine_tuning/testing/test_finetuning.py", line 143, in _run_test
    run.run_toolbox_from_config("fine_tuning", "run_fine_tuning_job",
  File "/opt/topsail/src/projects/core/library/run.py", line 49, in run_toolbox_from_config
    return run(f'{cmd_env} ./run_toolbox.py from_config {group} {command} {_dict_to_run_toolbox_args(kwargs)}', **run_kwargs)
  File "/opt/topsail/src/projects/core/library/run.py", line 105, in run
    proc = subprocess.run(command, **args)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run

[...]

kpouget · 2024-11-25T10:50:00Z

merging, I don't know why the g5.xlarge node keeps failing :/

  Warning  NodeNotReady            5m14s  node-controller          Node is not ready

kpouget force-pushed the instruct-scaleup branch from 40d469c to 0081588 Compare November 22, 2024 08:24

kpouget force-pushed the instruct-scaleup branch 2 times, most recently from 2e0df88 to d92d55a Compare November 22, 2024 10:20

kpouget force-pushed the instruct-scaleup branch 2 times, most recently from 999af36 to f7454ac Compare November 22, 2024 10:41

kpouget force-pushed the instruct-scaleup branch 2 times, most recently from 880a8ef to 7721432 Compare November 22, 2024 11:11

kpouget force-pushed the instruct-scaleup branch from 7721432 to c5e1b27 Compare November 22, 2024 11:19

kpouget added 7 commits November 22, 2024 14:04

[fine_tuning] toolbox: fine_tuning_run_fine_tuning_job: allow giving …

661e5af

…an emphemeral PVC as output storage instead of the `emptyDir`, which may fill up the node's ephemeral storage

[fine_tuning] testing: add cluster scale up in the Ilab-on-RHOAI trai…

2c05f8a

…ning

[fine_tuning] testing: explicitely set the response template for FMS

eb41474

[fine_tuning] toolbox: fine_tuning_run_fine_tuning_job: remove the FM…

9b0725f

…S-specific default response template

[fine_tuning] testing: config: force-undefine the taint in the 'clust…

9163385

…er_instructlab' preset

[fine_tuning] testing: command_args: correctly pass the image to the …

7f40db8

…preload command

[fine_tuning] testing: prepare_finetuning: preload_image do not crash…

689db93

… on error

kpouget force-pushed the instruct-scaleup branch from c5e1b27 to 4fffe0c Compare November 22, 2024 13:05

kpouget added 6 commits November 22, 2024 16:10

[fine_tuning] testing: config: include the ilab_large_10000samples_sk…

cef47a6

…ills_data.jsonl dataset

[fine_tuning] toolbox: fine_tuning_run_fine_tuning_job: tasks/main: g…

6a1f3ac

…ive 'unlimited' time for the job to complete

[fine_tuning] testing: config: use the ephemeral storage on the instr…

bfa91e4

…uctlab cluster

[fine_tuning] testing: config: update the 'ilab_scale' preset

3f5522a

[storage] toolbox: storage_download_to_pvc: files/entrypoint remove u…

df29f2b

…nnecessary ${}

[storage] toolbox: storage_download_to_pvc: files/entrypoint: delete …

9956745

…the source if it already exists

kpouget force-pushed the instruct-scaleup branch from 1c6feea to 50f14a3 Compare November 24, 2024 07:30

kpouget force-pushed the instruct-scaleup branch from 50f14a3 to 656218a Compare November 24, 2024 17:31

kpouget force-pushed the instruct-scaleup branch from 656218a to c483e5a Compare November 24, 2024 19:47

openshift-psap deleted a comment from topsail-bot bot Nov 24, 2024

kpouget force-pushed the instruct-scaleup branch from c483e5a to db4e8b1 Compare November 25, 2024 07:56

[fine_tuning] testing: config: give a GPU to the light profile

ddbf811

kpouget force-pushed the instruct-scaleup branch from db4e8b1 to ddbf811 Compare November 25, 2024 09:30

kpouget merged commit d22cc7b into openshift-psap:main Nov 25, 2024
6 of 8 checks passed

kpouget deleted the instruct-scaleup branch November 25, 2024 10:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fine_tuning] Add Cluster scale up for Ilab testing #595

[fine_tuning] Add Cluster scale up for Ilab testing #595

kpouget commented Nov 22, 2024

openshift-ci bot commented Nov 22, 2024

kpouget commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

kpouget commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

kpouget commented Nov 23, 2024

topsail-bot bot commented Nov 23, 2024

kpouget commented Nov 23, 2024

topsail-bot bot commented Nov 23, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 25, 2024

topsail-bot bot commented Nov 25, 2024

openshift-ci bot commented Nov 25, 2024

kpouget commented Nov 25, 2024

topsail-bot bot commented Nov 25, 2024

kpouget commented Nov 25, 2024

[fine_tuning] Add Cluster scale up for Ilab testing #595

[fine_tuning] Add Cluster scale up for Ilab testing #595

Conversation

kpouget commented Nov 22, 2024

openshift-ci bot commented Nov 22, 2024

kpouget commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

kpouget commented Nov 22, 2024

topsail-bot bot commented Nov 22, 2024

kpouget commented Nov 23, 2024

topsail-bot bot commented Nov 23, 2024

kpouget commented Nov 23, 2024

topsail-bot bot commented Nov 23, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 24, 2024

kpouget commented Nov 24, 2024

topsail-bot bot commented Nov 24, 2024

kpouget commented Nov 25, 2024

topsail-bot bot commented Nov 25, 2024

openshift-ci bot commented Nov 25, 2024

kpouget commented Nov 25, 2024

topsail-bot bot commented Nov 25, 2024

kpouget commented Nov 25, 2024