Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for submitting a prebuilt image #8802

Merged
merged 38 commits into from
May 27, 2024
Merged

Add support for submitting a prebuilt image #8802

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
0aa797d
Add support for submitting a prebuilt image
kiendang May 13, 2024
870f591
Remove the nbqa-black pre-commit hook
kiendang May 13, 2024
7f86a1e
Merge branch 'dev' into prebuilt-image-worker
shubham3121 May 14, 2024
6c48218
Rename get_by_docker_config to get_by_worker_config
kiendang May 14, 2024
85935b3
Add unit test for services.worker_image.get_by_config
kiendang May 14, 2024
81ae3ea
Rename
kiendang May 14, 2024
56a8ad6
Use get_by_image to retrieve the image in test
kiendang May 14, 2024
ce21987
Rename arg docker_config to worker_config
kiendang May 14, 2024
df646ea
Add integration tests for prebuilt images
kiendang May 16, 2024
6c13ba2
Merge branch 'dev' into prebuilt-image-worker
kiendang May 16, 2024
16bcf91
Fix worker pool integration tests
kiendang May 16, 2024
87861a3
Fix worker pool integration tests
kiendang May 16, 2024
749bc1e
Use a different prebuilt image for each tag
kiendang May 16, 2024
f2c4d5c
Add tag for images from PrebuiltWorkerConfig
kiendang May 16, 2024
9f24bab
No longer try to push prebuilt image to local registry
kiendang May 16, 2024
b0b658a
Use a different worker pool name for each test
kiendang May 16, 2024
74038fe
Get error while launching worker pool
kiendang May 16, 2024
b9554e5
debug
kiendang May 16, 2024
6d6a6a6
Increase k8s pool creation timeout
kiendang May 16, 2024
5636bf1
Test using a smaller image
kiendang May 16, 2024
ece69c1
Remove unused variables
kiendang May 21, 2024
9e84ea9
Remove unnecessary check
kiendang May 21, 2024
f9a35e1
Use different prebuilt images for tests
kiendang May 21, 2024
c67932f
Merge branch 'dev' into prebuilt-image-worker
kiendang May 22, 2024
dd138f5
Avoid username collision in tests
kiendang May 22, 2024
1286940
Get the correct job by id instead of the last job
kiendang May 22, 2024
7c535c7
Only require tag for DockerWorkerConfig pool request
kiendang May 22, 2024
5ecf05b
Update CreateCustomImageChange protocol version
kiendang May 22, 2024
135fbb2
Update tests
kiendang May 22, 2024
54c3aed
Merge branch 'dev' into prebuilt-image-worker
shubham3121 May 24, 2024
3618ad4
Return an error in case of invalid container name
kiendang May 24, 2024
39ac64a
Merge branch 'dev' into prebuilt-image-worker
shubham3121 May 27, 2024
3cb2a0a
Rename submit_container_image to just submit
kiendang May 27, 2024
6a7022a
Merge branch 'dev' into prebuilt-image-worker
kiendang May 27, 2024
a8d86ed
Use correct account client
kiendang May 27, 2024
c453ad5
Merge branch 'dev' into prebuilt-image-worker
kiendang May 27, 2024
9865e20
Merge branch 'dev' into prebuilt-image-worker
kiendang May 27, 2024
19d627f
Fix user permission in test
kiendang May 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion notebooks/admin/Custom API + Custom Worker.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@
"metadata": {},
"outputs": [],
"source": [
"submit_result = domain_client.api.services.worker_image.submit_container_image(\n",
"submit_result = domain_client.api.services.worker_image.submit(\n",
" worker_config=docker_config\n",
")\n",
"submit_result"
Expand Down
4 changes: 2 additions & 2 deletions notebooks/api/0.8/10-container-images.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@
"metadata": {},
"outputs": [],
"source": [
"submit_result = domain_client.api.services.worker_image.submit_container_image(\n",
"submit_result = domain_client.api.services.worker_image.submit(\n",
" worker_config=docker_config\n",
")"
]
Expand Down Expand Up @@ -1095,7 +1095,7 @@
"metadata": {},
"outputs": [],
"source": [
"submit_result = domain_client.api.services.worker_image.submit_container_image(\n",
"submit_result = domain_client.api.services.worker_image.submit(\n",
" worker_config=docker_config_2\n",
")\n",
"submit_result"
Expand Down
4 changes: 2 additions & 2 deletions notebooks/api/0.8/11-container-images-k8s.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@
"metadata": {},
"outputs": [],
"source": [
"submit_result = domain_client.api.services.worker_image.submit_container_image(\n",
"submit_result = domain_client.api.services.worker_image.submit(\n",
" worker_config=docker_config\n",
")\n",
"submit_result"
Expand Down Expand Up @@ -935,7 +935,7 @@
"outputs": [],
"source": [
"submit_result = None\n",
"submit_result = domain_client.api.services.worker_image.submit_container_image(\n",
"submit_result = domain_client.api.services.worker_image.submit(\n",
" worker_config=docker_config_opendp\n",
")\n",
"submit_result"
Expand Down
2 changes: 1 addition & 1 deletion packages/syft/src/syft/service/request/request.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ def _run(
worker_image_service = context.node.get_service("SyftWorkerImageService")

service_context = context.to_service_ctx()
result = worker_image_service.submit_container_image(
result = worker_image_service.submit(
service_context, worker_config=self.config
)

Expand Down
6 changes: 3 additions & 3 deletions packages/syft/src/syft/service/worker/worker_image_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ def __init__(self, store: DocumentStore) -> None:
self.stash = SyftWorkerImageStash(store=store)

@service_method(
path="worker_image.submit_container_image",
name="submit_container_image",
path="worker_image.submit",
name="submit",
roles=DATA_OWNER_ROLE_LEVEL,
shubham3121 marked this conversation as resolved.
Show resolved Hide resolved
)
def submit_container_image(
def submit(
self, context: AuthedServiceContext, worker_config: WorkerConfig
) -> SyftSuccess | SyftError:
image_identifier: SyftWorkerImageIdentifier | None = None
Expand Down
6 changes: 2 additions & 4 deletions packages/syft/tests/syft/worker_pool/worker_pool_service_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ def test_create_pool_request_accept(
assert root_client.credentials != ds_client.credentials

# the DO submits the docker config to build an image
submit_result = root_client.api.services.worker_image.submit_container_image(
submit_result = root_client.api.services.worker_image.submit(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏾

worker_config=worker_config
)
assert isinstance(submit_result, SyftSuccess)
Expand Down Expand Up @@ -161,9 +161,7 @@ def test_get_by_worker_config(
) -> None:
root_client = worker.root_client
for config in WORKER_CONFIGS:
root_client.api.services.worker_image.submit_container_image(
worker_config=config
)
root_client.api.services.worker_image.submit(worker_config=config)

worker_image = root_client.api.services.worker_image.get_by_config(worker_config)
assert worker_image.config == worker_config
2 changes: 1 addition & 1 deletion packages/syft/tests/syft/worker_pool/worker_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def test_syft_worker(worker: Worker):
"""
root_client = worker.root_client
docker_config = get_docker_config()
submit_result = root_client.api.services.worker_image.submit_container_image(
submit_result = root_client.api.services.worker_image.submit(
worker_config=docker_config
)
assert isinstance(submit_result, SyftSuccess)
Expand Down
4 changes: 2 additions & 2 deletions tests/integration/container_workload/pool_image_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def test_image_build(domain_1_port: int, external_registry_uid: UID) -> None:

docker_config, docker_tag = make_docker_config_test_case("recordlinkage")

submit_result = domain_client.api.services.worker_image.submit_container_image(
submit_result = domain_client.api.services.worker_image.submit(
worker_config=docker_config
)
assert isinstance(submit_result, SyftSuccess)
Expand Down Expand Up @@ -119,7 +119,7 @@ def test_pool_launch(
if prebuilt
else make_docker_config_test_case("opendp")
)
submit_result = domain_client.api.services.worker_image.submit_container_image(
submit_result = domain_client.api.services.worker_image.submit(
worker_config=worker_config
)
assert isinstance(submit_result, SyftSuccess)
Expand Down
Loading