-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: create task #207
base: template
Are you sure you want to change the base?
feat: create task #207
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -3,9 +3,14 @@ | |||||||||||||||||||
import logging | ||||||||||||||||||||
|
||||||||||||||||||||
# from connexion import request # type: ignore | ||||||||||||||||||||
from typing import Any | ||||||||||||||||||||
|
||||||||||||||||||||
from foca.utils.logging import log_traffic # type: ignore | ||||||||||||||||||||
|
||||||||||||||||||||
from tesk.api.ga4gh.tes.models import TesTask | ||||||||||||||||||||
from tesk.api.ga4gh.tes.service_info.service_info import ServiceInfo | ||||||||||||||||||||
from tesk.api.ga4gh.tes.task.create_task import CreateTesTask | ||||||||||||||||||||
from tesk.exceptions import InternalServerError | ||||||||||||||||||||
|
||||||||||||||||||||
# Get logger instance | ||||||||||||||||||||
logger = logging.getLogger(__name__) | ||||||||||||||||||||
|
@@ -26,14 +31,19 @@ def CancelTask(id, *args, **kwargs) -> dict: # type: ignore | |||||||||||||||||||
|
||||||||||||||||||||
# POST /tasks | ||||||||||||||||||||
@log_traffic | ||||||||||||||||||||
def CreateTask(*args, **kwargs) -> dict: # type: ignore | ||||||||||||||||||||
def CreateTask(**kwargs) -> dict: # type: ignore | ||||||||||||||||||||
"""Create task. | ||||||||||||||||||||
|
||||||||||||||||||||
Args: | ||||||||||||||||||||
*args: Variable length argument list. | ||||||||||||||||||||
**kwargs: Arbitrary keyword arguments. | ||||||||||||||||||||
""" | ||||||||||||||||||||
pass | ||||||||||||||||||||
try: | ||||||||||||||||||||
request_body: Any = kwargs.get("body") | ||||||||||||||||||||
tes_task = TesTask(**request_body) | ||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Shouldn't we catch a |
||||||||||||||||||||
response = CreateTesTask(tes_task).response() | ||||||||||||||||||||
return response | ||||||||||||||||||||
Comment on lines
+43
to
+44
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Might as well return it - no point of creating a variable only to return it immediately. |
||||||||||||||||||||
except Exception as e: | ||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. suggestion: Implement more granular error handling While catching and returning an internal server error is better than uncaught exceptions, consider implementing more granular error handling. This could provide better feedback to API consumers about what went wrong.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is a very good point - with the current code we couldn't get any other error than In any case, FOCA takes care of this gracefully, so I think we don't actually need more granual error handling, but no additional error handling at all! What FOCA does is to handle the errors according to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thank you for the detailed explanation, @uniqueg! It's great to hear that FOCA is handling error responses according to the Given this setup, it makes sense to rely on FOCA's built-in mechanisms for transforming exceptions into appropriate JSON responses. This not only reduces redundancy in error handling but also ensures consistency across the application. If FOCA is already providing the necessary granularity and coverage for error responses, then maintaining the current structure with a catch-all If you have any further questions or need additional clarification, feel free to ask! |
||||||||||||||||||||
raise InternalServerError from e | ||||||||||||||||||||
|
||||||||||||||||||||
|
||||||||||||||||||||
# GET /tasks/service-info | ||||||||||||||||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
"""Task API controller logic.""" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
"""TESK API module for creating a task.""" | ||
|
||
import logging | ||
|
||
from tesk.api.ga4gh.tes.models import TesCreateTaskResponse, TesResources, TesTask | ||
from tesk.api.ga4gh.tes.task.task_request import TesTaskRequest | ||
from tesk.exceptions import KubernetesError | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
class CreateTesTask(TesTaskRequest): | ||
"""Create TES task.""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Attributes? See below. |
||
|
||
def __init__( | ||
self, | ||
task: TesTask, | ||
): | ||
"""Initialize the CreateTask class. | ||
|
||
Args: | ||
task: TES task to create. | ||
""" | ||
super().__init__() | ||
self.task = task | ||
|
||
def handle_request(self) -> TesCreateTaskResponse: | ||
"""Create TES task.""" | ||
attempts_no = 0 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add |
||
while ( | ||
attempts_no < self.tesk_k8s_constants.job_constants.JOB_CREATE_ATTEMPTS_NO | ||
): | ||
try: | ||
attempts_no += 1 | ||
jemaltahir marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider logging (in "Creating Kubernetes job (attempt 3/8)" |
||
resources = self.task.resources | ||
minimum_ram_gb = self.kubernetes_client_wrapper.minimum_ram_gb() | ||
|
||
if not self.task.resources: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Careful with implicit checks against falsey values. Are you sure you are okay with any falsey value for |
||
self.task.resources = TesResources(cpu_cores=int(minimum_ram_gb)) | ||
JaeAeich marked this conversation as resolved.
Show resolved
Hide resolved
|
||
if resources and resources.ram_gb and resources.ram_gb < minimum_ram_gb: | ||
self.task.resources.ram_gb = minimum_ram_gb | ||
Comment on lines
+35
to
+41
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I find this confusing to read. The focus seems to be jumping back and forth and I don't know why you defined How about sth like this: minimum_ram_gb = self.kubernetes_client_wrapper.minimum_ram_gb()
if self.task.resources is None:
self.task.resources = TesResources(cpu_cores=1, ram_gb=minimum_ram_gb)
elif (
self.task.resources.ram_gb is None or
self.task.resources.ram_gb < minimum_ram_gb
):
self.task.resources.ram_gb = minimum_ram_gb You could also consider adding a method for setting resources (or a higher level method |
||
|
||
taskmaster_job = self.tes_kubernetes_converter.from_tes_task_to_k8s_job( | ||
self.task, | ||
) | ||
taskmaster_config_map = ( | ||
self.tes_kubernetes_converter.from_tes_task_to_k8s_config_map( | ||
self.task, | ||
taskmaster_job, | ||
) | ||
) | ||
|
||
_ = self.kubernetes_client_wrapper.create_config_map( | ||
taskmaster_config_map | ||
) | ||
created_job = self.kubernetes_client_wrapper.create_job(taskmaster_job) | ||
|
||
assert created_job.metadata is not None | ||
assert created_job.metadata.name is not None | ||
Comment on lines
+58
to
+59
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider raising some sort of |
||
|
||
return TesCreateTaskResponse(id=created_job.metadata.name) | ||
|
||
except KubernetesError as e: | ||
if ( | ||
not e.is_object_name_duplicated() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I find this much harder to read/parse than: if (
e.status != HTTPStatus.CONFLICT
or ...
) Why the function and the use of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also, if your client wrapper already checked for except KubernetesConflict:
pass All the other errors you can just let bubble up. I'm pretty sure FOCA will handle these gracefully (as mentioned below). I think this would be considerably cleaner - and more informative. |
||
or attempts_no | ||
>= self.tesk_k8s_constants.job_constants.JOB_CREATE_ATTEMPTS_NO | ||
): | ||
raise e | ||
|
||
except Exception as exc: | ||
logging.error("ERROR: In createTask", exc_info=True) | ||
raise exc | ||
Comment on lines
+71
to
+73
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Isn't this handled automatically by FOCA? |
||
|
||
return TesCreateTaskResponse(id="") # To silence mypy, should never be reached | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I suppose this wouldn't be necessary if you defined the loop with |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it a base class for a TESK request (any request, including |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
"""Base class for tesk request.""" | ||
|
||
import json | ||
import logging | ||
from abc import ABC, abstractmethod | ||
|
||
from pydantic import BaseModel | ||
|
||
from tesk.k8s.constants import tesk_k8s_constants | ||
from tesk.k8s.converter.converter import TesKubernetesConverter | ||
from tesk.k8s.wrapper import KubernetesClientWrapper | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
class TesTaskRequest(ABC): | ||
"""Base class for tesk request ecapsulating common methods and members.""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please define "Args" and "Attributes" sections, either both in the class docstring or in the constructor. Or, even better, the "Args" in the constructor and the "Attributes" in the class. Same goes elsewhere, i.e., when you deal with classes, make sure that both of "Args" and "Attributes" sections are present (even if they are identical). |
||
|
||
def __init__(self): | ||
"""Initialise base class for tesk request.""" | ||
self.kubernetes_client_wrapper = KubernetesClientWrapper() | ||
self.tes_kubernetes_converter = TesKubernetesConverter() | ||
self.tesk_k8s_constants = tesk_k8s_constants | ||
|
||
@abstractmethod | ||
def handle_request(self) -> BaseModel: | ||
"""Business logic for the request.""" | ||
pass | ||
|
||
def response(self) -> dict: | ||
"""Get response for the request.""" | ||
response: BaseModel = self.handle_request() | ||
try: | ||
res: dict = json.loads(json.dumps(response)) | ||
return res | ||
except (TypeError, ValueError) as e: | ||
logger.info(e) | ||
Comment on lines
+33
to
+37
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's the significance of this? Why not going straight for the Pydantic way of marshalling the model? |
||
return response.dict() |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,7 @@ | ||
"""App exceptions.""" | ||
|
||
from http import HTTPStatus | ||
|
||
from connexion.exceptions import ( | ||
BadRequestProblem, | ||
ExtraParameterProblem, | ||
|
@@ -26,6 +28,10 @@ class ConfigInvalidError(ValueError): | |
class KubernetesError(ApiException): | ||
"""Kubernetes error.""" | ||
Comment on lines
28
to
29
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider adding |
||
|
||
def is_object_name_duplicated(self) -> bool: | ||
"""Check if object name is duplicated.""" | ||
return self.status == HTTPStatus.CONFLICT | ||
|
||
|
||
# exceptions raised in app context | ||
exceptions = { | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
"""Module for converting Kubernetes objects to Task objects.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove?