Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there any way I can get workflow status #1071

Open
mxnoob opened this issue Nov 25, 2024 · 13 comments
Open

Is there any way I can get workflow status #1071

mxnoob opened this issue Nov 25, 2024 · 13 comments

Comments

@mxnoob
Copy link

mxnoob commented Nov 25, 2024

I tried to get with await hatchet.rest.aio.workflow_get(workflow_id)

Error > Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 735, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 73, in app response = await f(request) File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app raw_response = await run_endpoint_function( File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function return await dependant.call(**values) File "/app/router.py", line 28, in test_back_task res = await hatchet.rest.aio.workflow_get(workflow_id) File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest_client.py", line 117, in workflow_get return await self.workflow_api.workflow_get( File "/usr/local/lib/python3.10/site-packages/pydantic/_internal/_validate_call.py", line 33, in wrapper_function return await wrapper(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api/workflow_api.py", line 702, in workflow_get response_data = await self.api_client.call_api( File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 262, in call_api response_data = await self.rest_client.request( File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/rest.py", line 180, in request r = await pool_manager.request(**args) File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 622, in _request raise err_exc_cls(url) aiohttp.client_exceptions.InvalidUrlClientError: /api/v1/workflows/0d19625e-7c4c-4358-9304-1f1498df3dda
@abelanger5
Copy link
Contributor

Hey @mxnoob, I think you're looking for workflow_run_get. For example:

run = hatchet.admin.run_workflow("MyWorkflow", {})
await run.result()

workflow_run = await hatchet.rest.aio.workflow_run_get(run.workflow_run_id)

# can now access workflow_run.status

@mxnoob
Copy link
Author

mxnoob commented Nov 26, 2024

Hi @abelanger5, thanks for the reply, I have rewritten the code and am getting the error again. I am trying to get data from another docker compose, could this be the cause?

@router.get("/workflow/{workflow_id}")
async def test_back_task(workflow_id: str):
    res = await hatchet.rest.aio.workflow_run_get(workflow_id)
    return {"workflow_status": res.status}
Error > Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__ await self.middleware_stack(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 735, in app await route.handle(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 73, in app response = await f(request) File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app raw_response = await run_endpoint_function( File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function return await dependant.call(**values) File "/app/router.py", line 31, in test_back_task res = await hatchet.rest.aio.workflow_run_get(workflow_id) File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest_client.py", line 159, in workflow_run_get return await self.workflow_api.workflow_run_get( File "/usr/local/lib/python3.10/site-packages/pydantic/_internal/_validate_call.py", line 33, in wrapper_function return await wrapper(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api/workflow_api.py", line 1842, in workflow_run_get response_data = await self.api_client.call_api( File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 262, in call_api response_data = await self.rest_client.request( File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/rest.py", line 180, in request r = await pool_manager.request(**args) File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 622, in _request raise err_exc_cls(url) aiohttp.client_exceptions.InvalidUrlClientError: /api/v1/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/workflow-runs/b0a0d8b0-5338-40f2-a0d9-471c8db22ee7

@abelanger5
Copy link
Contributor

Hey @mxnoob how are you instantiating the Hatchet client? This looks like potentially a configuration issue or a bug on our end -- it may be that when you overwrite the hostPort value in the config, it's unsetting the URL for the API server. But checking how you're initializing the Hatchet client would be helpful.

@mxnoob
Copy link
Author

mxnoob commented Nov 26, 2024

.env HATCHET_CLIENT_TOKEN=token

HATCHET_CLIENT_TLS_STRATEGY=none
HATCHET_CLIENT_HOST_PORT=host.docker.internal:7077

I got docker-compose.yml from https://docs.hatchet.run/self-hosting/docker-compose

What settings did I miss?

@abelanger5
Copy link
Contributor

abelanger5 commented Nov 26, 2024

Got it, I believe this is a bug on our end -- when you use the HATCHET_CLIENT_HOST_PORT variable, we are no longer setting the API server URL. As a quick fix, could you do the following:

  1. In the setup-config service, set the following env vars:

    SERVER_GRPC_BROADCAST_ADDRESS=host.docker.internal:7077
    SERVER_URL=http://host.docker.internal:8080
    

    Then restart the docker compose services.

  2. Regenerate an API token, and use the new token.

  3. Remove HATCHET_CLIENT_HOST_PORT from the .env file.

Let me know if that works while we investigate this.

@mxnoob
Copy link
Author

mxnoob commented Nov 27, 2024

I did everything according to the points, unfortunately to no avail
In setup-config I set SERVER_GRPC_BROADCAST_ADDRESS and GRPC_BROADCAST_ADDRESS

error INFO: 172.23.0.1:40706 - "GET /workflow/ef1d4bf8-be2c-4ea5-a786-28604ff0b5f9 HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1109, in _wrap_create_connection sock = await aiohappyeyeballs.start_connection( File "/usr/local/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 123, in start_connection raise OSError(first_errno, msg) ConnectionRefusedError: [Errno 111] Multiple exceptions: [Errno 111] Connect call failed ('::1', 8080, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 8080)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
File "/app/router.py", line 29, in test_back_task
res = await hatchet.rest.aio.workflow_run_get(workflow_id)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest_client.py", line 159, in workflow_run_get
return await self.workflow_api.workflow_run_get(
File "/usr/local/lib/python3.10/site-packages/pydantic/_internal/_validate_call.py", line 33, in wrapper_function
return await wrapper(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api/workflow_api.py", line 1842, in workflow_run_get
response_data = await self.api_client.call_api(
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 262, in call_api
response_data = await self.rest_client.request(
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/rest.py", line 180, in request
r = await pool_manager.request(**args)
File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 701, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 544, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1050, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1394, in _create_direct_connection
raise last_exc
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1363, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 1124, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host localhost:8080 ssl:default [Multiple exceptions: [Errno 111] Connect call failed ('::1', 8080, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 8080)]

@abelanger5
Copy link
Contributor

@mxnoob could you try one more step -- setting SERVER_URL in the setup-config step to the value http://host.docker.internal:8080, restart and regenerate the API token again.

And you are correct, it should be SERVER_GRPC_BROADCAST_ADDRESS, nice catch.

@mxnoob
Copy link
Author

mxnoob commented Nov 27, 2024

@abelanger5 thanks! I got a new error, now pydantic

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
File "/app/router.py", line 29, in test_back_task
res = await hatchet.rest.aio.workflow_run_get(workflow_id)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest_client.py", line 159, in workflow_run_get
return await self.workflow_api.workflow_run_get(
File "/usr/local/lib/python3.10/site-packages/pydantic/_internal/_validate_call.py", line 33, in wrapper_function
return await wrapper(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api/workflow_api.py", line 1846, in workflow_run_get
return self.api_client.response_deserialize(
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 316, in response_deserialize
return_data = self.deserialize(
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 415, in deserialize
return self.__deserialize(data, response_type)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 460, in __deserialize
return self.__deserialize_model(data, klass)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/api_client.py", line 746, in __deserialize_model
return klass.from_dict(data)
File "/usr/local/lib/python3.10/site-packages/hatchet_sdk/clients/rest/models/workflow_run.py", line 144, in from_dict
return cls.model_validate(obj)
File "/usr/local/lib/python3.10/site-packages/pydantic/main.py", line 627, in model_validate
return cls.pydantic_validator.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for WorkflowRun
Input should be a valid dictionary or instance of WorkflowRun [type=model_type, input_value='', input_type=str]
For further information visit https://errors.pydantic.dev/2.10/v/model_type

@abelanger5
Copy link
Contributor

@mxnoob just to double check, which version of the Python SDK are you using?

@mxnoob
Copy link
Author

mxnoob commented Nov 28, 2024

@mxnoob just to double check, which version of the Python SDK are you using?

hatchet-sdk 0.38.3

@hatchet-temporary
Copy link
Contributor

hey @mxnoob - what's the code you're running that triggers this error? is it still what you put in this earlier comment? If yes, I'll try to reproduce this!

@mxnoob
Copy link
Author

mxnoob commented Dec 6, 2024

Hi @hatchet-temporary, that's right, the error occurs when this code is called

@hatchet-temporary
Copy link
Contributor

Hey @mxnoob - looking into this a bit more! One thing I'm noticing is in the code, it looks like you're passing a workflow_id to workflow_run_get, which takes a workflow run id as its parameter. I just wanted to make sure that the value you're passing is, in fact, a run id! (One note is I think this will cause a 500 if you pass an id that doesn't exist, so I think you're likely all good on this front 👍)

To me, this looks like it's likely a bug in an old version of the SDK. The TL;DR of what's going on is that somehow, an empty string is being passed to a Pydantic model_validate, which is then failing. In reality, model_validate should never be called on a string, which I think is likely a bug on our side.

Would it be possible to upgrade the version of the SDK on your end and see if the issue persists? If it does, it's pretty likely it'd be a straightforward fix I could push!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants