Unable to complete basic service call against restate-server on aarch64-apple-darwin

[pcholakov]

Using the aarch64-apple-darwin binary from https://github.com/restatedev/restate/actions/runs/7465082170 or locally from source built using Rust 1.75.0-aarch64-apple-darwin, I get strange behavior when running a basic stateful doGreet keyed handler against it.

The client gets a successful response from the ingress endpoint:

❯ http :8080/hello/greet key=pavel
HTTP/1.1 200 OK
access-control-allow-credentials: true
content-length: 30
content-type: application/json
date: Wed, 10 Jan 2024 09:19:36 GMT
vary: origin
vary: access-control-request-method
vary: access-control-request-headers

{
    "response": "Hello pavel :-)"
}

However the server seems stuck repeatedly trying to invoke the handler over and over, believing the invocation is not complete:

2024-01-10T09:19:36.564491Z INFO restate_ingress_grpc::handler
  Processing ingress request
  in restate_ingress_grpc::handler::ingress_invoke
    otel.name: "ingress_invoke greet"
    rpc.system: "grpc"
    rpc.service: hello
    rpc.method: greet
    client.socket.address: 127.0.0.1
    client.socket.port: 64729
2024-01-10T09:19:36.572689Z INFO restate_invoker_impl::invocation_task
  Executing invocation at deployment
    deployment.address: http://localhost:9080/
    path: /invoke/hello/greet
2024-01-10T09:19:36.592269Z WARN restate_invoker_impl
  Error when executing the invocation, retrying in 50ms.
    error: [RT0007] [Unknown] unknown. For more details, look at the docs with https://docs.restate.dev/references/errors#RT0007
    restate.error.code: RT0007
    restate.invocation.id: NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag
RT0007

A retry-able error was received from the service deployment while processing the invocation.

Suggestions:

• Check the service/deployment logs to get more info about the error cause, like the stacktrace.
• Look at the https://docs.restate.dev/services/sdk/error-handling for more info about error handling in services.

On the worker-side, the handler keeps getting the request replayed:

[restate] [2024-01-10T09:17:35.878Z] INFO: Listening on 9080...
[restate] [2024-01-10T09:17:58.983Z] INFO: Answering discovery request. Registering these services: ["hello"]
[restate] [2024-01-10T09:18:03.478Z] INFO: Answering discovery request. Registering these services: ["hello"]
[restate] [2024-01-10T09:19:36.585Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Invoking function.
[restate] [2024-01-10T09:19:36.591Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Function completed successfully.
[restate] [2024-01-10T09:19:36.648Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Resuming (replaying) function.
[restate] [2024-01-10T09:19:36.649Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Function completed successfully.
[restate] [2024-01-10T09:19:36.756Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Resuming (replaying) function.
[restate] [2024-01-10T09:19:36.757Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Function completed successfully.
[restate] [2024-01-10T09:19:36.967Z] DEBUG: [hello/greet] [NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag] : Resuming (replaying) function.
...

This is the invocation status:

  ❯ [2024-01-10 11:19:36.568 +02:00] NS-fNXAbfe0AYzyq4Z1fWS50UpqzdB3ag [hello @ pavel]::greet
     Status:      backing-off  (5 minutes, 29 seconds and 241 ms. Retried 40 time(s). Next retry in in 4
                  seconds and 19 ms))
     Deployment:  bG9jYWxob3N0OjkwODAv [required]
     Trace ID:    00000000000000000000000000000000
     Error:       [2024-01-10 11:24:59.833 +02:00]
                  [Unknown] unknown

[pcholakov]

The SDK used is:

❯ npm ls @restatedev/restate-sdk
@restatedev/[email protected] /Users/pavel/restate/examples/typescript/hello-world-standalone
└── @restatedev/[email protected]

Root cause likely is on the SDK-side due to breaking protocol change: restatedev/sdk-typescript#222

[tillrohrmann]

The problem has been solved on the SDK side via restatedev/sdk-typescript#223.