Running SpiceDB in a Nomad Job with gRPC Health Check

[theronic]

Is anyone running SpiceDB using Hashicorp's Nomad? I can get it running on my local machine using nomad agent -dev & nomad job run spicedb.nomad, but I'm having some trouble getting the gRPC health check to work. Here is my spicedb.nomad job file:

job "spicedb" {
  datacenters = ["dc1"]
  type        = "service"

  group "spicedb" {
    count = 1

    network {
      port "grpc" {
        to = 50051
      }
    }

    task "spicedb" {
      driver = "docker"

      config {
        image = "authzed/spicedb:latest"
        ports = ["grpc", "http"]

        args = [
          "serve",
          "--log-level=debug",
          "--grpc-preshared-key", "spicedb",
          "--datastore-engine", "memory"
        ]
      }

      resources {
        cpu    = 500 # MHz
        memory = 256 # MB
      }

#       service {
#         name = "spicedb"
#         port = "grpc"
#
#         tags = ["spicedb"]
#
#         check {
#           type     = "grpc"
#           port     = "grpc"
#           interval = "5s"
#           timeout  = "2s"
#           tls_skip_verify = true
#         }
#       }
    }
  }
}

I've commented out the gRPC health check because it won't come up otherwise. I then tried running nomad agent -dev-consul & nomad job run spicedb.nomad because it seems only the Consul integration supports gRPC health checks. However, then it does not come up at all (even with health check disabled).

Alternatively, I assume I'd need to install grpc-health-check in a derived Docker image to conduct a health check, or is there a gRPC "service class" I can point it to? (Nomad supports a grpc_service parameter on gRPC health checks even without the health check).

[vroldanbet]

@theronic as a workaround, if you enable the HTTP Gateway, you should get the REST HTTP endpoint at /healthz

Also the official SpiceDB container has grpc-health-check in it

[theronic]

What is the path of grpc-health-probe? When I run the following command, I get the following error:

$ docker run --rm -it authzed/spicedb:latest grpc_health_probe
{"level":"debug","message":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined"}
{"level":"info","time":"2024-07-05T12:47:19Z","GOMEMLIMIT":7404598886,"package":"github.com/KimMachineGun/automemlimit/memlimit","message":"GOMEMLIMIT is updated"}
{"level":"error","error":"unknown command \"grpc_health_probe\" for \"spicedb\"","message":"terminated with errors"}

According to the Dockerfile, grpc-health-probe is copied to /bin/grpc_health_probe, but I can't find it at that path either.

btw. previously you mentioned that Spice does not expose an HTTP health endpoint. Is it a new feature? Thx.

[vroldanbet]

You want to look into Dockerfile.release

https://github.com/authzed/spicedb/blob/main/Dockerfile.release#L14-L15

btw. previously you mentioned that Spice does not expose an HTTP health endpoint. Is it a new feature? Thx.

that was actually me not knowing that we wired the gRPC health endpoint via the HTTP Gateway. So my bad, I'll correct that answer

[theronic]

When I try to run the grpc_health_probe the path in Dockerfile.release in the latest SpiceDB image, it says "unknown command":

docker run --rm -it authzed/spicedb:latest /usr/local/bin/grpc_health_probe
{"level":"debug","message":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined"}
{"level":"info","time":"2024-07-05T13:06:10Z","GOMEMLIMIT":7404598886,"package":"github.com/KimMachineGun/automemlimit/memlimit","message":"GOMEMLIMIT is updated"}
{"level":"error","error":"unknown command \"/usr/local/bin/grpc_health_probe\" for \"spicedb\"","message":"terminated with errors"}

[vroldanbet]

That's not how you invoke a command in a docker container that has an entrypoint, so you have to override it:
Both these commands work:

docker run --rm -it --entrypoint=grpc_health_probe authzed/spicedb:latest --help
docker run --rm -it --entrypoint=/usr/local/bin/grpc_health_probe authzed/spicedb:latest --help

[jzelinskie]

Once you get this working, would you be interested in sharing an example in our examples repository?