Forwarding from routing receiver to ingesting receiver

[sourcehawk]

I am attempting to set up a receiver in routing only mode on an "on premise" server to then forward remote writes made to it, to a receiver in the cloud which is running in ingestor mode. However, I am unable to get the traffic forwarded from the routing receiver to the ingesting receiver properly. Visualization of the setup:

This is my routing receiver configuration in docker compose:

version: '3.5'

services:
  thanos_receiver:
    image: quay.io/thanos/thanos:v0.34.1
    container_name: thanos-receiver
    env_file:
      - .env
    command: >
      receive
      --receive.hashrings='[{ "endpoints" : [ "$FORWARD_ENDPOINT" ] }]'
      --receive.local-endpoint=""
      --remote-write.address="$RECEIVE_ENDPOINT"
      --label=receive_instance='"edge-server"'
    network_mode: host
    pid: host
    restart: unless-stopped

.env file:

TENANT_ID="staging"
RECEIVE_ENDPOINT="0.0.0.0:10908"
FORWARD_ENDPOINT="thanos-remote-grpc.example.com:443"

The routing receiver logs indicate that it is correctly attempting to forward the request to the ingesting receiver, but fails for reasons that are beyond my understanding:

thanos-receiver    | ts=2024-04-06T02:54:30.457724697Z caller=receive.go:136 level=info component=receive mode=RouterOnly msg="running receive"
...
thanos-receiver    | ts=2024-04-06T03:05:34.510273282Z caller=handler.go:600 level=error component=receive component=receive-handler tenant=default-tenant err="forwarding request to endpoint thanos-remote-grpc.example.com:443: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"" msg="internal server error"

Similar logs coming from vm agent which I am using to push logs to the routing receiver via the remote write endpoint:

vm-agent         | 2024-04-06T03:05:34.523Z	error	VictoriaMetrics/app/vmagent/remotewrite/client.go:466	unexpected status code received after sending a block with size 16583 bytes to "1:secret-url" during retry #1: 500; response body="forwarding request to endpoint thanos-remote-grpc.doman.com:443: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"\n"; re-sending the block in 2.075 seconds

I deployed the receive ingestor in eks with the bitnami thanos helm chart without making any changes to it. I just set receive.enabled to true and added an extraFlag --tsdb.out-of-order.time-window=24h. Here's the resource description of the deployed pod:

kubectl describe pod monitoring-thanos-receive-0 -n monitoring
Name:             monitoring-thanos-receive-0
Namespace:        monitoring
Priority:         0
Service Account:  monitoring-thanos-receive
Node:             ip-<>.eu-west-1.compute.internal/<>
Start Time:       Fri, 05 Apr 2024 20:18:11 -0500
Labels:           app.kubernetes.io/component=receive
                  app.kubernetes.io/instance=monitoring
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=thanos
                  app.kubernetes.io/version=0.34.1
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=monitoring-thanos-receive-<>
                  helm.sh/chart=thanos-14.0.2
                  statefulset.kubernetes.io/pod-name=monitoring-thanos-receive-0
Annotations:      checksum/receive-configuration: <>
Status:           Running
IP:               <>
IPs:
  IP:           <>
Controlled By:  StatefulSet/monitoring-thanos-receive
Containers:
  receive:
    Container ID:    containerd://<>
    Image:           docker.io/bitnami/thanos:0.34.1-debian-12-r1
    Image ID:        docker.io/bitnami/thanos@sha256:<>
    Ports:           10901/TCP, 10902/TCP, 19291/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      receive
      --log.level=debug
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --remote-write.address=0.0.0.0:19291
      --objstore.config=$(OBJSTORE_CONFIG)
      --tsdb.path=/var/thanos/receive
      --label=replica="$(NAME)"
      --label=receive="true"
      --tsdb.retention=15d
      --receive.local-endpoint=127.0.0.1:10901
      --receive.hashrings-file=/var/lib/thanos-receive/hashrings.json
      --receive.replication-factor=1
      --tsdb.out-of-order.time-window=24h
    State:          Running
      Started:      Fri, 05 Apr 2024 20:18:20 -0500
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/-/healthy delay=30s timeout=30s period=10s #success=1 #failure=6
    Readiness:      http-get http://:http/-/ready delay=30s timeout=30s period=10s #success=1 #failure=6
    Environment:
      NAME:             monitoring-thanos-receive-0 (v1:metadata.name)
      NAMESPACE:        monitoring (v1:metadata.namespace)
      OBJSTORE_CONFIG:  <set to the key 'objstore.yml' in secret '<>'>  Optional: false
    Mounts:
      /var/lib/thanos-receive from hashring-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qgw7m (ro)
      /var/thanos/receive from data (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-monitoring-thanos-receive-0
    ReadOnly:   false
  hashring-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      monitoring-thanos-receive
    Optional:  false
  kube-api-access-qgw7m:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

I exposed the 10901 (grpc) port via a traefik IngressRoute resource and a dummy ingess for external dns to create the record. Here's the IngressRoute I deployed to get access to the grpc endpoint externally:

Name:         thanos-receive-remote-grpc
Namespace:    monitoring
Labels:       app.kubernetes.io/instance=monitoring
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=monitoring
              app.kubernetes.io/version=1.0.0
              helm.sh/chart=monitoring-3.5.11
Annotations:  meta.helm.sh/release-name: monitoring
              meta.helm.sh/release-namespace: monitoring
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2024-04-06T01:18:10Z
  Generation:          3
  Resource Version:    23821693
  UID:                 343e3af6-1f28-4faa-af65-85dcc4245110
Spec:
  Entry Points:
    websecure
  Routes:
    Kind:      Rule
    Match:     Host(`thanos-remote-grpc.example.com`)
    Priority:  11
    Services:
      Name:              monitoring-thanos-receive
      Namespace:         monitoring
      Pass Host Header:  true
      Port:              10901
      Scheme:            h2c
  Tls:
    Secret Name:  letsencrypt
Events:           <none>

and the ingress:

ame:             thanos-receive-remote-grpc-dummy
Labels:           app.kubernetes.io/managed-by=Helm
Namespace:        monitoring
Address:          examplealb.eu-west-1.elb.amazonaws.com
Ingress Class:    traefik
Default backend:  <default>
TLS:
  letsencrypt terminates example.com,*.example.com
Rules:
  Host                                      Path  Backends
  ----                                      ----  --------
  thanos-remote-grpc.example.com  
                                            /   monitoring-thanos-receive:10902 (<>:10902)
Annotations:                                cert-manager.io/cluster-issuer: letsencrypt
                                            meta.helm.sh/release-name: monitoring
                                            meta.helm.sh/release-namespace: monitoring
                                            traefik.ingress.kubernetes.io/router.entrypoints: websecure
                                            traefik.ingress.kubernetes.io/router.priority: 10
                                            traefik.ingress.kubernetes.io/router.tls: true
Events:                                     <none>

When I grpcurl the domain that the ingestor grpc endpoint is exposed on I get a response, which indicates to me that the grpc endpoint is exposed correctly:

$ grpcurl thanos-remote-grpc.example.com:443 list
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection
thanos.Exemplars
thanos.Store
thanos.WriteableStore
thanos.info.Info

[sourcehawk]

Never found an answer. Maybe a bug, maybe traefik, we will never know.

Resorted to using nginx as a forwarding proxy between vmagent and the cloud. In that case forwarding the traffic to the remote write endpoint, not the grpc endpoint.

Probably a better solution anyway.