Memory consuption when running on a docker swarm cluster

[lepiroupo]

Describe the bug

Hi! I'm running a cluster using Docker Swarm, consul and RabbitMq 3.13.

During the use, the memory shown in my monitoring tool is higher than the memory that is reported by the rabbits nodes.

I've updated until the 3.13 latest version (i'm doing tests with 4.0 version) but it seems not to be a version problem.

Any sugestions?

Reproduction steps

Running this compose on a swarm cluster and send and consume a 300 m/s.

Docker Swarm config

version: '3.3'
services:
  consul:
    image: consul:1.12.3
    command:
     - agent
     - -dev
     - -ui
     - -client=0.0.0.0
     - -dns-port=53
     - -recursor=8.8.8.8
    environment:
      CONSUL_BIND_INTERFACE: eth0
    networks:
     - messagequeue
     - proxy
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    deploy:
      labels:
        traefik.http.routers.consulrabbitmq.entrypoints: websecure
        traefik.http.routers.consulrabbitmq.rule: Host(`consul-rabbitmq-cloud.ibiuna.local`)
        traefik.http.routers.consulrabbitmq.tls: 'true'
        traefik.http.services.consulrabbitmq-service.loadbalancer.server.port: '8500'
      resources:
        limits:
          cpus: '2.0'
          memory: 1024M
  rabbitmq:
    image: rabbitmq:3.12.13-management-alpine
    hostname: "rabbitmq-{{.Task.Slot}}"
    healthcheck:
      test:
       - CMD-SHELL
       - rabbitmq-diagnostics -q ping
      interval: 30s
      timeout: 30s
      retries: 3
    volumes:
     - /ibiuna/swarm-cluster-volumes/volumes/rabbitmq/mqdata-{{.Task.Slot}}:/var/lib/rabbitmq/mnesia
    networks:
     - messagequeue
     - observability_observability
     - proxy
    secrets:
     -
      source: RABBITMQ_ERLANG_COOKIE
      target: /var/lib/rabbitmq/.erlang.cookie
      gid: "101"
      uid: "100"
      mode: 0600
    configs:
     -
      source: RABBITMQ_CONFIG_V2
      target: /etc/rabbitmq/rabbitmq.conf
     -
      source: RABBITMQ_PLUGINS
      target: /etc/rabbitmq/enabled_plugins
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    deploy:
      replicas: 3
      labels:
        prometheus_job: rabbitmq
        traefik.http.routers.rabbitmq.entrypoints: websecure
        traefik.http.routers.rabbitmq.rule: Host(`rabbitmq-cloud.ibiuna.local`)
        traefik.http.routers.rabbitmq.tls: 'true'
        traefik.http.services.rabbitmq-service.loadbalancer.server.port: '15672'
      resources:
        limits:
          cpus: '4.0'
          memory: 3072M
networks:
  messagequeue:
    external: true
  observability_observability:
    external: true
  proxy:
    external: true
configs:
  RABBITMQ_CONFIG_V2:
    external: true
  RABBITMQ_PLUGINS:
    external: true
secrets:
  RABBITMQ_ERLANG_COOKIE:
    external: true

Expected behavior

Memory should increase and decrease over time.

Additional context

No response

[lukebakken]

Neither RabbitMQ 3.12 nor 3.13 receives free community support - https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md

To receive free support, you must provide a detailed and well researched report. Your report is neither of those.

This is what I expect:

• A docker compose project that sets up your cluster. I don't believe anyone on Team RabbitMQ has experience with swarm. You can use my project as a starting point. Fork the code and modify to your needs - https://github.com/lukebakken/docker-rabbitmq-cluster
• As part of the compose project, a PerfTest workload that mimics yours (https://github.com/lukebakken/docker-rabbitmq-cluster/blob/main/docker-compose.yml#L52-L70)

If you can provide reproduction using the latest version of RabbitMQ, reply to this discussion and I will re-open it.

[michaelklishin]

Even if this was on 4.0.5, we would not guess what consumes memory on your system, what you consider "high" and what can be suggested. Either provide relevant data or please seek RabbitMQ help elsewhere.

Two highly relevant doc guides:

• Memory Use Breakdown
• Kernel Page Cache (a crucial doc section to read for those running RabbitMQ in containers, in particular in environments where cgroups v1 could be used