[k8s, helm]nginx: [emerg] socket() :8080 failed (97: Address family not supported by protocol)

[MaJJangJJang]

hello.
While installing to test mimir, the following log occurs in gateway-nginx.

The version I'm using is:
k8s: 1.27.9
helm: helm.sh/chart=mimir-distributed-5.3.0

nginx: [emerg] socket() :8080 failed (97: Address family not supported by protocol)

I looked up the bug and found that it has been resolved, but I'm curious why it's happening again.
#7210

values.yaml

gateway:
  nginx:
    config:
      enableIPv6: false

[dimitarvdimitrov]

this looks like an nginx config bug. Happy to help with making a change in the chart if you can help with finding how to fix it in nginx

[jammy-d]

Having experienced a similar issue, I'm leaning towards this is triggered because of some confusion between nginx/nginx-configmap.yaml and gateway/nginx-config.yaml, coupled with some discrepancy between the defaults of .Values.nginx.nginxConfig.file and .Values.gateway.nginx.config.file.

The behaviour is also coming differently when directly using mimir-distributed vs calling lgtm-distributed which pulls in mimir-distributed as a dependency

gateway:
  enabledNonEnterprise: false
  nginx:
    config:
      enableIPv6: false
nginx:
  enabled: true

helm template lgtm grafana/mimir-distributed --dry-run --debug -f values.yaml --version 5.5.1 2>/dev/null | grep -A 3 "server {"
      server {
        listen 8080;
        listen [::]:8080;

helm template lgtm grafana/lgtm-distributed --dry-run --debug -f values.yaml --version 2.1.0  2>/dev/null | grep -A 3 "server {"
      server {
        listen             8080;
    
        location = / {
--
      server {
        listen 8080;
        listen [::]:8080;

---

gateway:
  enabledNonEnterprise: true
  nginx:
    config:
      enableIPv6: false
nginx:
  enabled: true

helm template lgtm grafana/mimir-distributed --dry-run --debug -f values.yaml --version 5.5.1 2>/dev/null | grep -A 3 "server {"
      server {
        listen 8080;
        client_max_body_size 540M;
    
--
      server {
        listen 8080;
        listen [::]:8080;

helm template lgtm grafana/lgtm-distributed --dry-run --debug -f values.yaml --version 2.1.0  2>/dev/null | grep -A 3 "server {"
      server {
        listen             8080;
    
        location = / {
--
      server {
        listen 8080;
        listen [::]:8080;

---

gateway:
  enabledNonEnterprise: false
  nginx:
    config:
      enableIPv6: false
nginx:
  enabled: false

helm template lgtm grafana/mimir-distributed --dry-run --debug -f values.yaml --version 5.5.1 2>/dev/null | grep -A 3 "server {"

helm template lgtm grafana/lgtm-distributed --dry-run --debug -f values.yaml --version 2.1.0  2>/dev/null | grep -A 3 "server {"
      server {
        listen             8080;
    
        location = / {
--
      server {
        listen 8080;
        listen [::]:8080;

---

gateway:
  enabledNonEnterprise: true
  nginx:
    config:
      enableIPv6: false
nginx:
  enabled: false

helm template lgtm grafana/mimir-distributed --dry-run --debug -f values.yaml --version 5.5.1 2>/dev/null | grep -A 3 "server {"
      server {
        listen 8080;
        client_max_body_size 540M;

helm template lgtm grafana/lgtm-distributed --dry-run --debug -f values.yaml --version 2.1.0  2>/dev/null | grep -A 3 "server {"
      server {
        listen             8080;
    
        location = / {
--
      server {
        listen 8080;
        listen [::]:8080;

In addition, it seems that it is possible to workaround this by overriding nginx.nginxConfig.file when using mimir-distributed directly, but that doesn't work when calling it via lgtm-distributed

[dimitarvdimitrov]

do i understand right that the problem is in the lgtm-distributed helm chart?

[jammy-d]

The problem looks like it's in the underlying mimir-distrbuted chart, but causing different behaviour in the base chart and the lgtm-distributed chart, which I'm not fully able to follow.

If you compare the difference between nginx/nginx-configmap.yaml and gateway/nginx-config.yaml - they're refering to .Values.nginx.nginxConfig.file and .Values.gateway.nginx.config.file, respectively.

If you compare the default values for .Values.nginx.nginxConfig.file and .Values.gateway.nginx.config.file, one has directly hardcoded:

listen [::]:8080;

while the other has:

{{- if .Values.gateway.nginx.config.enableIPv6 }}
listen [::]:{{ include "mimir.serverHttpListenPort" . }};
{{- end }}

This is likely one of the factors at play.

However, there's a more tricky problem around based on the combination of nginx.enabled and gateway.enabledNonEnterprise flags causing different configmaps to be created (sometimes multiple). This is experienced differently when using mimir-distrbuted and lgtm-distributed, and I don't quite understand the trend, nor which flags should be used together.