Alertmanager sending duplicate notifications after a 'resolved' event when running with multiple replicas.

[pilgaonkarpranav]

We're encountering an issue with Alertmanager sending duplicate notifications after a 'resolved' event when running with multiple replicas. Currently, we have 3 replicas running with zone-aware replication enabled.
Here's the observed sequence of events:
Trigger (initial trigger timestamp)
Resolve (initial resolve timestamp)
Trigger (received ~<10s after resolve timestamp)
Trigger (received ~<10s after resolve timestamp)
Resolve (received ~<10s after resolve timestamp)
This sequence is resulting in duplicate notifications.
Please find our Alertmanager configuration below for reference:

alertmanager:
    data_dir: /data
    retention: 120h0m0s
    external_url: /alertmanager
    poll_interval: 15s
    max_recv_msg_size: 104857600
    sharding_ring:
        kvstore:
            store: etcd
            prefix: alertmanagers/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints:
                    - test-etcd:2379
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: test-mimir-alertmanager-zone-a-0
        instance_interface_names:
            - eth0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
        replication_factor: 3
        zone_awareness_enabled: true
        instance_availability_zone: zone-a
    fallback_config_file: /configs/alertmanager_fallback_config.yaml
    peer_timeout: 15s
    enable_api: true
    grafana_alertmanager_compatibility_enabled: false
    grafana_alertmanager_conditionally_skip_tenant_suffix: ""
    max_concurrent_get_requests_per_tenant: 0
    alertmanager_client:
        remote_timeout: 2s
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
    persist_interval: 15m0s
    enable_state_cleanup: true
    utf8_strict_mode: false
    log_parsing_label_matchers: false
    utf8_migration_logging: false
    enable_notify_hooks: false

Expression:

    expr: "max by(instance) (rate(alloy_resources_process_cpu_seconds_total{instance='172.22.1.4'}[1m])) > 0.057"
    for: 2m

AM Config:

	global:
      resolve_timeout: 10s
    route:
      group_by: ['...']
      group_interval: 10s
      group_wait: 10s
      repeat_interval: 5m

When we changed group_interval to 1m additional notifications are not triggered after resolved notifications. What are the implications of changing group_interval to 1m?
analysis.pdf
mimir-alertmanager-zone_alertmanager_txt.txt