Split Brain Resolver downs whole cluster during Kubernetes rolling update due to indirectly connected nodes

[mohammad258852]

Description

We are running Akka in Kubernetes and have multiple DCs. In DC-1 we have about 100 nodes. These nodes (in DC-1) are split into separate Kubernetes deployments. During a rolling update (We perform rolling updates deployment by deployment), we consistently encounter issues where Split Brain Resolver (SBR) downs the entire cluster.

There are two problems we are observing:

1. Nodes do not leave the cluster cleanly during rolling update

• During termination, some pods never complete the coordinated shutdown.
• The last log line we see is:
  Performing phase [cluster-sharding-shutdown-region]
  After that, there are no more logs until the coordinated shutdown times out with:
  Coordinated shutdown phase [actor-system-terminate] timed out
• We increased terminationGracePeriodSeconds, but the issue persists.
• The main concern is that the pod remains alive but produces no logs until timeout.
• This causes other nodes to mark this node as Unreachable.

2. SBR downs the whole cluster due to "indirectly connected" nodes

• Example scenario:
• Old nodes: A, B, C
• New nodes: D, E, F
• When A, B, C fail to leave gracefully, they are seen as unreachable by D, E, F.
• At the same time, in the gossip, we see that for example A reports C as unreachable.
• This creates an indirectly connected situation. (Some nodes in the unreachability list appear both as observers and as subjects. After attempting to remove some records to solve indirectly connected nodes, some indirectly connected nodes still remain)
• SBR detects this and attempts to resolve it, but fails, and as a result downs the entire cluster. (logging SBR double AcquireLeaseAndDownIndirectlyConnected(0 days) decision, downing all instead)

Logs

Log of SBR taking decision and double indirectly connected situation are provided. Node ips are replaced with pod name for better readability. for example dc1-08-69fc5fc589-z6zbt represent a node in deployment 08

SBR_took_decision

SBR took decision AcquireLeaseAndDownIndirectlyConnected(0 days) and is downing 
[
	dc1-08-69fc5fc589-s7ss7,
	dc1-07-68569cd95c-mfrfl,
	dc1-07-5bcf588fb5-mtp9s,
	dc1-05-5c5774b575-vvbch,
	dc1-06-8457566bb5-r9vzj,
	dc1-03-8668f75cbc-ws524,
	dc1-11-5886f59c47-b2db9,
	dc1-04-5d9bc7cfd7-rbt5v,
	dc1-07-5bcf588fb5-76zpg,
	dc1-07-5bcf588fb5-2r7c4,
	dc1-06-8457566bb5-d7bsh,
	dc1-13-7754984647-7flhg,
	dc1-11-5886f59c47-bt8c9,
	dc1-02-59f5d666b9-lmqk2,
	dc1-09-d54fc845-9xpvd,
	dc1-06-866954db9-4q8jd,
	dc1-10-57cf958677-fkzc7,
	dc1-03-8668f75cbc-8476v,
	dc1-08-69fc5fc589-mjbw4,
	dc1-04-5d9bc7cfd7-kmzb2,
	dc1-06-866954db9-6f5p2,
	dc1-11-5886f59c47-m6x7f,
	dc1-09-d54fc845-bp7c2,
	dc1-08-69fc5fc589-wjcmg,
	dc1-07-5bcf588fb5-5wcm4,
	dc1-12-cd7d59977-crjwv,
	dc1-03-8668f75cbc-7lzgd,
	dc1-02-59f5d666b9-vxxkq,
	dc1-06-866954db9-g5rm6,
	dc1-06-8457566bb5-2t897,
	dc1-06-866954db9-cgf5w,
	dc1-07-68569cd95c-kbs9v,
	dc1-03-8668f75cbc-vr5hh,
	dc1-08-69fc5fc589-28vx6,
	dc1-07-68569cd95c-npbtl,
	dc1-05-5c5774b575-bzmzs,
	dc1-09-d54fc845-vvhpj,
	dc1-10-57cf958677-zg8qw,
	dc1-06-8457566bb5-qczd2,
	dc1-08-69fc5fc589-xnmv7,
	dc1-13-7754984647-627fx,
	dc1-06-8457566bb5-d2hl4,
	dc1-01-598f8f6d55-sk7mv,
	dc1-10-57cf958677-zgf7v,
	dc1-06-866954db9-45jzt,
	dc1-04-5d9bc7cfd7-q4542,
	dc1-08-69fc5fc589-f5kcq,
	dc1-05-5c5774b575-gj2r7,
	dc1-03-8668f75cbc-tvj5t,
	dc1-08-69fc5fc589-c5lxt,
	dc1-01-598f8f6d55-dxn2v,
	dc1-05-5c5774b575-lgmm2,
	dc1-08-69fc5fc589-5b9c4,
	dc1-01-598f8f6d55-qtxjw,
	dc1-01-598f8f6d55-rzhnz,
	dc1-07-68569cd95c-n9hfc,
	dc1-08-69fc5fc589-49jb5,
	dc1-02-59f5d666b9-j6nwf,
	dc1-05-5c5774b575-cggkt,
	dc1-10-57cf958677-8xbwq,
	dc1-06-866954db9-csv5c,
	dc1-04-5d9bc7cfd7-hdwjq,
	dc1-12-cd7d59977-wn4rr,
	dc1-03-8668f75cbc-cx2sr,
	dc1-07-68569cd95c-2wczm,
	dc1-02-59f5d666b9-jk5w6,
	dc1-01-598f8f6d55-m5cp8,
	dc1-06-866954db9-gdqbb,
	dc1-08-69fc5fc589-hq9mx,
	dc1-06-8457566bb5-skr7c,
	dc1-02-59f5d666b9-slb62,
	dc1-01-598f8f6d55-p8xs8,
	dc1-04-5d9bc7cfd7-ds4fl,
	dc1-08-69fc5fc589-gr9bs,
	dc1-13-7754984647-hrp4m,
	dc1-07-5bcf588fb5-hrcxz,
	dc1-06-8457566bb5-gstzk,
	dc1-01-598f8f6d55-rbwpb,
	dc1-09-d54fc845-xqbpj,
	dc1-09-d54fc845-4w2fc,
	dc1-13-7754984647-rl5rs,
	dc1-09-d54fc845-hw965,
	dc1-07-5bcf588fb5-jgx7d,
	dc1-08-69fc5fc589-8qbwj,
	dc1-05-5c5774b575-mppvk,
	dc1-10-57cf958677-67wws,
	dc1-07-68569cd95c-gw4jk,
	dc1-11-5886f59c47-kr96j,
	dc1-07-68569cd95c-lrbmp,
	dc1-02-59f5d666b9-4hmq6,
	dc1-07-5bcf588fb5-6hldb,
	dc1-05-5c5774b575-cdq7b,
	dc1-06-866954db9-lhlnc,
	dc1-03-8668f75cbc-6nldb,
	dc1-02-59f5d666b9-snq4t,
	dc1-11-5886f59c47-7d66s,
	dc1-04-5d9bc7cfd7-wt2gs,
	dc1-07-5bcf588fb5-pg872,
	dc1-10-57cf958677-44ld2,
	dc1-08-69fc5fc589-z6zbt,
	dc1-09-d54fc845-sjmzq,
	dc1-12-cd7d59977-gjzwk,
	dc1-13-7754984647-696dv,
	dc1-12-cd7d59977-lmttl,
	dc1-07-68569cd95c-rddq9,
	dc1-06-8457566bb5-5bgfg,
	dc1-06-8457566bb5-jm65k,
	dc1-07-68569cd95c-4qq7m,
	dc1-08-69fc5fc589-zts8f,
	dc1-05-5c5774b575-5b8sw,
	dc1-11-5886f59c47-j6q4m,
	dc1-07-5bcf588fb5-6wmps,
	dc1-05-5c5774b575-g9hpk,
	dc1-09-d54fc845-2w75n,
	dc1-06-866954db9-b8bq8,
	dc1-04-5d9bc7cfd7-zss8x,
	dc1-07-68569cd95c-gsx2v,
	dc1-07-5bcf588fb5-k5nvz
] 
including myself, [10] unreachable of [100] members, indirectly connected 
[
	UniqueAddress(dc1-07-68569cd95c-mfrfl,-8912227370481045792),
	UniqueAddress(dc1-07-68569cd95c-4qq7m,-1898628812236640612),
	UniqueAddress(dc1-07-68569cd95c-n9hfc,7854319559869018800)
], 
all members in DC [
	Member(dc1-09-d54fc845-4w2fc, Up, 3.46.1),
	Member(dc1-02-59f5d666b9-j6nwf, Up, 3.45.2),
	Member(dc1-11-5886f59c47-b2db9, Up, 3.46.1),
	Member(dc1-08-69fc5fc589-xnmv7, Up, 3.46.1),
	Member(dc1-01-598f8f6d55-dxn2v, Up, 3.45.2),
	Member(dc1-02-59f5d666b9-lmqk2, Up, 3.45.2),
	Member(dc1-09-d54fc845-bp7c2, Up, 3.46.1),
	Member(dc1-01-598f8f6d55-rzhnz, Up, 3.45.2),
	Member(dc1-10-57cf958677-8xbwq, Up, 3.46.1),
	Member(dc1-09-d54fc845-vvhpj, Up, 3.46.1),
	Member(dc1-02-59f5d666b9-vxxkq, Up, 3.45.2),
	Member(dc1-10-57cf958677-fkzc7, Up, 3.46.1),
	Member(dc1-09-d54fc845-9xpvd, Up, 3.46.1),
	Member(dc1-07-5bcf588fb5-2r7c4, WeaklyUp, 3.46.1),
	Member(dc1-02-59f5d666b9-4hmq6, Up, 3.45.2),
	Member(dc1-04-5d9bc7cfd7-hdwjq, Up, 3.45.2),
	Member(dc1-06-866954db9-4q8jd, Up, 3.45.2),
	Member(dc1-07-5bcf588fb5-k5nvz, WeaklyUp, 3.46.1),
	Member(dc1-04-5d9bc7cfd7-kmzb2, Up, 3.45.2),
	Member(dc1-08-69fc5fc589-s7ss7, Up, 3.46.1),
	Member(dc1-11-5886f59c47-bt8c9, Up, 3.46.1),
	Member(dc1-07-5bcf588fb5-mtp9s, WeaklyUp, 3.46.1),
	Member(dc1-06-866954db9-cgf5w, Leaving, 3.45.2),
	Member(dc1-07-5bcf588fb5-jgx7d, WeaklyUp, 3.46.1),
	Member(dc1-08-69fc5fc589-hq9mx, Up, 3.46.1),
	Member(dc1-07-5bcf588fb5-6wmps, Up, 3.46.1),
	Member(dc1-06-866954db9-6f5p2, Leaving, 3.45.2),
	Member(dc1-05-5c5774b575-bzmzs, Up, 3.45.2),
	Member(dc1-07-5bcf588fb5-pg872, WeaklyUp, 3.46.1),
	Member(dc1-07-68569cd95c-2wczm, Leaving, 3.45.2),
	Member(dc1-03-8668f75cbc-vr5hh, Up, 3.45.2),
	Member(dc1-06-8457566bb5-r9vzj, WeaklyUp, 3.46.1),
	Member(dc1-05-5c5774b575-vvbch, Up, 3.45.2),
	Member(dc1-13-7754984647-627fx, Up, 3.46.1),
	Member(dc1-08-69fc5fc589-c5lxt, Up, 3.46.1),
	Member(dc1-07-68569cd95c-mfrfl, Leaving, 3.45.2),
	Member(dc1-05-5c5774b575-5b8sw, Up, 3.45.2),
	Member(dc1-06-8457566bb5-2t897, WeaklyUp, 3.46.1),
	Member(dc1-13-7754984647-696dv, Up, 3.46.1),
	Member(dc1-12-cd7d59977-gjzwk, Up, 3.46.1),
	Member(dc1-09-d54fc845-xqbpj, Up, 3.46.1),
	Member(dc1-01-598f8f6d55-p8xs8, Up, 3.45.2),
	Member(dc1-06-866954db9-gdqbb, Up, 3.45.2),
	Member(dc1-07-68569cd95c-npbtl, Leaving, 3.45.2),
	Member(dc1-05-5c5774b575-gj2r7, Up, 3.45.2),
	Member(dc1-03-8668f75cbc-6nldb, Up, 3.45.2),
	Member(dc1-06-8457566bb5-d7bsh, WeaklyUp, 3.46.1),
	Member(dc1-13-7754984647-hrp4m, Up, 3.46.1),
	Member(dc1-03-8668f75cbc-cx2sr, Up, 3.45.2),
	Member(dc1-07-68569cd95c-gw4jk, Leaving, 3.45.2),
	Member(dc1-05-5c5774b575-g9hpk, Up, 3.45.2),
	Member(dc1-08-69fc5fc589-49jb5, Up, 3.46.1),
	Member(dc1-05-5c5774b575-lgmm2, Up, 3.45.2),
	Member(dc1-07-68569cd95c-4qq7m, Leaving, 3.45.2),
	Member(dc1-03-8668f75cbc-tvj5t, Up, 3.45.2),
	Member(dc1-06-866954db9-b8bq8, Leaving, 3.45.2),
	Member(dc1-07-68569cd95c-lrbmp, Up, 3.45.2),
	Member(dc1-08-69fc5fc589-zts8f, Up, 3.46.1),
	Member(dc1-06-866954db9-csv5c, Leaving, 3.45.2),
	Member(dc1-13-7754984647-7flhg, Up, 3.46.1),
	Member(dc1-12-cd7d59977-lmttl, Up, 3.46.1),
	Member(dc1-09-d54fc845-sjmzq, Up, 3.46.1),
	Member(dc1-02-59f5d666b9-snq4t, Up, 3.45.2),
	Member(dc1-07-5bcf588fb5-6hldb, WeaklyUp, 3.46.1),
	Member(dc1-05-5c5774b575-mppvk, Up, 3.45.2),
	Member(dc1-04-5d9bc7cfd7-zss8x, Up, 3.45.2),
	Member(dc1-08-69fc5fc589-mjbw4, Up, 3.46.1),
	Member(dc1-04-5d9bc7cfd7-ds4fl, Up, 3.45.2),
	Member(dc1-06-8457566bb5-d2hl4, WeaklyUp, 3.46.1),
	Member(dc1-07-68569cd95c-gsx2v, Leaving, 3.45.2),
	Member(dc1-07-68569cd95c-rddq9, Leaving, 3.45.2),
	Member(dc1-06-8457566bb5-5bgfg, WeaklyUp, 3.46.1),
	Member(dc1-08-69fc5fc589-8qbwj, Up, 3.46.1),
	Member(dc1-01-598f8f6d55-sk7mv, Up, 3.45.2),
	Member(dc1-10-57cf958677-zgf7v, Up, 3.46.1),
	Member(dc1-13-7754984647-rl5rs, Up, 3.46.1),
	Member(dc1-06-8457566bb5-qczd2, WeaklyUp, 3.46.1),
	Member(dc1-04-5d9bc7cfd7-wt2gs, Up, 3.45.2),
	Member(dc1-07-68569cd95c-n9hfc, Leaving, 3.45.2),
	Member(dc1-05-5c5774b575-cdq7b, Up, 3.45.2),
	Member(dc1-07-5bcf588fb5-76zpg, WeaklyUp, 3.46.1),
	Member(dc1-06-866954db9-g5rm6, Leaving, 3.45.2),
	Member(dc1-08-69fc5fc589-5b9c4, Up, 3.46.1),
	Member(dc1-06-866954db9-lhlnc, Up, 3.45.2),
	Member(dc1-05-5c5774b575-cggkt, Up, 3.45.2),
	Member(dc1-03-8668f75cbc-8476v, Up, 3.45.2),
	Member(dc1-07-68569cd95c-kbs9v, Leaving, 3.45.2),
	Member(dc1-01-598f8f6d55-rbwpb, Up, 3.45.2),
	Member(dc1-04-5d9bc7cfd7-rbt5v, Up, 3.45.2),
	Member(dc1-10-57cf958677-zg8qw, Up, 3.46.1),
	Member(dc1-08-69fc5fc589-wjcmg, Up, 3.46.1),
	Member(dc1-10-57cf958677-44ld2, Up, 3.46.1),
	Member(dc1-02-59f5d666b9-slb62, Up, 3.45.2),
	Member(dc1-08-69fc5fc589-28vx6, Up, 3.46.1),
	Member(dc1-12-cd7d59977-wn4rr, Up, 3.46.1),
	Member(dc1-02-59f5d666b9-jk5w6, Up, 3.45.2),
	Member(dc1-10-57cf958677-67wws, Up, 3.46.1),
	Member(dc1-03-8668f75cbc-ws524, Up, 3.45.2),
	Member(dc1-04-5d9bc7cfd7-q4542, Up, 3.45.2),
	Member(dc1-11-5886f59c47-m6x7f, Up, 3.46.1),
	Member(dc1-06-866954db9-45jzt, Leaving, 3.45.2),
	Member(dc1-07-5bcf588fb5-hrcxz, WeaklyUp, 3.46.1),
	Member(dc1-01-598f8f6d55-qtxjw, Up, 3.45.2),
	Member(dc1-12-cd7d59977-crjwv, Up, 3.46.1),
	Member(dc1-09-d54fc845-hw965, Up, 3.46.1),
	Member(dc1-06-8457566bb5-skr7c, WeaklyUp, 3.46.1),
	Member(dc1-11-5886f59c47-7d66s, Up, 3.46.1),
	Member(dc1-07-5bcf588fb5-5wcm4, WeaklyUp, 3.46.1),
	Member(dc1-06-8457566bb5-gstzk, WeaklyUp, 3.46.1),
	Member(dc1-11-5886f59c47-j6q4m, Up, 3.46.1),
	Member(dc1-08-69fc5fc589-z6zbt, Up, 3.46.1),
	Member(dc1-06-8457566bb5-jm65k, WeaklyUp, 3.46.1),
	Member(dc1-11-5886f59c47-kr96j, Up, 3.46.1),
	Member(dc1-08-69fc5fc589-gr9bs, Up, 3.46.1),
	Member(dc1-01-598f8f6d55-m5cp8, Up, 3.45.2),
	Member(dc1-09-d54fc845-2w75n, Up, 3.46.1),
	Member(dc1-03-8668f75cbc-7lzgd, Up, 3.45.2),
	Member(dc1-08-69fc5fc589-f5kcq, Up, 3.46.1)
], 
full reachability status: 
[
	dc1-09-d54fc845-4w2fc -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-b2db9 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-b2db9 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-xnmv7 -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-xnmv7 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-lmqk2 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (2),
	dc1-09-d54fc845-bp7c2 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-8xbwq -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-vxxkq -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-09-d54fc845-9xpvd -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-2r7c4 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-4hmq6 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-k5nvz -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-k5nvz -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-s7ss7 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-mtp9s -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-6wmps -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-6wmps -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-pg872 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-pg872 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-03-8668f75cbc-vr5hh -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-07-68569cd95c-mfrfl -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-13-7754984647-696dv -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-13-7754984647-696dv -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-09-d54fc845-xqbpj -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-p8xs8 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-06-866954db9-gdqbb -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-06-866954db9-gdqbb -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-g9hpk -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-g9hpk -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-lgmm2 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-07-68569cd95c-4qq7m -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-06-866954db9-csv5c -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-02-59f5d666b9-snq4t -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-snq4t -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-04-5d9bc7cfd7-ds4fl -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (1),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (2),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-13-7754984647-rl5rs -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-07-68569cd95c-n9hfc -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (4),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (3),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (5),
	dc1-07-5bcf588fb5-76zpg -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-76zpg -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-06-866954db9-g5rm6 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-06-866954db9-lhlnc -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (3),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (2),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (4),
	dc1-03-8668f75cbc-8476v -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-04-5d9bc7cfd7-rbt5v -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-zg8qw -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-wjcmg -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-02-59f5d666b9-slb62 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-slb62 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-10-57cf958677-67wws -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-04-5d9bc7cfd7-q4542 -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-04-5d9bc7cfd7-q4542 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-06-866954db9-45jzt -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-hrcxz -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-qtxjw -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-06-8457566bb5-skr7c -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-7d66s -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-5wcm4 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-5wcm4 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-06-8457566bb5-gstzk -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-z6zbt -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-kr96j -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-m5cp8 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-09-d54fc845-2w75n -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-03-8668f75cbc-7lzgd -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-03-8668f75cbc-7lzgd -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (2)
]

SBR_double

SBR double AcquireLeaseAndDownIndirectlyConnected(0 days) decision, downing all instead. originalReachability: 
[
	dc1-09-d54fc845-4w2fc -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-b2db9 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-b2db9 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-xnmv7 -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-xnmv7 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-lmqk2 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (2),
	dc1-09-d54fc845-bp7c2 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-8xbwq -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-vxxkq -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-09-d54fc845-9xpvd -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-2r7c4 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-4hmq6 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-k5nvz -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-k5nvz -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-s7ss7 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-mtp9s -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-6wmps -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-6wmps -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-pg872 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-pg872 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-03-8668f75cbc-vr5hh -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-07-68569cd95c-mfrfl -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-13-7754984647-696dv -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-13-7754984647-696dv -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-09-d54fc845-xqbpj -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-p8xs8 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-06-866954db9-gdqbb -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-06-866954db9-gdqbb -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-g9hpk -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-g9hpk -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-lgmm2 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-07-68569cd95c-4qq7m -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-06-866954db9-csv5c -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-02-59f5d666b9-snq4t -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-snq4t -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-04-5d9bc7cfd7-ds4fl -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (1),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (2),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-13-7754984647-rl5rs -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-07-68569cd95c-n9hfc -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (4),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (3),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (5),
	dc1-07-5bcf588fb5-76zpg -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-76zpg -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-06-866954db9-g5rm6 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-06-866954db9-lhlnc -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (3),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (2),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (4),
	dc1-03-8668f75cbc-8476v -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-04-5d9bc7cfd7-rbt5v -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-zg8qw -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-wjcmg -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-02-59f5d666b9-slb62 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-slb62 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-10-57cf958677-67wws -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-04-5d9bc7cfd7-q4542 -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-04-5d9bc7cfd7-q4542 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-06-866954db9-45jzt -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-hrcxz -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-qtxjw -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-06-8457566bb5-skr7c -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-7d66s -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-5wcm4 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-5wcm4 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-06-8457566bb5-gstzk -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-z6zbt -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-kr96j -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-m5cp8 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-09-d54fc845-2w75n -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-03-8668f75cbc-7lzgd -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-03-8668f75cbc-7lzgd -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (2)
], 
filtered reachability 
[
	dc1-09-d54fc845-4w2fc -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-b2db9 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-b2db9 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-xnmv7 -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-xnmv7 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-lmqk2 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (2),
	dc1-09-d54fc845-bp7c2 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-8xbwq -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-vxxkq -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-09-d54fc845-9xpvd -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-2r7c4 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-4hmq6 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-k5nvz -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-k5nvz -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-s7ss7 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-mtp9s -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-6wmps -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-6wmps -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-pg872 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-pg872 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-03-8668f75cbc-vr5hh -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-07-68569cd95c-mfrfl -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-13-7754984647-696dv -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-13-7754984647-696dv -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-09-d54fc845-xqbpj -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-p8xs8 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-06-866954db9-gdqbb -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-06-866954db9-gdqbb -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-g9hpk -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-g9hpk -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-08-69fc5fc589-49jb5 -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-lgmm2 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-07-68569cd95c-4qq7m -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-06-866954db9-csv5c -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-02-59f5d666b9-snq4t -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-snq4t -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-04-5d9bc7cfd7-ds4fl -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (3),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (1),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (2),
	dc1-06-8457566bb5-5bgfg -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-13-7754984647-rl5rs -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-07-68569cd95c-n9hfc -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (4),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (3),
	dc1-05-5c5774b575-cdq7b -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (5),
	dc1-07-5bcf588fb5-76zpg -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-07-5bcf588fb5-76zpg -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-06-866954db9-g5rm6 -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-06-866954db9-lhlnc -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (2),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (3),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (2),
	dc1-05-5c5774b575-cggkt -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (4),
	dc1-03-8668f75cbc-8476v -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-04-5d9bc7cfd7-rbt5v -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-zg8qw -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-wjcmg -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-10-57cf958677-44ld2 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-02-59f5d666b9-slb62 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-02-59f5d666b9-slb62 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-28vx6 -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-10-57cf958677-67wws -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-04-5d9bc7cfd7-q4542 -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (2),
	dc1-04-5d9bc7cfd7-q4542 -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-gw4jk: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-m6x7f -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-06-866954db9-45jzt -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-hrcxz -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-qtxjw -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-mfrfl: Unreachable [Unreachable] (2),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-4qq7m: Unreachable [Unreachable] (3),
	dc1-12-cd7d59977-crjwv -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (1),
	dc1-06-8457566bb5-skr7c -> dc1-07-68569cd95c-rddq9: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-7d66s -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-5wcm4 -> dc1-07-68569cd95c-2wczm: Unreachable [Unreachable] (1),
	dc1-07-5bcf588fb5-5wcm4 -> dc1-07-68569cd95c-kbs9v: Unreachable [Unreachable] (2),
	dc1-06-8457566bb5-gstzk -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-08-69fc5fc589-z6zbt -> dc1-07-68569cd95c-npbtl: Unreachable [Unreachable] (1),
	dc1-11-5886f59c47-kr96j -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-01-598f8f6d55-m5cp8 -> dc1-07-68569cd95c-n9hfc: Unreachable [Unreachable] (1),
	dc1-09-d54fc845-2w75n -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-03-8668f75cbc-7lzgd -> dc1-07-68569cd95c-lrbmp: Unreachable [Unreachable] (1),
	dc1-03-8668f75cbc-7lzgd -> dc1-07-68569cd95c-gsx2v: Unreachable [Unreachable] (2)
], 
still indirectlyConnected: 
[HashSet(
	UniqueAddress(dc1-07-68569cd95c-mfrfl,-8912227370481045792),
	UniqueAddress(dc1-07-68569cd95c-4qq7m,-1898628812236640612),
	UniqueAddress(dc1-07-68569cd95c-n9hfc,7854319559869018800))
], 
seenBy: 
[HashSet(
	dc1-03-8668f75cbc-cx2sr,
	dc1-05-5c5774b575-vvbch,
	dc1-06-8457566bb5-r9vzj,
	dc1-03-8668f75cbc-ws524,
	dc1-11-5886f59c47-b2db9,
	dc1-07-5bcf588fb5-76zpg,
	dc1-07-5bcf588fb5-2r7c4,
	dc1-06-8457566bb5-d7bsh,
	dc1-13-7754984647-7flhg,
	dc1-08-69fc5fc589-s7ss7,
	dc1-11-5886f59c47-bt8c9,
	dc1-02-59f5d666b9-lmqk2,
	dc1-09-d54fc845-9xpvd,
	dc1-06-866954db9-4q8jd,
	dc1-10-57cf958677-fkzc7,
	dc1-03-8668f75cbc-8476v,
	dc1-08-69fc5fc589-mjbw4,
	dc1-11-5886f59c47-m6x7f,
	dc1-04-5d9bc7cfd7-kmzb2,
	dc1-06-866954db9-6f5p2,
	dc1-09-d54fc845-bp7c2,
	dc1-08-69fc5fc589-wjcmg,
	dc1-12-cd7d59977-crjwv,
	dc1-03-8668f75cbc-7lzgd,
	dc1-02-59f5d666b9-vxxkq,
	dc1-06-866954db9-g5rm6,
	dc1-06-8457566bb5-2t897,
	dc1-06-866954db9-cgf5w,
	dc1-03-8668f75cbc-vr5hh,
	dc1-08-69fc5fc589-28vx6,
	dc1-05-5c5774b575-bzmzs,
	dc1-09-d54fc845-vvhpj,
	dc1-10-57cf958677-zg8qw,
	dc1-06-8457566bb5-qczd2,
	dc1-08-69fc5fc589-xnmv7,
	dc1-13-7754984647-627fx,
	dc1-06-8457566bb5-d2hl4,
	dc1-01-598f8f6d55-sk7mv,
	dc1-10-57cf958677-zgf7v,
	dc1-06-866954db9-45jzt,
	dc1-04-5d9bc7cfd7-q4542,
	dc1-05-5c5774b575-gj2r7,
	dc1-03-8668f75cbc-tvj5t,
	dc1-08-69fc5fc589-c5lxt,
	dc1-01-598f8f6d55-dxn2v,
	dc1-05-5c5774b575-lgmm2,
	dc1-08-69fc5fc589-5b9c4,
	dc1-01-598f8f6d55-qtxjw,
	dc1-01-598f8f6d55-rzhnz,
	dc1-08-69fc5fc589-49jb5,
	dc1-02-59f5d666b9-j6nwf,
	dc1-05-5c5774b575-cggkt,
	dc1-10-57cf958677-8xbwq,
	dc1-06-866954db9-csv5c,
	dc1-04-5d9bc7cfd7-hdwjq,
	dc1-02-59f5d666b9-jk5w6,
	dc1-01-598f8f6d55-m5cp8,
	dc1-06-866954db9-gdqbb,
	dc1-08-69fc5fc589-hq9mx,
	dc1-06-8457566bb5-skr7c,
	dc1-01-598f8f6d55-p8xs8,
	dc1-04-5d9bc7cfd7-ds4fl,
	dc1-08-69fc5fc589-gr9bs,
	dc1-13-7754984647-hrp4m,
	dc1-06-8457566bb5-gstzk,
	dc1-09-d54fc845-xqbpj,
	dc1-09-d54fc845-4w2fc,
	dc1-01-598f8f6d55-rbwpb,
	dc1-07-5bcf588fb5-mtp9s,
	dc1-13-7754984647-rl5rs,
	dc1-09-d54fc845-hw965,
	dc1-07-5bcf588fb5-jgx7d,
	dc1-08-69fc5fc589-8qbwj,
	dc1-05-5c5774b575-mppvk,
	dc1-10-57cf958677-67wws,
	dc1-11-5886f59c47-kr96j,
	dc1-02-59f5d666b9-4hmq6,
	dc1-07-5bcf588fb5-6hldb,
	dc1-05-5c5774b575-cdq7b,
	dc1-06-866954db9-lhlnc,
	dc1-03-8668f75cbc-6nldb,
	dc1-11-5886f59c47-7d66s,
	dc1-04-5d9bc7cfd7-wt2gs,
	dc1-07-5bcf588fb5-pg872,
	dc1-10-57cf958677-44ld2,
	dc1-02-59f5d666b9-snq4t,
	dc1-08-69fc5fc589-z6zbt,
	dc1-09-d54fc845-sjmzq,
	dc1-12-cd7d59977-gjzwk,
	dc1-13-7754984647-696dv,
	dc1-12-cd7d59977-lmttl,
	dc1-06-8457566bb5-5bgfg,
	dc1-06-8457566bb5-jm65k,
	dc1-08-69fc5fc589-zts8f,
	dc1-05-5c5774b575-5b8sw,
	dc1-11-5886f59c47-j6q4m,
	dc1-07-5bcf588fb5-6wmps,
	dc1-05-5c5774b575-g9hpk,
	dc1-06-866954db9-b8bq8,
	dc1-04-5d9bc7cfd7-zss8x,
	dc1-07-5bcf588fb5-k5nvz
)]

Environment

• Akka version: 2.10.9
• Kubernetes version: v1.29
• Number of nodes: ~100 nodes
• SBR strategy: LeaseMajority

[sebastian-alfers]

Hi @mohammad258852, for advanced reports like this one we typically use our dedicated support channel to assist further. Please forward your request to support@akka.io.

Thanks,
Sebastian

[mohammad258852]

Hi @sebastian-alfers,
Thanks for your response. I understand that detailed reports usually go through the dedicated support channel.

However, the main point I wanted to raise here is that the issue we’re observing seems to be a potential bug in Akka itself — specifically the occurrence of double indirectly connected nodes during rolling updates.

From my perspective, this is not only a deployment/setup problem in our environment but could happen in any cluster setup. In particular, whenever multiple nodes go down around the same time (for example during rolling updates or failures), this situation can appear and lead to the entire cluster being downed by SBR.

That’s why I believe it’s important to highlight it here, as it might be a more general bug in the project.

[patriknw]

Thanks for reporting. The coordinated shutdown is crucial for smooth rolling updates. SBR shouldn't be involved in a normal, smooth rolling update. As you already noticed, your system is not terminated within the timeout and then the JVMs are hard killed and that may cause unreachability and SBR to take over. We are happy to help you troubleshoot the root cause further in the support channel.