Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pod no longer able to mount persistent volume #459

Open
boedy opened this issue Apr 17, 2023 · 6 comments
Open

Pod no longer able to mount persistent volume #459

boedy opened this issue Apr 17, 2023 · 6 comments

Comments

@boedy
Copy link

boedy commented Apr 17, 2023

We are running a Strimzi Kafka cluster using Pireaus as the storage layer. Since a fews days (not sure how long), one of the kafka pods is no longer able to mount its persistent volume. The operator is now stuck in a loop recreating the pod every 10 min.

Events from the pod:

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    6m30s                 default-scheduler  Successfully assigned kafka/main-kafka-1 to eu-central-68d6
  Warning  FailedMount  4m28s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[cluster-ca broker-certs linkerd-identity-end-entity linkerd-identity-token client-ca-cert ready-files kube-api-access-d4vhg data-0 strimzi-tmp kafka-metrics-and-logging linkerd-proxy-init-xtables-lock]: timed out waiting for the condition
  Warning  FailedMount  2m10s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[kafka-metrics-and-logging linkerd-identity-end-entity data-0 linkerd-identity-token strimzi-tmp ready-files client-ca-cert kube-api-access-d4vhg cluster-ca broker-certs linkerd-proxy-init-xtables-lock]: timed out waiting for the condition
  Warning  FailedMount  17s (x11 over 6m29s)  kubelet            MountVolume.SetUp failed for volume "pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9" : rpc error: code = Internal desc = NodePublishVolume failed for pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: failed to set source device readwrite: exit status 1

# linstor volume list
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node            | Resource                                 | StoragePool      | VolNr | MinorNr | DeviceName    | Allocated | InUse  |              State |
|===========================================================================================================================================================|
| eu-central-1771 | pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | local-disk       |     0 |    1008 | /dev/drbd1008 | 40.78 GiB | InUse  |           UpToDate |
| eu-central-42f5 | pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | local-disk       |     0 |    1008 | /dev/drbd1008 | 40.78 GiB | Unused |           UpToDate |
| eu-central-68d6 | pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | local-disk       |     0 |    1008 | /dev/drbd1008 | 40.78 GiB | Unused |           UpToDate |

# linstor-csi-node
time="2023-04-17T11:16:20Z" level=info msg="mounting volume" blockAccessMode=false filesystem=ext4 linstorCSIComponent=client mountOpts="[_netdev]" source=/dev/drbd1008 target="/var/lib/kubelet/pods/c45e681a-598e-47d2-8548-9dc33c84408b/volumes/kubernetes.io~csi/pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/mount"
time="2023-04-17T11:16:20Z" level=error msg="method failed" error="rpc error: code = Internal desc = NodePublishVolume failed for pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: failed to set source device readwrite: exit status 1" linstorCSIComponent=driver method=/csi.v1.Node/NodePublishVolume nodeID=eu-central-68d6 provisioner=linstor.csi.linbit.com req="volume_id:\"pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9\" publish_context:<key:\"Aux/csi-publish-readonly\" value:\"false\" > target_path:\"/var/lib/kubelet/pods/c45e681a-598e-47d2-8548-9dc33c84408b/volumes/kubernetes.io~csi/pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:\"csi.storage.k8s.io/ephemeral\" value:\"false\" > volume_context:<key:\"csi.storage.k8s.io/pod.name\" value:\"main-kafka-1\" > volume_context:<key:\"csi.storage.k8s.io/pod.namespace\" value:\"kafka\" > volume_context:<key:\"csi.storage.k8s.io/pod.uid\" value:\"c45e681a-598e-47d2-8548-9dc33c84408b\" > volume_context:<key:\"csi.storage.k8s.io/serviceAccount.name\" value:\"main-kafka\" > volume_context:<key:\"linstor.csi.linbit.com/mount-options\" value:\"\" > volume_context:<key:\"linstor.csi.linbit.com/post-mount-xfs-opts\" value:\"\" > volume_context:<key:\"linstor.csi.linbit.com/remote-access-policy\" value:\"true\" > volume_context:<key:\"linstor.csi.linbit.com/uses-volume-context\" value:\"true\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1679407629478-8081-linstor.csi.linbit.com\" > " resp="<nil>" version=v1.0.0-3fbda5a7e31517786796ae4834c159caa9ad8ac7

Not sure how to move forward here. I've found a similar error message in #424, but don't believe it's the same issue.

I've generated an SOS report in the current state: sos-report.tar.gz
@WanzenBug
Copy link
Member

Something has kept the resource open on eu-central-1771, as you can see in the linstor volume list. This is not allowed by DRBD, as only one node can access the device at any point in time. I can't find anything related to pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 in the kernel logs, so it probably got promoted some time ago.

You may try kubectl exec eu-central-1771 -- drbdadm secondary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9, which should produce an error message, telling you what kind of process kept the device open. I would guess it's some kind of mount process. You can search for where the device remains mounted using mount | grep drbd1008

@boedy
Copy link
Author

boedy commented Apr 17, 2023 •
edited
Loading

Thanks for the quick reply @WanzenBug! I ran the command you suggested. It returned no errors and caused the volume on node eu-central-68d6 to be promoted to primary automatically. See:

~ linstor r l -r pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9
+---------------------------------------------------------------------------------------------------------------------+
| ResourceName                             | Node            | Port | Usage  | Conns |    State | CreatedOn           |
|=====================================================================================================================|
| pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | eu-central-1771 | 7008 | Unused | Ok    | UpToDate | 2023-03-29 18:00:35 |
| pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | eu-central-42f5 | 7008 | Unused | Ok    | UpToDate | 2023-03-29 18:00:33 |
| pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | eu-central-68d6 | 7008 | InUse  | Ok    | UpToDate | 2023-03-29 18:00:35 |
+---------------------------------------------------------------------------------------------------------------------+

If this should ever happen again I now know how to solve it, but this should have been resolved by the controller right? Are there any other logs which I could check to see what might have caused this?

@WanzenBug
Copy link
Member

That kubectl exec eu-central-1771 -- drbdadm secondary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 suceeded would suggest that something or someone ran drbdadm primary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9, which disables automatic demotion on removing the last writer.

So I suspect some manual inspection took place. Only other possibilty I could think of is LINSTOR itself timed out during resource creation, but somehow the resource was still usable.

I'd take a look at the kernel logs for 1771, grep for pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9. Normally there is some log message indicating on why the resource was promoted to Primary.

@boedy
Copy link
Author

boedy commented Apr 17, 2023

I ran initially ran the kubectl exec eu-central-1771 -- drbdadm secondary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 command twice as I thought I made mistake the first time, so that might show up in the logs.

Here are the results for that pvc. All these logs are from the moment I ran the command it seems.

root@eu-central-1771:/# cat /var/log/kern.log | grep pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.219727] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 853234971 (1->-1 3/2)
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.244095] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 853234971: primary_nodes=0, weak_nodes=0
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.244103] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 853234971 (24ms)
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.244263] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Primary -> Secondary )
Apr 17 11:44:09 eu-central-1771 kernel: [1720101.132850] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 115516091
Apr 17 11:44:09 eu-central-1771 kernel: [1720101.160600] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 115516091 (primary_nodes=4)
Apr 17 11:44:09 eu-central-1771 kernel: [1720101.160616] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: peer( Secondary -> Primary )
Apr 17 11:44:11 eu-central-1771 kernel: [1720103.045400] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 2489766596
Apr 17 11:44:11 eu-central-1771 kernel: [1720103.069434] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 2489766596 (primary_nodes=0)
Apr 17 11:44:11 eu-central-1771 kernel: [1720103.069491] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: peer( Primary -> Secondary )
Apr 17 11:44:12 eu-central-1771 kernel: [1720104.623706] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 1849616096
Apr 17 11:44:12 eu-central-1771 kernel: [1720104.649163] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 1849616096 (primary_nodes=4)
Apr 17 11:44:12 eu-central-1771 kernel: [1720104.649183] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: peer( Secondary -> Primary )

@WanzenBug
Copy link
Member

WanzenBug commented Apr 17, 2023 •
edited
Loading

You might want to check older logs (kern.log.1, kern.log.2, etc..) to find the original role( Secondary -> Primary ) message.

@boedy
Copy link
Author

boedy commented Apr 17, 2023

Ah you're right. Found the relevant logs in kern.log.3

Mar 29 18:00:35 eu-central-1771 kernel: [101113.161400] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Starting worker thread (from drbdsetup [593999])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.163721] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbdsetup [594005])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.165024] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Starting sender thread (from drbdsetup [594008])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.243859] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: meta-data IO uses: blk-bio
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244043] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: susp-io( no -> quorum )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244046] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Diskless -> Attaching )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244057] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: Maximum number of peer devices = 7
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244312] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Method to ensure write ordering: barrier
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244321] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: drbd_bm_resize called with capacity == 104857600
Mar 29 18:00:35 eu-central-1771 kernel: [101113.253005] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: resync bitmap: bits=13107200 words=1433600 pages=2800
Mar 29 18:00:35 eu-central-1771 kernel: [101113.253016] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: size = 50 GB (52428800 KB)
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269051] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: recounting of set bits took additional 8ms
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269072] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Attaching -> UpToDate )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269074] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: attached to current UUID: C8594E29160A6996
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269167] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: size = 50 GB (52428800 KB)
Mar 29 18:00:35 eu-central-1771 kernel: [101113.272023] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( StandAlone -> Unconnected )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.272706] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: conn( StandAlone -> Unconnected )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.272962] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting receiver thread (from drbd_w_pvc-22af [594000])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.273095] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Starting receiver thread (from drbd_w_pvc-22af [594000])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.273104] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.273210] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: conn( Unconnected -> Connecting )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.873810] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 29 18:00:35 eu-central-1771 kernel: [101113.873816] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 29 18:00:35 eu-central-1771 kernel: [101113.897579] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 29 18:00:35 eu-central-1771 kernel: [101113.936240] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Handshake to peer 2 successful: Agreed network protocol version 121
Mar 29 18:00:35 eu-central-1771 kernel: [101113.936246] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 29 18:00:35 eu-central-1771 kernel: [101113.936785] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Peer authenticated using 20 bytes HMAC
Mar 29 18:00:35 eu-central-1771 kernel: [101114.051797] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 2138728852 (1->2 499/146)
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075808] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: drbd_sync_handshake:
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075810] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: self C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075813] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: peer C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075816] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: uuid_compare()=no-sync by rule=both-off
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075827] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 2138728852: primary_nodes=0, weak_nodes=0
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075829] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 2138728852 (24ms)
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075871] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: susp-io( quorum -> no )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075873] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075874] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: quorum( no -> yes )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075876] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( DUnknown -> UpToDate ) repl( Off -> Established )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075977] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: cleared bm UUID and bitmap C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000
Mar 29 18:00:36 eu-central-1771 kernel: [101114.101733] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 1179600745
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115820] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115825] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115829] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:C8594E29160A6996:0000000000000000:0000000000000000 bits:66716 flags:1020
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115833] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=target-use-bitmap by rule=bitmap-peer
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154248] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 1179600745 (primary_nodes=0)
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154270] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154273] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( UpToDate -> Outdated )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154275] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.157310] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 183(1), total 183; compression: 100.0%
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160904] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 183(1), total 183; compression: 100.0%
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160928] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Outdated -> Inconsistent )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160932] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: resync-susp( no -> connection dependency )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160935] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: repl( WFBitMapT -> SyncTarget )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.161158] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Began resync as SyncTarget (will sync 266864 KB [66716 bits set]).
Mar 29 18:00:36 eu-central-1771 kernel: [101114.169779] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 655708152
Mar 29 18:00:36 eu-central-1771 kernel: [101114.213902] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 655708152 (primary_nodes=0)
Mar 29 18:00:36 eu-central-1771 kernel: [101114.214456] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( UpToDate -> Outdated )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.220718] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Outdated -> Inconsistent )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.629729] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Inconsistent -> UpToDate )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735489] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Resync done (total 2 sec; paused 0 sec; 133432 K/sec)
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735500] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: updated UUIDs 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735510] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Inconsistent -> UpToDate )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735512] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: resync-susp( connection dependency -> no )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735514] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: repl( SyncTarget -> Established )
Mar 29 18:00:40 eu-central-1771 kernel: [101118.974835] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 475334800
Mar 29 18:00:40 eu-central-1771 kernel: [101118.999881] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 475334800 (primary_nodes=1)
Mar 29 18:00:40 eu-central-1771 kernel: [101118.999898] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: peer( Secondary -> Primary )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601953] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: meta connection shut down by peer.
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601963] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601965] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( UpToDate -> Consistent )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601967] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.602000] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 tcp:eu-central-42f5: Not enough data received; missing 3716 bytes
Mar 31 20:11:19 eu-central-1771 kernel: [281754.602007] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: error receiving P_DATA, e: -5 l: 4096!
Mar 31 20:11:19 eu-central-1771 kernel: [281754.605439] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:19 eu-central-1771 kernel: [281754.605465] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653704] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653711] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653717] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653721] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653829] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 364178062 (1->-1 0/0)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.158765] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( UpToDate -> Consistent )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.218375] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 364178062: primary_nodes=0, weak_nodes=0
Mar 31 20:11:20 eu-central-1771 kernel: [281755.218379] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 364178062 (564ms)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.218387] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Consistent -> UpToDate )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.219530] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 55078831
Mar 31 20:11:20 eu-central-1771 kernel: [281755.219551] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Aborting remote state change 55078831
Mar 31 20:11:20 eu-central-1771 kernel: [281755.220102] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( Established -> WFBitMapS )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.220110] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Reconciliation resync because 'eu-central-42f5' disappeared. (o=32)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.220116] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Consistent -> Outdated )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.223345] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 53(1), total 53; compression: 100.0%
Mar 31 20:11:20 eu-central-1771 kernel: [281755.226284] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 53(1), total 53; compression: 100.0%
Mar 31 20:11:20 eu-central-1771 kernel: [281755.226297] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.226499] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Began resync as SyncSource (will sync 1856 KB [464 bits set]).
Mar 31 20:11:20 eu-central-1771 kernel: [281755.316606] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Resync done (total 1 sec; paused 0 sec; 1856 K/sec)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.316620] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.338686] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 234327999
Mar 31 20:11:20 eu-central-1771 kernel: [281755.340513] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 234327999 (primary_nodes=0)
Mar 31 20:11:25 eu-central-1771 kernel: [281760.150346] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 2293158436
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168279] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock_recvmsg returned -11
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168292] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168304] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: short read (expected size 8)
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168306] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: timeout while waiting for feature packet
Mar 31 20:11:25 eu-central-1771 kernel: [281760.171705] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:25 eu-central-1771 kernel: [281760.171735] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:25 eu-central-1771 kernel: [281760.221661] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:25 eu-central-1771 kernel: [281760.221675] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:26 eu-central-1771 kernel: [281761.255906] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:26 eu-central-1771 kernel: [281761.852592] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 tcp:eu-central-42f5: initial packet M crossed
Mar 31 20:11:31 eu-central-1771 kernel: [281766.467899] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:31 eu-central-1771 kernel: [281766.467903] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693437] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693544] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> NetworkFailure )
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693608] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693630] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737349] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737360] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737366] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737371] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:35 eu-central-1771 kernel: [281770.453773] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:35 eu-central-1771 kernel: [281770.453776] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:36 eu-central-1771 kernel: [281771.851035] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:36 eu-central-1771 kernel: [281771.973852] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 2027767180 because of state change 2293158436
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987815] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987823] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987828] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987832] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987834] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:39 eu-central-1771 kernel: [281774.249929] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 2908187164 because of state change 2293158436
Mar 31 20:11:39 eu-central-1771 kernel: [281774.809699] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 2622719287 because of state change 2293158436
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831729] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831734] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831738] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831741] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831743] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361450] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock was shut down by peer
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361465] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361561] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361602] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412825] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412833] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412839] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412843] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:43 eu-central-1771 kernel: [281778.339773] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:43 eu-central-1771 kernel: [281778.342297] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:44 eu-central-1771 kernel: [281779.946455] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:46 eu-central-1771 kernel: [281781.541579] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 3516829557 because of state change 2293158436
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563614] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563617] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563622] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563626] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563628] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:48 eu-central-1771 kernel: [281783.797991] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3330022623 because of state change 2293158436
Mar 31 20:11:49 eu-central-1771 kernel: [281784.273554] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 3367399829 because of state change 2293158436
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287591] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287597] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287600] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287603] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287605] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805598] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock was shut down by peer
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805612] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805690] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805708] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:49 eu-central-1771 kernel: [281784.806150] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 1677594074 because of state change 2293158436
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840895] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840907] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840917] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840924] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:50 eu-central-1771 kernel: [281785.314158] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3281136363 because of state change 2293158436
Mar 31 20:11:50 eu-central-1771 kernel: [281785.614903] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3396001944 because of state change 2293158436
Mar 31 20:11:50 eu-central-1771 kernel: [281785.897856] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3100672356 because of state change 2293158436
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251437] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock_recvmsg returned -11
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251453] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251465] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: short read (expected size 8)
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251467] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: timeout while waiting for feature packet
Mar 31 20:11:54 eu-central-1771 kernel: [281789.254285] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:54 eu-central-1771 kernel: [281789.254313] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:54 eu-central-1771 kernel: [281789.288483] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:54 eu-central-1771 kernel: [281789.288493] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:55 eu-central-1771 kernel: [281790.307428] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:55 eu-central-1771 kernel: [281790.781696] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 1128590826 because of state change 2293158436
Mar 31 20:11:55 eu-central-1771 kernel: [281790.881570] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 547725012 because of state change 2293158436
Mar 31 20:11:55 eu-central-1771 kernel: [281790.981877] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 4294843850 because of state change 2293158436
Mar 31 20:11:55 eu-central-1771 kernel: [281791.075624] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 2939194751 because of state change 2293158436
Mar 31 20:11:56 eu-central-1771 kernel: [281791.143391] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Two-phase commit 2293158436 timeout
Mar 31 20:11:56 eu-central-1771 kernel: [281791.185651] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 1137476348
Mar 31 20:11:56 eu-central-1771 kernel: [281791.225790] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 1137476348 (primary_nodes=1)
Mar 31 20:11:56 eu-central-1771 kernel: [281791.225805] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( UpToDate -> Outdated )
Mar 31 20:11:56 eu-central-1771 kernel: [281791.541222] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:56 eu-central-1771 kernel: [281791.541227] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:56 eu-central-1771 kernel: [281791.564915] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:56 eu-central-1771 kernel: [281791.565053] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> NetworkFailure )
Mar 31 20:11:56 eu-central-1771 kernel: [281791.565140] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:56 eu-central-1771 kernel: [281791.565160] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608948] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608961] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608971] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608975] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:57 eu-central-1771 kernel: [281792.249264] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:57 eu-central-1771 kernel: [281792.249270] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:57 eu-central-1771 kernel: [281792.273305] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:57 eu-central-1771 kernel: [281792.306605] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 2678655836 (1->-1 3/1)
Mar 31 20:11:57 eu-central-1771 kernel: [281792.330744] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Aborting cluster-wide state change 2678655836 (24ms) rv = -10
Mar 31 20:11:57 eu-central-1771 kernel: [281792.330801] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Declined by peer eu-central-68d6 (id: 2), see the kernel log there
Mar 31 20:11:57 eu-central-1771 kernel: [281792.355011] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 3878372240
Mar 31 20:12:28 eu-central-1771 kernel: [281823.669385] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3169814138 because of state change 3878372240
Mar 31 20:13:01 eu-central-1771 kernel: [281856.404878] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3856063766 because of state change 3878372240
Mar 31 20:13:34 eu-central-1771 kernel: [281889.172273] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 4201528038 because of state change 3878372240
Mar 31 20:14:06 eu-central-1771 kernel: [281921.939661] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 302133202 because of state change 3878372240
Mar 31 20:14:39 eu-central-1771 kernel: [281954.707140] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 4284797115 because of state change 3878372240
Mar 31 20:33:57 eu-central-1771 kernel: [283112.643635] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> NetworkFailure )
Mar 31 20:33:57 eu-central-1771 kernel: [283112.643781] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change failed: In transient state, retry after next state change
Mar 31 20:33:57 eu-central-1771 kernel: [283112.649571] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Two-phase commit 3878372240 timeout
Mar 31 20:33:57 eu-central-1771 kernel: [283112.652532] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:33:57 eu-central-1771 kernel: [283112.657138] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:33:57 eu-central-1771 kernel: [283112.697768] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 2828581286 (1->-1 3/1)
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698884] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698890] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698904] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698907] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.217978] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Aborting cluster-wide state change 2828581286 (520ms) rv = -19
Mar 31 20:33:58 eu-central-1771 kernel: [283113.218194] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( UpToDate -> Consistent )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219488] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: drbd_sync_handshake:
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219491] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219494] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: peer 9ECC182ED68E217C:39E73F8692AA2282:0000000000000000:0000000000000000 bits:322 flags:1060
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219497] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: uuid_compare()=target-use-bitmap by rule=bitmap-peer
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219499] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Becoming WFBitMapT after unstable
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219505] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( Established -> WFBitMapT )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.283315] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 2500614562
Mar 31 20:33:58 eu-central-1771 kernel: [283113.283755] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 2500614562 (primary_nodes=0)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.284542] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Consistent -> UpToDate )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.287085] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 102(1), total 102; compression: 100.0%
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289502] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 102(1), total 102; compression: 100.0%
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289520] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Outdated -> Inconsistent )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289522] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( WFBitMapT -> SyncTarget )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289524] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: resync-susp( no -> connection dependency )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289801] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Began resync as SyncTarget (will sync 1288 KB [322 bits set]).
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327865] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Resync done (total 1 sec; paused 0 sec; 1288 K/sec)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327874] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: updated UUIDs 9ECC182ED68E217C:0000000000000000:0000000000000000:0000000000000000
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327884] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Inconsistent -> UpToDate )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327886] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( SyncTarget -> Established )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327888] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: resync-susp( connection dependency -> no )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426064] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 205524943 (1->-1 3/1)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426621] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 205524943: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF9
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426624] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 205524943 (0ms)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426629] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Secondary -> Primary )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.428703] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: new current UUID: 02F7C57DF6F0892D weak: FFFFFFFFFFFFFFF9
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434186] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 3382725511 (1->-1 3/2)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434618] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 3382725511: primary_nodes=0, weak_nodes=0
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434625] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 3382725511 (0ms)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434670] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Primary -> Secondary )
Mar 31 20:33:59 eu-central-1771 kernel: [283114.413647] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:33:59 eu-central-1771 kernel: [283114.417354] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:34:00 eu-central-1771 kernel: [283115.627073] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:34:00 eu-central-1771 kernel: [283115.803468] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 4290203759
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837602] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837608] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 02F7C57DF6F0892C:9ECC182ED68E217C:0000000000000000:0000000000000000 bits:322 flags:120
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837612] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 9ECC182ED68E217D:39E73F8692AA2282:0000000000000000:0000000000000000 bits:803 flags:1520
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837615] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=source-use-bitmap by rule=bitmap-self
Mar 31 20:34:02 eu-central-1771 kernel: [283117.645818] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Aborting remote state change 4290203759
Mar 31 20:34:02 eu-central-1771 kernel: [283117.671936] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 3016942592
Mar 31 20:34:02 eu-central-1771 kernel: [283118.053903] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 3016942592 (primary_nodes=0)
Mar 31 20:34:03 eu-central-1771 kernel: [283118.655561] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 3964007541
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669535] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669538] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 02F7C57DF6F0892C:9ECC182ED68E217C:0000000000000000:0000000000000000 bits:322 flags:120
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669543] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 02F7C57DF6F0892C:39E73F8692AA2282:0000000000000000:0000000000000000 bits:803 flags:1924
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669546] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669548] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: strategy = source-set-bitmap due to disk states. (UpToDate/Inconsistent)
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669551] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Setting and writing one bitmap slot, after drbd_sync_handshake
Mar 31 20:34:03 eu-central-1771 kernel: [283118.687235] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: bitmap WRITE of 2800 pages took 16 ms
Mar 31 20:34:04 eu-central-1771 kernel: [283119.167606] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Aborting remote state change 3964007541
Mar 31 20:34:04 eu-central-1771 kernel: [283119.203553] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 750722015
Mar 31 20:34:04 eu-central-1771 kernel: [283119.204220] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 750722015 (primary_nodes=0)
Mar 31 20:34:04 eu-central-1771 kernel: [283119.219501] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 3957309307
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233562] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233568] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 02F7C57DF6F0892C:9ECC182ED68E217C:0000000000000000:0000000000000000 bits:13107200 flags:120
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233573] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 9ECC182ED68E217D:39E73F8692AA2282:0000000000000000:0000000000000000 bits:13107200 flags:1124
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233577] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=source-use-bitmap by rule=bitmap-self
Mar 31 20:34:04 eu-central-1771 kernel: [283119.437449] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 3957309307 (primary_nodes=0)
Mar 31 20:34:04 eu-central-1771 kernel: [283119.437463] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Mar 31 20:34:04 eu-central-1771 kernel: [283119.437466] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS )
Mar 31 20:34:04 eu-central-1771 kernel: [283119.442364] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
Mar 31 20:34:05 eu-central-1771 kernel: [283120.294671] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
Mar 31 20:34:05 eu-central-1771 kernel: [283120.294691] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: repl( WFBitMapS -> SyncSource )
Mar 31 20:34:05 eu-central-1771 kernel: [283120.294963] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Began resync as SyncSource (will sync 52428800 KB [13107200 bits set]).
Mar 31 20:34:10 eu-central-1771 kernel: [283125.405033] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 2838873287
Mar 31 20:34:10 eu-central-1771 kernel: [283125.856391] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 2838873287 (primary_nodes=0)
Mar 31 20:34:40 eu-central-1771 kernel: [283155.557219] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 4115924573 (1->-1 3/1)
Mar 31 20:34:48 eu-central-1771 kernel: [283163.286904] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 4115924573: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF8
Mar 31 20:34:48 eu-central-1771 kernel: [283163.286911] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 4115924573 (7728ms)
Mar 31 20:34:48 eu-central-1771 kernel: [283163.286921] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Secondary -> Primary )
Mar 31 20:49:23 eu-central-1771 kernel: [284039.044227] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: updated UUIDs 02F7C57DF6F0892C:0000000000000000:9ECC182ED68E217C:0000000000000000
Mar 31 20:49:24 eu-central-1771 kernel: [284039.076002] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Resync done (total 918 sec; paused 0 sec; 57108 K/sec)
Mar 31 20:49:24 eu-central-1771 kernel: [284039.076016] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@WanzenBug @boedy