Log abnormal events for resources we are waiting on

[djwhatle]

Overview

• Logs deduplicated events related to a resource we are waiting on.
• The eventReason field is used to perform deduplication.
• Shows the newest event of a particular eventReason.
• If no abnormal events are spotted, nothing will be logged.

Logs abnormal events while waiting on:

• Registry Pods
• Rsync Pods
• Stunnel Pods
• Rsync Routes
• MigHook Jobs
• Stage Pods

Logs look like this.

{
  "level": "info",
  "ts": 1617296522250,
  "logger": "directvolume|gr7lc",
  "msg": "Found abnormal event for Rsync transfer Pod on destination cluster",
  "dvm": "a95b82a0-92fd-11eb-876e-1f4707652b42-j5fxj",
  "migMigration": "a95b82a0-92fd-11eb-876e-1f4707652b42",
  "phase": "WaitForRsyncTransferPodsRunning",
  "pod": "mssql-persistent/directvolumemigration-rsync-transfer",
  "eventType": "Warning",
  "eventReason": "FailedAttachVolume",
  "eventMessage": "(combined from similar events): AttachVolume.Attach failed for volume \"pvc-0f061633-a6ef-4604-9020-9b06cb89a660\" : InvalidVolume.NotFound: The volume 'vol-0412f5c8cb6538964' does not exist.\n\tstatus code: 400, request id: efddaa59-a586-4f39-9334-1648afc60e05",
  "eventFirstTimestamp": "2021-04-01 11:24:57 -0400 EDT"
}
{
  "level": "info",
  "ts": 1617296522250,
  "logger": "directvolume|gr7lc",
  "msg": "Found abnormal event for Rsync transfer Pod on destination cluster",
  "dvm": "a95b82a0-92fd-11eb-876e-1f4707652b42-j5fxj",
  "migMigration": "a95b82a0-92fd-11eb-876e-1f4707652b42",
  "phase": "WaitForRsyncTransferPodsRunning",
  "pod": "mssql-persistent/directvolumemigration-rsync-transfer",
  "eventType": "Warning",
  "eventReason": "FailedMount",
  "eventMessage": "Unable to attach or mount volumes: unmounted volumes=[mssql-pvc], unattached volumes=[stunnel-conf stunnel-certs mssql-pvc rsyncd-conf rsync-creds default-token-79xdn]: timed out waiting for the condition",
  "eventFirstTimestamp": "2021-04-01 11:50:07 -0400 EDT"
}

Closes #1051

[pranavgaikwad]

VISIACK