Annotate PVC with errors from Controller

[phenixblue]

Need to add functionality to the controller to update target PVC resources when certain errors occur within the reconciliation.

At a minimum for when an incorrect volume retention policy is specified.

Example:

When labeling a PVC with a retention policy that isn't Retain, Delete, or Recycle the controller logs the following errors:

2021-08-17T20:28:05.659Z	INFO	controllers.PersistentVolumeClaim	Setting reclaim policy to match PVC label	{"pvc": "test2/datadir-crdb1-2", "pv": "pvc-70a047e8-bc55-4ff8-b389-76cd07440f3d", "policy-from-pvc-label": "Dummy", "policy-from-pv": "Delete"}
2021-08-17T20:28:05.663Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "persistentvolumeclaim", "request": "test2/datadir-crdb1-2", "error": "could not update PV: PersistentVolume \"pvc-70a047e8-bc55-4ff8-b389-76cd07440f3d\" is invalid: spec.persistentVolumeReclaimPolicy: Unsupported value: \"Dummy\": supported values: \"Delete\", \"Recycle\", \"Retain\""}

Information about the error could be added to an annotation on the target PVC resource to provide a feedback mechanism to end-users who probably don't have access to view the volrec controller logs.

[phenixblue]

something like storage.k8s.twr.io/volrec-errors might be a good annotation key to use.

These should also probably be cleaned up on the first successful reconcile following the initial error.

[phenixblue]

Maybe also write a Kubernetes event to the target namespace as well. I feel like the more methods of feedback the better. In cases where there's a simple typo in the reclaim policy (ie. Retains instead of Retain) a user could inadvertently think the underlying volume would be retained, and it could actually be deleted.

[phenixblue]

This could be made generic to just reflect the status overall to let a user know if the labeling was successful or not.

[phenixblue]

Could also add a MagTape policy to warn users directly (using warning subsystem from Admission Controllers on k8s 1.19+)