Skip to content
This repository has been archived by the owner on Jan 28, 2020. It is now read-only.

API Server crashlooping because it can't find etcd #94

Open
csrwng opened this issue Feb 14, 2018 · 7 comments
Open

API Server crashlooping because it can't find etcd #94

csrwng opened this issue Feb 14, 2018 · 7 comments

Comments

@csrwng
Copy link
Contributor

csrwng commented Feb 14, 2018

After running on my cluster for a couple of hours, I started seeing the api server crash loop. Here's a log from a run:

❯ oc logs -f cluster-operator-apiserver-5fd88df5b9-jm7l5 -c apiserver -n cluster-operator
I0214 20:44:46.929434       1 server.go:59] Preparing to run API server
I0214 20:44:47.100606       1 round_trippers.go:417] curl -k -v -XGET  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLW9wZXJhdG9yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyLXRva2VuLTlxbmpuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2VhZWExMTktMTFiZi0xMWU4LWI1MmItNGUyZmI2YmFjYWE3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItb3BlcmF0b3I6Y2x1c3Rlci1vcGVyYXRvci1hcGlzZXJ2ZXIifQ.Q0bHyN-_FFpUTvjxO2mBDujZj7Twi8NIF_s6eLCfq1f8YM29wnbY2grlRBfzoK-2OyGtRGgSXQMdD9BcviruL8SkHLSz1GltVBb-IGMaT44FWek1E0LU3hQzKdFQNSqu-rxI0GX2RCk6swtjeoNJsVZ0mh-SqfkmhaW2di5o817lQvzghoUbBioSEHZyMvidgx9UqeWbG0TKXsTjI3d_VO8CvvRPwDlNeOZ4YSAXTfIiJLFTWvrI3dfj7PR7FkKnVE535rrzMwVqiGaj7v1v8KCPBwgHC_MroyWsS9AexkL3dFN9tSHOLN6W_zTIPP5sf8hvWmHfNYkbysJSTJRgqw" -H "User-Agent: cluster-operator/v0.0.0 (linux/amd64) kubernetes/790d284" -H "Accept: application/json, */*" https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
I0214 20:44:47.111407       1 round_trippers.go:436] GET https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 10 milliseconds
I0214 20:44:47.111433       1 round_trippers.go:442] Response Headers:
I0214 20:44:47.111437       1 round_trippers.go:445]     Cache-Control: no-store
I0214 20:44:47.111440       1 round_trippers.go:445]     Content-Type: application/json
I0214 20:44:47.111442       1 round_trippers.go:445]     Content-Length: 2690
I0214 20:44:47.111444       1 round_trippers.go:445]     Date: Wed, 14 Feb 2018 20:44:47 GMT
I0214 20:44:47.111648       1 request.go:873] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"18fe5de7-11bf-11e8-b52b-4e2fb6bacaa7","resourceVersion":"69","creationTimestamp":"2018-02-14T19:41:49Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMwHhcNMTgwMjE0MTkyNzAyWhcNMjMwMjEz\nMTkyNzAzWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuWb2ZHhjbBPrFaJ7Hi1sO\nzYx/u1477bI92MX7ZcL/Kx3/huXj4RkbE+TBjCkpO6xTfjI5tWc0+5jXkc2lt2cP\n1YYJdtP9LWfNVg0TN0HU3nqaBR2OGtkuzqXYYfUfKJNU1e6Kg8zh3xi5BoI1LwNM\np5slISyjJR76FAwWhlcx9fRZOS324EOQBujx0ZuH1qwXfXsrt80oMMZWMGDTMmEt\nr0Kd6WODuYow9KbqouQrCbQdv9RCh9OGBtSRh8WivKq+BntaCZVWbFE8qEGg7shd\ncGTpgW4idHOUSFx49EOqTp4cVDEctUWp9MR6L5TqId7alHwFAHUrOnl7NXcCdyzR\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQBTusNrTl/ba/4ZPGaaYBLmzTx5H48JSCugmnWKIeI8JnnJ\nD+XwCqMT8lEPXeveY/dstNBITl3LnVT8+fpAMNsB/bSnb+qirPtr9RzNgFHr2N3M\nUOCNNBYOJ0Yj8tkKum6I3rdsZh0WRRU9SNBmpUmHoAQGCJes1m9+OOEkPnhUi0pI\nw38pq/FjILyyWYH9p+wOgAOheqVs/KFLxaVi5n0fwyaF10Bf8pUFdcon4rzWFH1c\nk0AsPWqftOU2I+p6rvP553gW2XrnfKC/03CiT4fFf5VGWWBPGsiuZmxRmZhgDqjE\nH352IP0AxodUjkJdvvR4GDji7+tvy4BI/cMktyt1\n-----END CERTIFICATE-----\n","requestheader-allowed-names":"[\"aggregator-front-proxy\"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIICnjCCAYagAwIBAgIBATANBgkqhkiG9w0BAQsFADAAMB4XDTE4MDIxNDE5NDEz\nN1oXDTIzMDIxMzE5NDEzOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAM9iOcCuPDdL4+vToMbIAyWz6FayEL2vcCw9q0TkRN+0B0yt1NWG3M+eeS+s\nmOIj8Wb8/+Z4/Thejrza20QjmLdMraV88BfdXbTb4HnsiVTk1e7c7QbFP7YZZtQ1\n0wz2jtB1+uPEJaC+LfZmJv2mb89WhFwOhuTiTj4NzDvvnDsm1vL9aerdXCH7ZnvZ\nTKlLnl4HdKH4Q6WhMro2HB792tZGoZq7ZBSDRYCGVhhW6Sg10Id5Qc2FP1X2duCW\nYiiN00jWjC8G6UucZUdcspUAQz9z4ZCE9Zjm7pvc2LPRLraireYlyoEMCj5nFyF1\n5sONYVqBvNSPVGK9muHcAoehOgsCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8G\nA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACsWEX1YGTjbqg9kO0a5\naa38BoagLkygO+/qw6b2cByeGvdM9vGUn5j5LWmIKIT3TVNy7pA2EtpVtw1CUdB5\nGO7O4KeYJ/uxW/9tRYf7Uzokkd0iEwd5RY1bhoZBb1hergQHYsBMgf9jKusfPLH+\nI6fDYZEW5jWKkd6BRNW/XyW5RSEUf6Sh59ZTjNhdbTFjOsuoDMLrARGilP/qYav0\nKIQ1wAostR4TFtEZJ/Kf6Z1ufQVDpZmx6IGyZECEHDIYLPvsK9PwAcYxd7sbHGO4\nfHdy1MD/VHLWZhZ6q5UhylmLGrdyxqimWNXSy93lrdCpBRSAlDpkCygQp+015idA\ngH8=\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
I0214 20:44:47.113393       1 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: cluster-operator/v0.0.0 (linux/amd64) kubernetes/790d284" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLW9wZXJhdG9yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyLXRva2VuLTlxbmpuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2VhZWExMTktMTFiZi0xMWU4LWI1MmItNGUyZmI2YmFjYWE3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItb3BlcmF0b3I6Y2x1c3Rlci1vcGVyYXRvci1hcGlzZXJ2ZXIifQ.Q0bHyN-_FFpUTvjxO2mBDujZj7Twi8NIF_s6eLCfq1f8YM29wnbY2grlRBfzoK-2OyGtRGgSXQMdD9BcviruL8SkHLSz1GltVBb-IGMaT44FWek1E0LU3hQzKdFQNSqu-rxI0GX2RCk6swtjeoNJsVZ0mh-SqfkmhaW2di5o817lQvzghoUbBioSEHZyMvidgx9UqeWbG0TKXsTjI3d_VO8CvvRPwDlNeOZ4YSAXTfIiJLFTWvrI3dfj7PR7FkKnVE535rrzMwVqiGaj7v1v8KCPBwgHC_MroyWsS9AexkL3dFN9tSHOLN6W_zTIPP5sf8hvWmHfNYkbysJSTJRgqw" https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
I0214 20:44:47.114585       1 round_trippers.go:436] GET https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0214 20:44:47.114606       1 round_trippers.go:442] Response Headers:
I0214 20:44:47.114609       1 round_trippers.go:445]     Cache-Control: no-store
I0214 20:44:47.114611       1 round_trippers.go:445]     Content-Type: application/json
I0214 20:44:47.114613       1 round_trippers.go:445]     Content-Length: 2690
I0214 20:44:47.114615       1 round_trippers.go:445]     Date: Wed, 14 Feb 2018 20:44:47 GMT
I0214 20:44:47.114644       1 request.go:873] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"18fe5de7-11bf-11e8-b52b-4e2fb6bacaa7","resourceVersion":"69","creationTimestamp":"2018-02-14T19:41:49Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMwHhcNMTgwMjE0MTkyNzAyWhcNMjMwMjEz\nMTkyNzAzWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuWb2ZHhjbBPrFaJ7Hi1sO\nzYx/u1477bI92MX7ZcL/Kx3/huXj4RkbE+TBjCkpO6xTfjI5tWc0+5jXkc2lt2cP\n1YYJdtP9LWfNVg0TN0HU3nqaBR2OGtkuzqXYYfUfKJNU1e6Kg8zh3xi5BoI1LwNM\np5slISyjJR76FAwWhlcx9fRZOS324EOQBujx0ZuH1qwXfXsrt80oMMZWMGDTMmEt\nr0Kd6WODuYow9KbqouQrCbQdv9RCh9OGBtSRh8WivKq+BntaCZVWbFE8qEGg7shd\ncGTpgW4idHOUSFx49EOqTp4cVDEctUWp9MR6L5TqId7alHwFAHUrOnl7NXcCdyzR\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQBTusNrTl/ba/4ZPGaaYBLmzTx5H48JSCugmnWKIeI8JnnJ\nD+XwCqMT8lEPXeveY/dstNBITl3LnVT8+fpAMNsB/bSnb+qirPtr9RzNgFHr2N3M\nUOCNNBYOJ0Yj8tkKum6I3rdsZh0WRRU9SNBmpUmHoAQGCJes1m9+OOEkPnhUi0pI\nw38pq/FjILyyWYH9p+wOgAOheqVs/KFLxaVi5n0fwyaF10Bf8pUFdcon4rzWFH1c\nk0AsPWqftOU2I+p6rvP553gW2XrnfKC/03CiT4fFf5VGWWBPGsiuZmxRmZhgDqjE\nH352IP0AxodUjkJdvvR4GDji7+tvy4BI/cMktyt1\n-----END CERTIFICATE-----\n","requestheader-allowed-names":"[\"aggregator-front-proxy\"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIICnjCCAYagAwIBAgIBATANBgkqhkiG9w0BAQsFADAAMB4XDTE4MDIxNDE5NDEz\nN1oXDTIzMDIxMzE5NDEzOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAM9iOcCuPDdL4+vToMbIAyWz6FayEL2vcCw9q0TkRN+0B0yt1NWG3M+eeS+s\nmOIj8Wb8/+Z4/Thejrza20QjmLdMraV88BfdXbTb4HnsiVTk1e7c7QbFP7YZZtQ1\n0wz2jtB1+uPEJaC+LfZmJv2mb89WhFwOhuTiTj4NzDvvnDsm1vL9aerdXCH7ZnvZ\nTKlLnl4HdKH4Q6WhMro2HB792tZGoZq7ZBSDRYCGVhhW6Sg10Id5Qc2FP1X2duCW\nYiiN00jWjC8G6UucZUdcspUAQz9z4ZCE9Zjm7pvc2LPRLraireYlyoEMCj5nFyF1\n5sONYVqBvNSPVGK9muHcAoehOgsCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8G\nA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACsWEX1YGTjbqg9kO0a5\naa38BoagLkygO+/qw6b2cByeGvdM9vGUn5j5LWmIKIT3TVNy7pA2EtpVtw1CUdB5\nGO7O4KeYJ/uxW/9tRYf7Uzokkd0iEwd5RY1bhoZBb1hergQHYsBMgf9jKusfPLH+\nI6fDYZEW5jWKkd6BRNW/XyW5RSEUf6Sh59ZTjNhdbTFjOsuoDMLrARGilP/qYav0\nKIQ1wAostR4TFtEZJ/Kf6Z1ufQVDpZmx6IGyZECEHDIYLPvsK9PwAcYxd7sbHGO4\nfHdy1MD/VHLWZhZ6q5UhylmLGrdyxqimWNXSy93lrdCpBRSAlDpkCygQp+015idA\ngH8=\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
I0214 20:44:47.115175       1 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: cluster-operator/v0.0.0 (linux/amd64) kubernetes/790d284" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLW9wZXJhdG9yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyLXRva2VuLTlxbmpuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2VhZWExMTktMTFiZi0xMWU4LWI1MmItNGUyZmI2YmFjYWE3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItb3BlcmF0b3I6Y2x1c3Rlci1vcGVyYXRvci1hcGlzZXJ2ZXIifQ.Q0bHyN-_FFpUTvjxO2mBDujZj7Twi8NIF_s6eLCfq1f8YM29wnbY2grlRBfzoK-2OyGtRGgSXQMdD9BcviruL8SkHLSz1GltVBb-IGMaT44FWek1E0LU3hQzKdFQNSqu-rxI0GX2RCk6swtjeoNJsVZ0mh-SqfkmhaW2di5o817lQvzghoUbBioSEHZyMvidgx9UqeWbG0TKXsTjI3d_VO8CvvRPwDlNeOZ4YSAXTfIiJLFTWvrI3dfj7PR7FkKnVE535rrzMwVqiGaj7v1v8KCPBwgHC_MroyWsS9AexkL3dFN9tSHOLN6W_zTIPP5sf8hvWmHfNYkbysJSTJRgqw" https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
I0214 20:44:47.116303       1 round_trippers.go:436] GET https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0214 20:44:47.116325       1 round_trippers.go:442] Response Headers:
I0214 20:44:47.116328       1 round_trippers.go:445]     Cache-Control: no-store
I0214 20:44:47.116330       1 round_trippers.go:445]     Content-Type: application/json
I0214 20:44:47.116332       1 round_trippers.go:445]     Content-Length: 2690
I0214 20:44:47.116334       1 round_trippers.go:445]     Date: Wed, 14 Feb 2018 20:44:47 GMT
I0214 20:44:47.116358       1 request.go:873] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"18fe5de7-11bf-11e8-b52b-4e2fb6bacaa7","resourceVersion":"69","creationTimestamp":"2018-02-14T19:41:49Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMwHhcNMTgwMjE0MTkyNzAyWhcNMjMwMjEz\nMTkyNzAzWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuWb2ZHhjbBPrFaJ7Hi1sO\nzYx/u1477bI92MX7ZcL/Kx3/huXj4RkbE+TBjCkpO6xTfjI5tWc0+5jXkc2lt2cP\n1YYJdtP9LWfNVg0TN0HU3nqaBR2OGtkuzqXYYfUfKJNU1e6Kg8zh3xi5BoI1LwNM\np5slISyjJR76FAwWhlcx9fRZOS324EOQBujx0ZuH1qwXfXsrt80oMMZWMGDTMmEt\nr0Kd6WODuYow9KbqouQrCbQdv9RCh9OGBtSRh8WivKq+BntaCZVWbFE8qEGg7shd\ncGTpgW4idHOUSFx49EOqTp4cVDEctUWp9MR6L5TqId7alHwFAHUrOnl7NXcCdyzR\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQBTusNrTl/ba/4ZPGaaYBLmzTx5H48JSCugmnWKIeI8JnnJ\nD+XwCqMT8lEPXeveY/dstNBITl3LnVT8+fpAMNsB/bSnb+qirPtr9RzNgFHr2N3M\nUOCNNBYOJ0Yj8tkKum6I3rdsZh0WRRU9SNBmpUmHoAQGCJes1m9+OOEkPnhUi0pI\nw38pq/FjILyyWYH9p+wOgAOheqVs/KFLxaVi5n0fwyaF10Bf8pUFdcon4rzWFH1c\nk0AsPWqftOU2I+p6rvP553gW2XrnfKC/03CiT4fFf5VGWWBPGsiuZmxRmZhgDqjE\nH352IP0AxodUjkJdvvR4GDji7+tvy4BI/cMktyt1\n-----END CERTIFICATE-----\n","requestheader-allowed-names":"[\"aggregator-front-proxy\"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIICnjCCAYagAwIBAgIBATANBgkqhkiG9w0BAQsFADAAMB4XDTE4MDIxNDE5NDEz\nN1oXDTIzMDIxMzE5NDEzOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAM9iOcCuPDdL4+vToMbIAyWz6FayEL2vcCw9q0TkRN+0B0yt1NWG3M+eeS+s\nmOIj8Wb8/+Z4/Thejrza20QjmLdMraV88BfdXbTb4HnsiVTk1e7c7QbFP7YZZtQ1\n0wz2jtB1+uPEJaC+LfZmJv2mb89WhFwOhuTiTj4NzDvvnDsm1vL9aerdXCH7ZnvZ\nTKlLnl4HdKH4Q6WhMro2HB792tZGoZq7ZBSDRYCGVhhW6Sg10Id5Qc2FP1X2duCW\nYiiN00jWjC8G6UucZUdcspUAQz9z4ZCE9Zjm7pvc2LPRLraireYlyoEMCj5nFyF1\n5sONYVqBvNSPVGK9muHcAoehOgsCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8G\nA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACsWEX1YGTjbqg9kO0a5\naa38BoagLkygO+/qw6b2cByeGvdM9vGUn5j5LWmIKIT3TVNy7pA2EtpVtw1CUdB5\nGO7O4KeYJ/uxW/9tRYf7Uzokkd0iEwd5RY1bhoZBb1hergQHYsBMgf9jKusfPLH+\nI6fDYZEW5jWKkd6BRNW/XyW5RSEUf6Sh59ZTjNhdbTFjOsuoDMLrARGilP/qYav0\nKIQ1wAostR4TFtEZJ/Kf6Z1ufQVDpZmx6IGyZECEHDIYLPvsK9PwAcYxd7sbHGO4\nfHdy1MD/VHLWZhZ6q5UhylmLGrdyxqimWNXSy93lrdCpBRSAlDpkCygQp+015idA\ngH8=\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
I0214 20:44:47.116859       1 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: cluster-operator/v0.0.0 (linux/amd64) kubernetes/790d284" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLW9wZXJhdG9yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyLXRva2VuLTlxbmpuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2VhZWExMTktMTFiZi0xMWU4LWI1MmItNGUyZmI2YmFjYWE3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItb3BlcmF0b3I6Y2x1c3Rlci1vcGVyYXRvci1hcGlzZXJ2ZXIifQ.Q0bHyN-_FFpUTvjxO2mBDujZj7Twi8NIF_s6eLCfq1f8YM29wnbY2grlRBfzoK-2OyGtRGgSXQMdD9BcviruL8SkHLSz1GltVBb-IGMaT44FWek1E0LU3hQzKdFQNSqu-rxI0GX2RCk6swtjeoNJsVZ0mh-SqfkmhaW2di5o817lQvzghoUbBioSEHZyMvidgx9UqeWbG0TKXsTjI3d_VO8CvvRPwDlNeOZ4YSAXTfIiJLFTWvrI3dfj7PR7FkKnVE535rrzMwVqiGaj7v1v8KCPBwgHC_MroyWsS9AexkL3dFN9tSHOLN6W_zTIPP5sf8hvWmHfNYkbysJSTJRgqw" https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
I0214 20:44:47.118065       1 round_trippers.go:436] GET https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 1 milliseconds
I0214 20:44:47.118088       1 round_trippers.go:442] Response Headers:
I0214 20:44:47.118092       1 round_trippers.go:445]     Cache-Control: no-store
I0214 20:44:47.118094       1 round_trippers.go:445]     Content-Type: application/json
I0214 20:44:47.118097       1 round_trippers.go:445]     Content-Length: 2690
I0214 20:44:47.118099       1 round_trippers.go:445]     Date: Wed, 14 Feb 2018 20:44:47 GMT
I0214 20:44:47.118153       1 request.go:873] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"18fe5de7-11bf-11e8-b52b-4e2fb6bacaa7","resourceVersion":"69","creationTimestamp":"2018-02-14T19:41:49Z"},"data":{"client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMwHhcNMTgwMjE0MTkyNzAyWhcNMjMwMjEz\nMTkyNzAzWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTg2MzY0MjMw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCuWb2ZHhjbBPrFaJ7Hi1sO\nzYx/u1477bI92MX7ZcL/Kx3/huXj4RkbE+TBjCkpO6xTfjI5tWc0+5jXkc2lt2cP\n1YYJdtP9LWfNVg0TN0HU3nqaBR2OGtkuzqXYYfUfKJNU1e6Kg8zh3xi5BoI1LwNM\np5slISyjJR76FAwWhlcx9fRZOS324EOQBujx0ZuH1qwXfXsrt80oMMZWMGDTMmEt\nr0Kd6WODuYow9KbqouQrCbQdv9RCh9OGBtSRh8WivKq+BntaCZVWbFE8qEGg7shd\ncGTpgW4idHOUSFx49EOqTp4cVDEctUWp9MR6L5TqId7alHwFAHUrOnl7NXcCdyzR\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQBTusNrTl/ba/4ZPGaaYBLmzTx5H48JSCugmnWKIeI8JnnJ\nD+XwCqMT8lEPXeveY/dstNBITl3LnVT8+fpAMNsB/bSnb+qirPtr9RzNgFHr2N3M\nUOCNNBYOJ0Yj8tkKum6I3rdsZh0WRRU9SNBmpUmHoAQGCJes1m9+OOEkPnhUi0pI\nw38pq/FjILyyWYH9p+wOgAOheqVs/KFLxaVi5n0fwyaF10Bf8pUFdcon4rzWFH1c\nk0AsPWqftOU2I+p6rvP553gW2XrnfKC/03CiT4fFf5VGWWBPGsiuZmxRmZhgDqjE\nH352IP0AxodUjkJdvvR4GDji7+tvy4BI/cMktyt1\n-----END CERTIFICATE-----\n","requestheader-allowed-names":"[\"aggregator-front-proxy\"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIICnjCCAYagAwIBAgIBATANBgkqhkiG9w0BAQsFADAAMB4XDTE4MDIxNDE5NDEz\nN1oXDTIzMDIxMzE5NDEzOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\nggEBAM9iOcCuPDdL4+vToMbIAyWz6FayEL2vcCw9q0TkRN+0B0yt1NWG3M+eeS+s\nmOIj8Wb8/+Z4/Thejrza20QjmLdMraV88BfdXbTb4HnsiVTk1e7c7QbFP7YZZtQ1\n0wz2jtB1+uPEJaC+LfZmJv2mb89WhFwOhuTiTj4NzDvvnDsm1vL9aerdXCH7ZnvZ\nTKlLnl4HdKH4Q6WhMro2HB792tZGoZq7ZBSDRYCGVhhW6Sg10Id5Qc2FP1X2duCW\nYiiN00jWjC8G6UucZUdcspUAQz9z4ZCE9Zjm7pvc2LPRLraireYlyoEMCj5nFyF1\n5sONYVqBvNSPVGK9muHcAoehOgsCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8G\nA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACsWEX1YGTjbqg9kO0a5\naa38BoagLkygO+/qw6b2cByeGvdM9vGUn5j5LWmIKIT3TVNy7pA2EtpVtw1CUdB5\nGO7O4KeYJ/uxW/9tRYf7Uzokkd0iEwd5RY1bhoZBb1hergQHYsBMgf9jKusfPLH+\nI6fDYZEW5jWKkd6BRNW/XyW5RSEUf6Sh59ZTjNhdbTFjOsuoDMLrARGilP/qYav0\nKIQ1wAostR4TFtEZJ/Kf6Z1ufQVDpZmx6IGyZECEHDIYLPvsK9PwAcYxd7sbHGO4\nfHdy1MD/VHLWZhZ6q5UhylmLGrdyxqimWNXSy93lrdCpBRSAlDpkCygQp+015idA\ngH8=\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
I0214 20:44:47.120872       1 util.go:152] Admission control plugin names: []
I0214 20:44:47.120899       1 server.go:65] Creating storage factory
I0214 20:44:47.120926       1 server.go:103] Completing API server configuration
I0214 20:44:47.121607       1 etcd_config.go:88] Created skeleton API server
I0214 20:44:47.121627       1 etcd_config.go:99] Installing API groups
I0214 20:44:47.121643       1 storage_factory.go:278] storing {clusteroperator.openshift.io clusters} in clusteroperator.openshift.io/v1alpha1, reading as clusteroperator.openshift.io/__internal from storagebackend.Config{Type:"", Prefix:"/clusteroperator", ServerList:[]string{"http://localhost:2379"}, KeyFile:"", CertFile:"", CAFile:"", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000}
I0214 20:44:47.121724       1 storage_factory.go:278] storing {clusteroperator.openshift.io clusterversions} in clusteroperator.openshift.io/v1alpha1, reading as clusteroperator.openshift.io/__internal from storagebackend.Config{Type:"", Prefix:"/clusteroperator", ServerList:[]string{"http://localhost:2379"}, KeyFile:"", CertFile:"", CAFile:"", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000}
I0214 20:44:47.121763       1 compact.go:54] compactor already exists for endpoints [http://localhost:2379]
I0214 20:44:47.121786       1 storage_factory.go:278] storing {clusteroperator.openshift.io machinesets} in clusteroperator.openshift.io/v1alpha1, reading as clusteroperator.openshift.io/__internal from storagebackend.Config{Type:"", Prefix:"/clusteroperator", ServerList:[]string{"http://localhost:2379"}, KeyFile:"", CertFile:"", CAFile:"", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000}
I0214 20:44:47.121825       1 compact.go:54] compactor already exists for endpoints [http://localhost:2379]
I0214 20:44:47.121847       1 storage_factory.go:278] storing {clusteroperator.openshift.io machines} in clusteroperator.openshift.io/v1alpha1, reading as clusteroperator.openshift.io/__internal from storagebackend.Config{Type:"", Prefix:"/clusteroperator", ServerList:[]string{"http://localhost:2379"}, KeyFile:"", CertFile:"", CAFile:"", Quorum:true, Paging:true, DeserializationCacheSize:0, Codec:runtime.Codec(nil), Transformer:value.Transformer(nil), CompactionInterval:300000000000}
I0214 20:44:47.121866       1 compact.go:54] compactor already exists for endpoints [http://localhost:2379]
I0214 20:44:47.121874       1 etcd_config.go:111] Installing API group clusteroperator.openshift.io
I0214 20:44:47.123586       1 etcd_config.go:132] Finished installing API groups
I0214 20:44:47.123610       1 server.go:52] Running the API server
[restful] 2018/02/14 20:44:47 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/02/14 20:44:47 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0214 20:44:47.127659       1 healthz.go:74] Installing healthz checkers:"ping", "poststarthook/generic-apiserver-start-informers", "poststarthook/start-cluster-operator-apiserver-informers", "etcd"
I0214 20:44:47.127974       1 serve.go:89] Serving securely on [::]:6443
I0214 20:44:47.128011       1 util.go:166] Starting shared informers
I0214 20:44:47.128017       1 util.go:171] Started shared informers
I0214 20:44:47.132453       1 request.go:873] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0214 20:44:47.132514       1 round_trippers.go:417] curl -k -v -XPOST  -H "User-Agent: cluster-operator/v0.0.0 (linux/amd64) kubernetes/790d284" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLW9wZXJhdG9yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyLXRva2VuLTlxbmpuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2VhZWExMTktMTFiZi0xMWU4LWI1MmItNGUyZmI2YmFjYWE3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItb3BlcmF0b3I6Y2x1c3Rlci1vcGVyYXRvci1hcGlzZXJ2ZXIifQ.Q0bHyN-_FFpUTvjxO2mBDujZj7Twi8NIF_s6eLCfq1f8YM29wnbY2grlRBfzoK-2OyGtRGgSXQMdD9BcviruL8SkHLSz1GltVBb-IGMaT44FWek1E0LU3hQzKdFQNSqu-rxI0GX2RCk6swtjeoNJsVZ0mh-SqfkmhaW2di5o817lQvzghoUbBioSEHZyMvidgx9UqeWbG0TKXsTjI3d_VO8CvvRPwDlNeOZ4YSAXTfIiJLFTWvrI3dfj7PR7FkKnVE535rrzMwVqiGaj7v1v8KCPBwgHC_MroyWsS9AexkL3dFN9tSHOLN6W_zTIPP5sf8hvWmHfNYkbysJSTJRgqw" -H "Accept: application/json, */*" -H "Content-Type: application/json" https://172.30.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0214 20:44:47.133444       1 round_trippers.go:436] POST https://172.30.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 0 milliseconds
I0214 20:44:47.133466       1 round_trippers.go:442] Response Headers:
I0214 20:44:47.133469       1 round_trippers.go:445]     Cache-Control: no-store
I0214 20:44:47.133472       1 round_trippers.go:445]     Content-Type: application/json
I0214 20:44:47.133474       1 round_trippers.go:445]     Content-Length: 301
I0214 20:44:47.133476       1 round_trippers.go:445]     Date: Wed, 14 Feb 2018 20:44:47 GMT
I0214 20:44:47.133488       1 request.go:873] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"allowed by cluster rule"}}
I0214 20:44:47.133941       1 handler.go:160] cluster-operator-apiserver: GET "/healthz" satisfied by nonGoRestful
I0214 20:44:47.133964       1 pathrecorder.go:240] cluster-operator-apiserver: "/healthz" satisfied by exact match
I0214 20:44:47.133972       1 server.go:132] etcd checker called
E0214 20:44:47.134174       1 server.go:141] etcd failed to reach any server
I0214 20:44:47.134193       1 healthz.go:112] healthz check etcd failed: etcd failed to reach any server
I0214 20:44:47.134299       1 wrap.go:42] GET /healthz: (2.127893ms) 500
goroutine 146 [running]:
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4202ee700, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4202ee700, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42021d7a0, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
net/http.Error(0x7f54d3ddc3e0, 0xc420106008, 0xc420788540, 0xb4, 0x1f4)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1930 +0xda
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508
net/http.HandlerFunc.ServeHTTP(0xc42073b160, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42075e940, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x190f857, 0x1a, 0xc42011a6c0, 0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4206afdc0, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	<autogenerated>:1 +0x75
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc420216640, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318
net/http.HandlerFunc.ServeHTTP(0xc420055a00, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc420216690, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc4202166e0, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4206afde0, 0x7f54d3ddc3e0, 0xc420106008, 0xc4204c4700)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4206afe60, 0x24b4cc0, 0xc420106008, 0xc4204c4700, 0xc420346420)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "[+]ping ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-cluster-operator-apiserver-informers ok\n[-]etcd failed: reason withheld\nhealthz check failed\n"
 [[kube-probe/.] 172.17.0.1:45424]
I0214 20:44:56.097572       1 handler.go:160] cluster-operator-apiserver: GET "/healthz" satisfied by nonGoRestful
I0214 20:44:56.097607       1 pathrecorder.go:240] cluster-operator-apiserver: "/healthz" satisfied by exact match
I0214 20:44:56.097622       1 server.go:132] etcd checker called
E0214 20:44:56.098034       1 server.go:141] etcd failed to reach any server
I0214 20:44:56.098083       1 healthz.go:112] healthz check etcd failed: etcd failed to reach any server
I0214 20:44:56.098253       1 wrap.go:42] GET /healthz: (768.943µs) 500
goroutine 201 [running]:
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420438070, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420438070, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4200df300, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
net/http.Error(0x7f54d3ddc3e0, 0xc420764098, 0xc4208460c0, 0xb4, 0x1f4)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1930 +0xda
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508
net/http.HandlerFunc.ServeHTTP(0xc42073b160, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42075e940, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x190f857, 0x1a, 0xc42011a6c0, 0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4206afdc0, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	<autogenerated>:1 +0x75
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc420216640, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318
net/http.HandlerFunc.ServeHTTP(0xc420055a00, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc420216690, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc4202166e0, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4206afde0, 0x7f54d3ddc3e0, 0xc420764098, 0xc420739b00)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4206afe60, 0x24b4cc0, 0xc420764098, 0xc420739b00, 0xc4205c87e0)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "[+]ping ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-cluster-operator-apiserver-informers ok\n[-]etcd failed: reason withheld\nhealthz check failed\n"
 [[kube-probe/.] 172.17.0.1:45580]
I0214 20:44:56.974886       1 handler.go:160] cluster-operator-apiserver: GET "/healthz" satisfied by nonGoRestful
I0214 20:44:56.974962       1 pathrecorder.go:240] cluster-operator-apiserver: "/healthz" satisfied by exact match
I0214 20:44:56.974975       1 server.go:132] etcd checker called
E0214 20:44:56.975415       1 server.go:141] etcd failed to reach any server
I0214 20:44:56.975485       1 healthz.go:112] healthz check etcd failed: etcd failed to reach any server
I0214 20:44:56.975665       1 wrap.go:42] GET /healthz: (874.595µs) 500
goroutine 228 [running]:
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420438230, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420438230, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4200dfa80, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
net/http.Error(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420846300, 0xb4, 0x1f4)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1930 +0xda
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508
net/http.HandlerFunc.ServeHTTP(0xc42073b160, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42075e940, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x190f857, 0x1a, 0xc42011a6c0, 0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4206afdc0, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	<autogenerated>:1 +0x75
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc420216640, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318
net/http.HandlerFunc.ServeHTTP(0xc420055a00, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc420216690, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc4202166e0, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4206afde0, 0x7f54d3ddc3e0, 0xc4207640c8, 0xc420646200)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4206afe60, 0x24b4cc0, 0xc4207640c8, 0xc420646200, 0xc4205c8a80)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "[+]ping ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-cluster-operator-apiserver-informers ok\n[-]etcd failed: reason withheld\nhealthz check failed\n"
 [[kube-probe/.] 172.17.0.1:45588]
I0214 20:45:06.099206       1 request.go:873] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0214 20:45:06.099323       1 round_trippers.go:417] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: cluster-operator/v0.0.0 (linux/amd64) kubernetes/790d284" -H "Accept: application/json, */*" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjbHVzdGVyLW9wZXJhdG9yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyLXRva2VuLTlxbmpuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXItb3BlcmF0b3ItYXBpc2VydmVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiM2VhZWExMTktMTFiZi0xMWU4LWI1MmItNGUyZmI2YmFjYWE3Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItb3BlcmF0b3I6Y2x1c3Rlci1vcGVyYXRvci1hcGlzZXJ2ZXIifQ.Q0bHyN-_FFpUTvjxO2mBDujZj7Twi8NIF_s6eLCfq1f8YM29wnbY2grlRBfzoK-2OyGtRGgSXQMdD9BcviruL8SkHLSz1GltVBb-IGMaT44FWek1E0LU3hQzKdFQNSqu-rxI0GX2RCk6swtjeoNJsVZ0mh-SqfkmhaW2di5o817lQvzghoUbBioSEHZyMvidgx9UqeWbG0TKXsTjI3d_VO8CvvRPwDlNeOZ4YSAXTfIiJLFTWvrI3dfj7PR7FkKnVE535rrzMwVqiGaj7v1v8KCPBwgHC_MroyWsS9AexkL3dFN9tSHOLN6W_zTIPP5sf8hvWmHfNYkbysJSTJRgqw" https://172.30.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0214 20:45:06.103588       1 round_trippers.go:436] POST https://172.30.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 4 milliseconds
I0214 20:45:06.103619       1 round_trippers.go:442] Response Headers:
I0214 20:45:06.103623       1 round_trippers.go:445]     Cache-Control: no-store
I0214 20:45:06.103626       1 round_trippers.go:445]     Content-Type: application/json
I0214 20:45:06.103628       1 round_trippers.go:445]     Content-Length: 301
I0214 20:45:06.103631       1 round_trippers.go:445]     Date: Wed, 14 Feb 2018 20:45:06 GMT
I0214 20:45:06.103688       1 request.go:873] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/healthz","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":true,"reason":"allowed by cluster rule"}}
I0214 20:45:06.103857       1 handler.go:160] cluster-operator-apiserver: GET "/healthz" satisfied by nonGoRestful
I0214 20:45:06.103883       1 pathrecorder.go:240] cluster-operator-apiserver: "/healthz" satisfied by exact match
I0214 20:45:06.103892       1 server.go:132] etcd checker called
E0214 20:45:06.104215       1 server.go:141] etcd failed to reach any server
I0214 20:45:06.104237       1 healthz.go:112] healthz check etcd failed: etcd failed to reach any server
I0214 20:45:06.104439       1 wrap.go:42] GET /healthz: (5.446118ms) 500
goroutine 119 [running]:
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc42025c1c0, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc42025c1c0, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4203ba740, 0x1f4)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
net/http.Error(0x7f54d3ddc3e0, 0xc420498050, 0xc420788780, 0xb4, 0x1f4)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1930 +0xda
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508
net/http.HandlerFunc.ServeHTTP(0xc42073b160, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42075e940, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x190f857, 0x1a, 0xc42011a6c0, 0xc4202ef7a0, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4206afdc0, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	<autogenerated>:1 +0x75
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc420216640, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318
net/http.HandlerFunc.ServeHTTP(0xc420055a00, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc420216690, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc4202166e0, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc4206afde0, 0x7f54d3ddc3e0, 0xc420498050, 0xc420121300)
	/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4206afe60, 0x24b4cc0, 0xc420498050, 0xc420121300, 0xc420602660)
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/Users/cewong/Code/cluster-operator/go/src/github.com/openshift/cluster-operator/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "[+]ping ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/start-cluster-operator-apiserver-informers ok\n[-]etcd failed: reason withheld\nhealthz check failed\n"
 [[kube-probe/.] 172.17.0.1:45626]
@csrwng
Copy link
Contributor Author

csrwng commented Feb 14, 2018

The etcd server seems to be healthy. Here's the log from that:

oc logs -f cluster-operator-apiserver-5fd88df5b9-jm7l5 -c etcd -n cluster-operator
2018-02-14 20:44:15.632535 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/etcd-data-dir
2018-02-14 20:44:15.632647 I | etcdmain: etcd Version: 3.3.1
2018-02-14 20:44:15.632663 I | etcdmain: Git SHA: 28f3f26c0
2018-02-14 20:44:15.632665 I | etcdmain: Go Version: go1.9.4
2018-02-14 20:44:15.632671 I | etcdmain: Go OS/Arch: linux/amd64
2018-02-14 20:44:15.632673 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2018-02-14 20:44:15.632679 I | etcdmain: advertising using detected default host "172.17.0.2"
2018-02-14 20:44:15.632707 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-02-14 20:44:15.632956 I | embed: listening for peers on http://localhost:2380
2018-02-14 20:44:15.633002 I | embed: listening for client requests on 0.0.0.0:2379
2018-02-14 20:44:15.828770 I | etcdserver: name = default
2018-02-14 20:44:15.828801 I | etcdserver: data dir = /etcd-data-dir
2018-02-14 20:44:15.828806 I | etcdserver: member dir = /etcd-data-dir/member
2018-02-14 20:44:15.828809 I | etcdserver: heartbeat = 100ms
2018-02-14 20:44:15.828811 I | etcdserver: election = 1000ms
2018-02-14 20:44:15.828813 I | etcdserver: snapshot count = 100000
2018-02-14 20:44:15.828885 I | etcdserver: advertise client URLs = http://172.17.0.2:2379
2018-02-14 20:44:17.829833 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 8520
2018-02-14 20:44:17.833699 I | raft: 8e9e05c52164694d became follower at term 2
2018-02-14 20:44:17.833730 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 8520, applied: 0, lastindex: 8520, lastterm: 2]
2018-02-14 20:44:17.838482 I | mvcc: restore compact to 1844
2018-02-14 20:44:24.833886 W | auth: simple token is not cryptographically signed
2018-02-14 20:44:24.836905 I | etcdserver: starting server... [version: 3.3.1, cluster version: to_be_decided]
2018-02-14 20:44:24.927792 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2018-02-14 20:44:24.928118 N | etcdserver/membership: set the initial cluster version to 3.3
2018-02-14 20:44:24.928250 I | etcdserver/api: enabled capabilities for version 3.3

@csrwng
Copy link
Contributor Author

csrwng commented Feb 14, 2018

It looks like at some point the liveness probe started failing:

oc describe pod cluster-operator-apiserver-5fd88df5b9-jm7l5 -n cluster-operator
Name:           cluster-operator-apiserver-5fd88df5b9-jm7l5
Namespace:      cluster-operator
Node:           localhost/192.168.65.2
Start Time:     Wed, 14 Feb 2018 14:42:52 -0500
Labels:         app=cluster-operator-apiserver
                pod-template-hash=1984489165
Annotations:    openshift.io/scc=restricted
Status:         Running
IP:             172.17.0.2
Controlled By:  ReplicaSet/cluster-operator-apiserver-5fd88df5b9
Containers:
  apiserver:
    Container ID:  docker://a198061bf4d2bf72f9f522c9dc841a9a800c99aa80aea306b7a637abc7a55fe9
    Image:         cluster-operator:canary
    Image ID:      docker://sha256:723e06a115eaf206c41f22afa49bcc268969f7d0d34bffaf12aa42beb7432762
    Port:          6443/TCP
    Command:
      /opt/services/cluster-operator
    Args:
      apiserver
      --secure-port
      6443
      --etcd-servers
      http://localhost:2379
      -v
      10
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 14 Feb 2018 15:50:36 -0500
      Finished:     Wed, 14 Feb 2018 15:50:56 -0500
    Ready:          False
    Restart Count:  19
    Liveness:       http-get https://:6443/healthz delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get https://:6443/healthz delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /var/run/openshift-cluster-operator from apiserver-ssl (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-operator-apiserver-token-9qnjn (ro)
  etcd:
    Container ID:  docker://60bba75bed87818c8b14b39ec4f94660b51aef4aa52e382d3ee15c7edcc646b4
    Image:         quay.io/coreos/etcd:latest
    Image ID:      docker-pullable://quay.io/coreos/etcd@sha256:6dd744ff28fec886e8789083a315f1fd59342b59a1dd3e73447c4b4ee3f22f99
    Port:          2379/TCP
    Command:
      /usr/local/bin/etcd
      --listen-client-urls
      http://0.0.0.0:2379
      --advertise-client-urls
      http://localhost:2379
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 14 Feb 2018 15:49:30 -0500
      Finished:     Wed, 14 Feb 2018 15:49:43 -0500
    Ready:          False
    Restart Count:  14
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:2379/health delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:  http-get http://:2379/health delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:
      ETCD_DATA_DIR:  /etcd-data-dir
    Mounts:
      /etcd-data-dir from etcd-data-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-operator-apiserver-token-9qnjn (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  apiserver-ssl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-operator-apiserver-cert
    Optional:    false
  etcd-data-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  cluster-operator-apiserver-token-9qnjn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-operator-apiserver-token-9qnjn
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                 Age                  From                Message
  ----     ------                 ----                 ----                -------
  Normal   Scheduled              1h                   default-scheduler   Successfully assigned cluster-operator-apiserver-5fd88df5b9-jm7l5 to localhost
  Normal   SuccessfulMountVolume  1h                   kubelet, localhost  MountVolume.SetUp succeeded for volume "etcd-data-dir"
  Normal   SuccessfulMountVolume  1h                   kubelet, localhost  MountVolume.SetUp succeeded for volume "apiserver-ssl"
  Normal   SuccessfulMountVolume  1h                   kubelet, localhost  MountVolume.SetUp succeeded for volume "cluster-operator-apiserver-token-9qnjn"
  Normal   Created                1h                   kubelet, localhost  Created container
  Normal   Started                1h                   kubelet, localhost  Started container
  Normal   Created                46m (x3 over 1h)     kubelet, localhost  Created container
  Normal   Pulling                46m (x3 over 1h)     kubelet, localhost  pulling image "quay.io/coreos/etcd:latest"
  Normal   Pulled                 46m (x3 over 1h)     kubelet, localhost  Successfully pulled image "quay.io/coreos/etcd:latest"
  Normal   Started                46m (x3 over 1h)     kubelet, localhost  Started container
  Warning  Unhealthy              45m (x4 over 46m)    kubelet, localhost  Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy              29m                  kubelet, localhost  Liveness probe failed: Get http://172.17.0.2:2379/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff                24m (x125 over 46m)  kubelet, localhost  Back-off restarting failed container
  Warning  Unhealthy              19m (x34 over 46m)   kubelet, localhost  Liveness probe failed: HTTP probe failed with statuscode: 500
  Normal   Pulled                 14m (x15 over 1h)    kubelet, localhost  Container image "cluster-operator:canary" already present on machine
  Warning  BackOff                4m (x173 over 43m)   kubelet, localhost  Back-off restarting failed container

@mfojtik
Copy link

mfojtik commented Jun 12, 2018

@csrwng we saw an issue earlier where the livenessprobe started failing because of slow DNS resolving... Can you confirm that all SDN pods are running on the master node? Also you can try to add static /etc/hosts entry and see if it resolves the crashloop.

@wongma7
Copy link

wongma7 commented Sep 27, 2018

@mfojtik I am seeing this as well, only after 10~ minutes. I did oc cluster up on some AWS instance, I don't think see any SDN pods running. I just ran ansible-playbook contrib/ansible/deploy-devel-playbook.yml and ansible-playbook contrib/ansible/create-cluster-playbook.yml and after a while cluster-operator-apiserver started crashlooping irrecoverably. What should I add to my /etc/hosts?

@wongma7
Copy link

wongma7 commented Sep 27, 2018

logs very similar to above, apiserver container panics the same way.

$ oc describe pod cluster-operator-apiserver-1-d6c7g
Name:           cluster-operator-apiserver-1-d6c7g
Namespace:      openshift-cluster-operator
Node:           localhost/172.18.11.37
Start Time:     Thu, 27 Sep 2018 18:08:45 +0000
Labels:         app=cluster-operator-apiserver
                deployment=cluster-operator-apiserver-1
                deploymentconfig=cluster-operator-apiserver
Annotations:    openshift.io/deployment-config.latest-version=1
                openshift.io/deployment-config.name=cluster-operator-apiserver
                openshift.io/deployment.name=cluster-operator-apiserver-1
                openshift.io/scc=restricted
Status:         Running
IP:             172.17.0.6
Controlled By:  ReplicationController/cluster-operator-apiserver-1
Containers:
  apiserver:
    Container ID:  docker://c3d16b164c47d209a5d7691b87eb843c9706599f96e9fd5d35d62aa1c2da1444
    Image:         172.30.1.1:5000/openshift-cluster-operator/cluster-operator@sha256:cca2a7d5026ba8b26dedcaae527b4fc67f9d61153ca275cb0236324a22f36c37
    Image ID:      docker-pullable://172.30.1.1:5000/openshift-cluster-operator/cluster-operator@sha256:cca2a7d5026ba8b26dedcaae527b4fc67f9d61153ca275cb0236324a22f36c37
    Port:          6443/TCP
    Args:
      apiserver
      --secure-port
      6443
      --etcd-servers
      http://localhost:2379
      -v
      10
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 27 Sep 2018 18:44:46 +0000
      Finished:     Thu, 27 Sep 2018 18:45:09 +0000
    Ready:          False
    Restart Count:  13
    Liveness:       http-get https://:6443/healthz delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get https://:6443/healthz delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /var/run/openshift-cluster-operator from apiserver-ssl (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-operator-apiserver-token-8vpm9 (ro)
  clusterapiserver:
    Container ID:  docker://e377cc0a68af1e1c91a7a80b02164e70cc0512889948a875d29c3a217472474a
    Image:         172.30.1.1:5000/openshift-cluster-operator/cluster-operator@sha256:cca2a7d5026ba8b26dedcaae527b4fc67f9d61153ca275cb0236324a22f36c37
    Image ID:      docker-pullable://172.30.1.1:5000/openshift-cluster-operator/cluster-operator@sha256:cca2a7d5026ba8b26dedcaae527b4fc67f9d61153ca275cb0236324a22f36c37
    Port:          7443/TCP
    Args:
      cluster-api-server
      --etcd-servers=http://localhost:2379
      --tls-cert-file=/var/run/cluster-api-apiserver/apiserver.crt
      --tls-private-key-file=/var/run/cluster-api-apiserver/apiserver.key
      --secure-port=7443
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 27 Sep 2018 18:44:45 +0000
      Finished:     Thu, 27 Sep 2018 18:45:06 +0000
    Ready:          False
    Restart Count:  13
    Liveness:       http-get https://:7443/healthz delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:      http-get https://:7443/healthz delay=10s timeout=2s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/cluster-api-apiserver from apiserver-ssl (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-operator-apiserver-token-8vpm9 (ro)
  etcd:
    Container ID:  docker://d30e5e14c40ea8233bb92a7f51200408e397aae3be5207110742491ef785836e
    Image:         quay.io/coreos/etcd:latest
    Image ID:      docker-pullable://quay.io/coreos/etcd@sha256:5b6691b7225a3f77a5a919a81261bbfb31283804418e187f7116a0a9ef65d21d
    Port:          2379/TCP
    Command:
      /usr/local/bin/etcd
      --listen-client-urls
      http://0.0.0.0:2379
      --advertise-client-urls
      http://localhost:2379
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 27 Sep 2018 18:42:56 +0000
      Finished:     Thu, 27 Sep 2018 18:44:15 +0000
    Ready:          False
    Restart Count:  9
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:2379/health delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:  http-get http://:2379/health delay=10s timeout=2s period=10s #success=1 #failure=1
    Environment:
      ETCD_DATA_DIR:  /etcd-data-dir
    Mounts:
      /etcd-data-dir from etcd-data-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-operator-apiserver-token-8vpm9 (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  apiserver-ssl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-operator-apiserver-cert
    Optional:    false
  etcd-data-dir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  etcd-storage
    ReadOnly:   false
  cluster-operator-apiserver-token-8vpm9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-operator-apiserver-token-8vpm9
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason                 Age                 From                Message
  ----     ------                 ----                ----                -------
  Normal   Scheduled              40m                 default-scheduler   Successfully assigned cluster-operator-apiserver-1-d6c7g to localhost
  Normal   SuccessfulMountVolume  40m                 kubelet, localhost  MountVolume.SetUp succeeded for volume "pv0013"
  Normal   SuccessfulMountVolume  40m                 kubelet, localhost  MountVolume.SetUp succeeded for volume "apiserver-ssl"
  Normal   SuccessfulMountVolume  40m                 kubelet, localhost  MountVolume.SetUp succeeded for volume "cluster-operator-apiserver-token-8vpm9"
  Normal   Started                40m                 kubelet, localhost  Started container
  Normal   Created                40m                 kubelet, localhost  Created container
  Normal   Started                40m                 kubelet, localhost  Started container
  Normal   Pulled                 40m                 kubelet, localhost  Container image "172.30.1.1:5000/openshift-cluster-operator/cluster-operator@sha256:cca2a7d5026ba8b26dedcaae527b4fc67f9d61153ca275cb0236324a22f36c37" already present on machine
  Normal   Created                40m                 kubelet, localhost  Created container
  Normal   Pulling                32m (x2 over 40m)   kubelet, localhost  pulling image "quay.io/coreos/etcd:latest"
  Normal   Pulled                 32m (x2 over 40m)   kubelet, localhost  Successfully pulled image "quay.io/coreos/etcd:latest"
  Normal   Created                32m (x2 over 40m)   kubelet, localhost  Created container
  Normal   Started                32m (x2 over 40m)   kubelet, localhost  Started container
  Warning  Unhealthy              32m                 kubelet, localhost  Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy              32m (x2 over 32m)   kubelet, localhost  Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy              32m (x2 over 32m)   kubelet, localhost  Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy              32m (x2 over 32m)   kubelet, localhost  Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff                20m (x69 over 32m)  kubelet, localhost  Back-off restarting failed container
  Normal   Pulled                 5m (x13 over 40m)   kubelet, localhost  Container image "172.30.1.1:5000/openshift-cluster-operator/cluster-operator@sha256:cca2a7d5026ba8b26dedcaae527b4fc67f9d61153ca275cb0236324a22f36c37" already present on machine
  Warning  BackOff                2s (x115 over 28m)  kubelet, localhost  Back-off restarting failed container

$ oc version
oc v3.9.0+191fece
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://127.0.0.1:8443
openshift v3.9.0+71543b2-33
kubernetes v1.9.1+a0ce1bc657

$ oc logs cluster-operator-apiserver-1-d6c7g  --namespace=openshift-cluster-operator -c etcd
2018-09-27 18:49:21.486665 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/etcd-data-dir
2018-09-27 18:49:21.486971 I | etcdmain: etcd Version: 3.3.8
2018-09-27 18:49:21.486984 I | etcdmain: Git SHA: 33245c6b5
2018-09-27 18:49:21.486988 I | etcdmain: Go Version: go1.9.7
2018-09-27 18:49:21.486993 I | etcdmain: Go OS/Arch: linux/amd64
2018-09-27 18:49:21.486999 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-09-27 18:49:21.487008 I | etcdmain: advertising using detected default host "172.17.0.6"
2018-09-27 18:49:21.487662 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-09-27 18:49:21.488055 I | embed: listening for peers on http://localhost:2380
2018-09-27 18:49:21.488144 I | embed: listening for client requests on 0.0.0.0:2379
2018-09-27 18:49:21.585433 I | etcdserver: name = default
2018-09-27 18:49:21.585465 I | etcdserver: data dir = /etcd-data-dir
2018-09-27 18:49:21.585471 I | etcdserver: member dir = /etcd-data-dir/member
2018-09-27 18:49:21.585475 I | etcdserver: heartbeat = 100ms
2018-09-27 18:49:21.585479 I | etcdserver: election = 1000ms
2018-09-27 18:49:21.585483 I | etcdserver: snapshot count = 100000
2018-09-27 18:49:21.585508 I | etcdserver: advertise client URLs = http://172.17.0.6:2379
2018-09-27 18:49:22.089654 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 1980
2018-09-27 18:49:22.089949 I | raft: 8e9e05c52164694d became follower at term 10
2018-09-27 18:49:22.089990 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 10, commit: 1980, applied: 0, lastindex: 1980, lastterm: 10]
2018-09-27 18:49:23.419859 W | auth: simple token is not cryptographically signed
2018-09-27 18:49:23.422103 I | etcdserver: starting server... [version: 3.3.8, cluster version: to_be_decided]
2018-09-27 18:49:23.424452 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2018-09-27 18:49:23.424549 N | etcdserver/membership: set the initial cluster version to 3.3
2018-09-27 18:49:23.424584 I | etcdserver/api: enabled capabilities for version 3.3
2018-09-27 18:49:25.186522 I | raft: 8e9e05c52164694d is starting a new election at term 10
2018-09-27 18:49:25.186565 I | raft: 8e9e05c52164694d became candidate at term 11
2018-09-27 18:49:25.186593 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 11
2018-09-27 18:49:25.186615 I | raft: 8e9e05c52164694d became leader at term 11
2018-09-27 18:49:25.186627 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 11
2018-09-27 18:49:25.187005 I | etcdserver: published {Name:default ClientURLs:[http://172.17.0.6:2379]} to cluster cdf818194e3a8c32
2018-09-27 18:49:25.187133 I | embed: ready to serve client requests
2018-09-27 18:49:25.187679 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!

@wongma7
Copy link

wongma7 commented Sep 27, 2018

So etcd is not healthy after all, readding logs more carefully etcd is what gets OOMKilled first and then the apiserver container starts panicking. My etcd log is filled with messages like

2018-09-27 19:33:29.803997 W | etcdserver: request "header:<ID:7587833219866082036 > txn:<compare:<target:MOD key:"/registry/k8s.io/cluster.k8s.io/clusters/myproject/fedora-9c8wk" mod_revision:1689 > success:<request_put:<key:"/registry/k8s.io/cluster.k8s.io/clusters/myproject/fedora-9c8wk" value_size:5901 >> failure:<request_range:<key:"/registry/k8s.io/cluster.k8s.io/clusters/myproject/fedora-9c8wk" > >>" with result "size:16" took too long (398.706759ms) to execute

and

2018-09-27 19:33:33.000241 W | etcdserver: read-only range request "key:"/registry/k8s.io/cluster.k8s.io/clusters/myproject/fedora-9c8wk" " with result "range_response_count:1 size:5987" took too long (2.196201228s) to execute

@wongma7
Copy link

wongma7 commented Sep 27, 2018

Well it seems like etcd is simply running out of memory idle because I increased

cluster-operator/build/cluster-operator-ansible/playbooks/cluster-api-prep/files/cluster-api-template.yaml

Line 134 in ae319de

memory: 20Mi
to 200 & 300Mi and everything is working fine now.
How much memory does the cluster operator's apiserver etcd need typically?

(Sorry for noise, could have done all this debugging first before commenting.)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mfojtik @csrwng @wongma7