Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with connectionDrainingTimeoutSec Not Updating to 0 in Config Connector #3194

Open
3 tasks done
marandalucas opened this issue Nov 19, 2024 · 0 comments
Open
3 tasks done
Labels
bug Something isn't working

Comments

@marandalucas
Copy link

marandalucas commented Nov 19, 2024

Checklist

Bug Description

  • When attempting to update the connectionDrainingTimeoutSec field of compute.backendServices from 300 to 0 using Config Connector (version 1.116.0), the update fails. The resource repeatedly switches between "UpToDate" and "Updating" states. This issue does not occur when setting the value to other integers like 1, but setting it to 0 results in continuous update attempts.

Additional Diagnostic Information

Expected Behavior:

  • Config Connector should successfully apply the change to connectionDrainingTimeoutSec set to 0, as 0 is a valid value for this field and has been set in other compute.backendServices without issues.

Actual Behavior:

  • Config Connector fails to update connectionDrainingTimeoutSec to 0. Instead, the resource is stuck in the "Updating" state, and the update is never applied. Setting the value to 1 works as expected, but setting it to 0 causes failure.

Investigation and Findings:

  • connectionDrainingTimeoutSec expects an integer, and 0 is a valid value for this field.
  • The issue is specific to updating the value to 0, which causes Config Connector to enter a continuous update loop.
  • No similar issue is observed when setting the value to 1 or other integers.
  • Manually setting the value to 0 via gcloud CLI resolves the issue for the affected backend services.

Temporary Fix Applied:

  • To resolve the inconsistent state, we manually updated connectionDrainingTimeoutSec for all affected backend services using the following command:
    for backend_service in $(gcloud compute backend-services list --global --filter="name~'sgnas'" --format="value(name)"); do
      gcloud compute backend-services update $backend_service \
        --global \
        --connection-draining-timeout 0
    done

Version:
Config Connector Version: 1.116.0

Impact:

  • This issue is causing instability across backend services managed by Config Connector, resulting in continuous update attempts and configuration drift.

Additional Notes:

  • No related issue or fix has been found in the Config Connector GitHub repository.
  • Manually updating the field via the gcloud CLI resolves the issue.

Kubernetes Cluster Version

v1.30.5-gke.1014003

Config Connector Version

1.116.0

Config Connector Mode

cluster mode

Log Output

{"severity":"info","timestamp":"2024-11-19T09:20:51.402Z","logger":"computebackendservice-controller","msg":"starting reconcile","resource":{"name":"pv-customer-c-bs-sgnas","namespace":"pv-customer-c"}}
{"severity":"info","timestamp":"2024-11-19T09:20:51.922Z","logger":"computebackendservice-controller","msg":"creating/updating underlying resource","resource":{"name":"pv-customer-c-bs-sgnas","namespace":"pv-customer-c"}}
{"severity":"info","timestamp":"2024-11-19T09:21:14.382Z","logger":"computebackendservice-controller","msg":"successfully finished reconcile","resource":{"name":"pv-customer-c-bs-sgnas","namespace":"pv-customer-c"},"time to next reconciliation":"12m9.220309803s"}

Steps to reproduce the issue

  • Use Config Connector to manage a compute.backendService resource with connectionDrainingTimeoutSec set to 300.
  • Attempt to update connectionDrainingTimeoutSec to 0.
  • Observe that Config Connector does not apply the update, and the resource goes into an "Updating" state repeatedly.
  • Set connectionDrainingTimeoutSec to 1 and observe that Config Connector applies the change immediately.

YAML snippets

No response

@marandalucas marandalucas added the bug Something isn't working label Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant