Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LP#2068597] Metallb deployments can get stuck in a config-changed busy loop #41

Merged
merged 4 commits into from
Jun 21, 2024

Conversation

addyess
Copy link
Member

@addyess addyess commented Jun 6, 2024

LP#2068597

Overview

When metallb doesn't have permissions because of lack of trust, the charm can get stuck trying to apply a configuration that will never finish causing the config-change hook to perpetually run.

Changes

  • use tenacity stop to quick retrying after a certain number of times
  • use a context_manager to block the charm when 403 error codes are returned from the API indicating a lack of permission
  • Don't let the update_status hook wipe out the blocked status
  • Only go to ActiveStatus when the update_status handler declares everything is ready
  • Use MaintenanceStatus to indicate the charm is working on something
  • Include tests to confirm the charm is blocked when not deployed with trust

@addyess addyess force-pushed the bug/lp2068597/stop-retrying-during-apifailures branch 2 times, most recently from 28cd9dc to 18f96cd Compare June 6, 2024 16:34
@addyess addyess force-pushed the bug/lp2068597/stop-retrying-during-apifailures branch from 18f96cd to fc992ee Compare June 6, 2024 17:25
@addyess addyess force-pushed the bug/lp2068597/stop-retrying-during-apifailures branch from eb06f62 to 2cb36be Compare June 6, 2024 21:07
src/charm.py Outdated
Comment on lines 54 to 55
if str_to_test.count("/") != 1:
return False

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or test .+/\d{1,2}$ ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing scarier here than using regex for ip address matching. I'll let python check it. See if you like my change here.

i want to make sure it's a ip_network and NOT an ip_address. The python library accepts a bare address to be processed as a "network" even when it's not in CIDR notation

assert ipaddress.ip_network("1.2.3.4") == ipaddress.ip_network("1.2.3.4/32")

HOWEVER, using ip_address totally raises a ValueError

>>> ipaddress.ip_address("1.2.3.4/32")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.pyenv/versions/3.12.3/lib/python3.12/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '1.2.3.4/32' does not appear to be an IPv4 or IPv6 address
```

src/charm.py Outdated
Comment on lines 78 to 86
def _block_on_forbidden(unit: ops.model.Unit):
try:
yield
except (ApiError, ManifestClientError) as ex:
http = ex.args[1] if isinstance(ex, ManifestClientError) else ex
if http.status.code == 403:
unit.status = BlockedStatus("API Access Forbidden, deploy with --trust")
else:
raise

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool pattern, had not seen before.

@@ -168,9 +175,10 @@ def test_update_status(harness, lk_manifests_client, lk_charm_client):
harness.begin()

# With nothing mocked, all resources will appear as missing
harness.charm._stored.configured = True
harness.charm.model.unit.status = expected = BlockedStatus("unchanged on missing")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unchanged on missing? Is that the best wording?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i just tried to make this more clear. I just wanted some text that clearly indicates the charm code isn't changing the blocked message

@addyess addyess merged commit 2b0687d into main Jun 21, 2024
8 checks passed
@addyess addyess deleted the bug/lp2068597/stop-retrying-during-apifailures branch June 21, 2024 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants