Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kfp-db could not connect to Pebble: socket not found #568

Open
mvlassis opened this issue Sep 5, 2024 · 2 comments
Open

kfp-db could not connect to Pebble: socket not found #568

mvlassis opened this issue Sep 5, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@mvlassis
Copy link
Contributor

mvlassis commented Sep 5, 2024

Bug Description

I found this issue in the CI, and it has occurred in multiple runs for different PRs:

Examples 1 and 2 are for the Test the bundle (v1) actions, Example 3 is for the Test the bundle (v2) actions.
The kfp-db charm goes into an error status:

=========================== short test summary info ============================
FAILED tests/integration/test_kfp_functional_v1.py::test_build_and_deploy - juju.errors.JujuUnitError: Unit in error: kfp-db/0

To Reproduce

It's not clear yet whether this is reproducible. All 3 example have been produced by triggering a PR and looking at the results of the Test the bundle action. All 3 examples use charmcraft version 3.1/stable.

Environment

  • juju version: 3.4.5
  • charmcraft version: 3.1/stable
  • branch: Example 1 is on track/2.2, example 2 is on track/2.0, example 3 is on main.
  • mysql-k8s channel: 8.0/stable

Relevant Log Output

unit-kfp-db-0: 2024-09-04 14:12:03 ERROR unit.kfp-db/0.juju-log database:5: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/./src/charm.py", line 888, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 551, in main
    manager.run()
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 530, in run
    self._emit()
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 519, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2886, in _on_relation_changed_event
    getattr(self.on, "database_requested").emit(
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/src/relations/mysql_provider.py", line 99, in _on_database_requested
    if not self.charm.cluster_initialized:
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/mysql/v0/mysql.py", line 617, in cluster_initialized
    if self._mysql.cluster_metadata_exists(self.get_unit_address(unit)):
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/mysql/v0/mysql.py", line 1539, in cluster_metadata_exists
    output = self._run_mysqlsh_script("\n".join(check_cluster_metadata_commands))
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/src/mysql_k8s_helpers.py", line 599, in _run_mysqlsh_script
    self.container.push(path=MYSQLSH_SCRIPT_FILE, source=script)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/model.py", line 2438, in push
    self._pebble.push(
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/pebble.py", line 2494, in push
    response = self._request_raw('POST', '/v1/files', None, headers, data)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/pebble.py", line 2048, in _request_raw
    raise ConnectionError(
ops.pebble.ConnectionError: Could not connect to Pebble: socket not found at '/charm/containers/mysql/pebble.socket' (container restarted?)

Additional Context

It's not clear to me whether this is an intermittent error or not.

@mvlassis mvlassis added the bug Something isn't working label Sep 5, 2024
Copy link

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6202.

This message was autogenerated

@mvlassis mvlassis changed the title kfp-db could not to Pebble: socket not found kfp-db could not connect to Pebble: socket not found Sep 5, 2024
@paulomach
Copy link

Hey @mvlassis , as convo I would recommend you to set the charm config profile=testing for CI tests for kfp-db here.

Quoting my message:

We are probably being OOM killed by the container runtime. We are currently further tweaking our memory settings, but for now I would say go with the testing profile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants