kfp-db could not connect to Pebble: socket not found

[mvlassis]

Bug Description

I found this issue in the CI, and it has occurred in multiple runs for different PRs:

• Example 1
• Example 2
• Example 3

Examples 1 and 2 are for the Test the bundle (v1) actions, Example 3 is for the Test the bundle (v2) actions.
The kfp-db charm goes into an error status:

=========================== short test summary info ============================
FAILED tests/integration/test_kfp_functional_v1.py::test_build_and_deploy - juju.errors.JujuUnitError: Unit in error: kfp-db/0

To Reproduce

It's not clear yet whether this is reproducible. All 3 example have been produced by triggering a PR and looking at the results of the Test the bundle action. All 3 examples use charmcraft version 3.1/stable.

Environment

• juju version: 3.4.5
• charmcraft version: 3.1/stable
• branch: Example 1 is on track/2.2, example 2 is on track/2.0, example 3 is on main.
• mysql-k8s channel: 8.0/stable

Relevant Log Output

unit-kfp-db-0: 2024-09-04 14:12:03 ERROR unit.kfp-db/0.juju-log database:5: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/./src/charm.py", line 888, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 551, in main
    manager.run()
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 530, in run
    self._emit()
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 519, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2886, in _on_relation_changed_event
    getattr(self.on, "database_requested").emit(
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/src/relations/mysql_provider.py", line 99, in _on_database_requested
    if not self.charm.cluster_initialized:
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/mysql/v0/mysql.py", line 617, in cluster_initialized
    if self._mysql.cluster_metadata_exists(self.get_unit_address(unit)):
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/mysql/v0/mysql.py", line 1539, in cluster_metadata_exists
    output = self._run_mysqlsh_script("\n".join(check_cluster_metadata_commands))
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/src/mysql_k8s_helpers.py", line 599, in _run_mysqlsh_script
    self.container.push(path=MYSQLSH_SCRIPT_FILE, source=script)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/model.py", line 2438, in push
    self._pebble.push(
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/pebble.py", line 2494, in push
    response = self._request_raw('POST', '/v1/files', None, headers, data)
  File "/var/lib/juju/agents/unit-kfp-db-0/charm/venv/ops/pebble.py", line 2048, in _request_raw
    raise ConnectionError(
ops.pebble.ConnectionError: Could not connect to Pebble: socket not found at '/charm/containers/mysql/pebble.socket' (container restarted?)

Additional Context

It's not clear to me whether this is an intermittent error or not.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6202.

This message was autogenerated

[paulomach]

Hey @mvlassis , as convo I would recommend you to set the charm config profile=testing for CI tests for kfp-db here.

Quoting my message:

We are probably being OOM killed by the container runtime. We are currently further tweaking our memory settings, but for now I would say go with the testing profile