PMM Upgrades doesn't work with external postgresql db for grafana

[GenLN]

Description

Trying pmm upgrade with helm chart (helm upgrade doesn't works, so I need to run uninstall/install helm chart)
pmm-upgrade fail
TASK [initialization : Change default admin id] ******************************** fatal: [localhost]: FAILED! => {"changed": false, "msg": "unable to connect to database: connection to server on socket \"/var/run/postgresql/.s.PGSQL.5432\" failed: FATAL: database \"grafana\" does not exist\n"}

Where does this task have defined or use this config for postgresql? How to change it?

Expected Results

Upgrade done

Actual Results

Failed/stuck/no revert to previous version

Version

PMM Server 2.41.0 -> 2.41.1

Steps to reproduce

define grafana external postgres db using pmmEnv in values file:

GF_DATABASE_HOST:
GF_DATABASE_NAME:
GF_DATABASE_TYPE:
GF_DATABASE_USER:
GF_DATABASE_PASSWORD:

Relevant logs

TASK [dashboards_upgrade : Copy plugins to the plugin directory] ***************
changed: [localhost]

TASK [dashboards_upgrade : Set permissions for the plugin directory] ***********
changed: [localhost]

TASK [dashboards_upgrade : Check that the SQLite grafana database exists] ******
ok: [localhost]

TASK [dashboards_upgrade : Restart grafana before deleting data EL7] ***********
skipping: [localhost]

TASK [dashboards_upgrade : Restart grafana before deleting data EL9] ***********
skipping: [localhost]

TASK [dashboards_upgrade : Remove old DBaaS dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Inventory (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Add Instance dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Database Checks dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Settings dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove the old clickhouse plugin] *******************
changed: [localhost]

TASK [dashboards_upgrade : Restart grafana with new plugins EL7] ***************
skipping: [localhost]

TASK [dashboards_upgrade : Restart grafana with new plugins EL9] ***************
changed: [localhost]

TASK [initialization : Create backup directory] ********************************
ok: [localhost]

TASK [initialization : Check if Postgres 11 exists] ****************************
ok: [localhost]

TASK [Upgrade Postgres database] ***********************************************
skipping: [localhost]

TASK [initialization : Create grafana database in postgres] ********************
skipping: [localhost]

TASK [initialization : Create grafana user in postgres] ************************
skipping: [localhost]

TASK [initialization : Upgrade grafana database (Get the latest schema)] *******
skipping: [localhost]

TASK [initialization : Create working directory for Alertmanager] **************
ok: [localhost]

TASK [initialization : Create working directory for VictoriaMetrics] ***********
ok: [localhost]

TASK [initialization : Create empty configuration file for VictoriaMetrics] ****
changed: [localhost]

TASK [initialization : Check that the SQLite grafana database exists] **********
ok: [localhost]

TASK [initialization : Remove database options (SQLite is default)] ************
skipping: [localhost]

TASK [initialization : Remove database host] ***********************************
skipping: [localhost]

TASK [initialization : Remove database user] ***********************************
skipping: [localhost]

TASK [initialization : Remove database password] *******************************
skipping: [localhost]

TASK [initialization : Upgrade grafana database (Get the latest schema)] *******
skipping: [localhost]

TASK [initialization : Start grafana again] ************************************
skipping: [localhost]

TASK [initialization : Wait for grafana] ***************************************
skipping: [localhost]

TASK [Migrate Grafana database from SQLite to Postgresql] **********************
skipping: [localhost]

TASK [initialization : Change default admin id] ********************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "unable to connect to database: connection to server on socket \"/var/run/postgresql/.s.PGSQL.5432\" failed: FATAL:  database \"grafana\" does not exist\n"}

PLAY RECAP *********************************************************************
localhost                  : ok=27   changed=8    unreachable=0    failed=1    skipped=25   rescued=0    ignored=0   

time="2024-03-21T10:41:22Z" level=fatal msg="RunPlaybook failed: exit status 2"
ProjectName: pmm-update
Version: 2.41.1
PMMVersion: 2.41.1
Timestamp: 2024-01-29 13:32:29 (UTC)
FullCommit: f4004eac5c03e97327ae235dbf40f91e31ba2208

Code of Conduct

• I agree to follow Percona Community Code of Conduct

[GenLN]

Hi this issue still persist, upgrade process is still trying to connect to localhost postgres using socket, even if i defined ENV vars in every possible grafana file where it could be, even the defaults.ini

Task will fail over and over again.

The bypassing of this issue is to not upgrade, but remove pmm by helm uninstall and delete pvc, and install it like new release, so no upgrade process here, then it works with external grafana postgresql.. Which means all metrics are lost...