Cloning a primary to standby fails if PGDATA is a symbolic link

[JP95Git]

I tried to clone my primary to my standby using repmgr. repmgr finished cloning without an error, but postgresql refused to start. I noticed that the cloned database was "cut in half".

First my storage layout:

[user@myhost1 user]$ ls -la /opt/mytool/
insgesamt 8
drwxrwxr-x  2 user group  4096 18. Okt 13:29 .
drwxr-xr-x. 5 root    root 4096  5. Feb 2024  ..
lrwxrwxrwx  1 user group    22 19. Sep 13:11 database -> /media/database/database
lrwxrwxrwx  1 user group    30 27. Mär 2024  database-backup -> /media/backup/database-backup/

I am running postgresql, barman and repmgr in a very restricted, but powerful environment. So the local HDD is very small, just enough for the operating system. Everything else is stored on network storages, which are mounted under /media/... . Users have to use symbolic links to "add" the storage to the right place.

First repmgr wrote everything to /media/database/database. Since the clone took "very long" to copy 600 GB of data, I did not watched the process.

But at the end we had this situation:

• Some data was written to /media/database/database
• Then someone removed the symbolic link and readded the database folder.
• This database folder contained the rest of PGDATA

I tried it several times, the clone was always "cut in half".

Versions:

postgres (PostgreSQL) 16.1
3.10.0 Barman by EnterpriseDB (www.enterprisedb.com)
repmgr 5.4.1

My configuration of barman can be found here:
EnterpriseDB/barman#1027

repmgr.conf (identical file for all servers, except the hostnames and node_id are different)

node_id=1
node_name='mytool-Node-1'
conninfo='host=myhost1 user=repmgr dbname=repmgr connect_timeout=2'
data_directory='/opt/mytool/database'
log_file='/opt/mytool/programs/repmgr/logs/repmgr.log'
failover='automatic'
promote_command='/opt/mytool/programs/postgres/bin/repmgr standby promote --config-file=/opt/mytool/programs/repmgr/repmgr.conf --log-to-file'
follow_command='/opt/mytool/programs/postgres/bin/repmgr standby follow ---config-file=/opt/mytool/programs/repmgr/repmgr.conf --log-to-file --upstream-node-id=%n'
monitor_interval_secs=2
connection_check_type='ping'
reconnect_attempts=4
reconnect_interval=8
primary_visibility_consensus=true
standby_disconnect_on_failover=true
repmgrd_service_start_command='sudo /usr/bin/systemctl start mytool-repmgr.service'
repmgrd_service_stop_command='sudo /usr/bin/systemctl stop mytool-repmgr.service'
service_start_command='sudo /usr/bin/systemctl start mytool-postgresql.service'
service_stop_command='sudo /usr/bin/systemctl stop mytool-postgresql.service'
service_restart_command='sudo /usr/bin/systemctl restart mytool-postgresql.service'
service_reload_command='sudo /usr/bin/systemctl reload mytool-postgresql.service'
monitoring_history=yes
log_status_interval=60
barman_server=localhost
barman_config='/opt/mytool/programs/barman/etc/barman.conf'
barman_host=mytool-Node-2
restore_command='/opt/mytool/programs/barman/bin/barman-wal-restore --config /opt/mytool/programs/barman/etc/barman.conf --user mytool mytool-Node-2 localhost %f %p'

The clone was done using this command line:

/opt/mytool/programs/postgres/bin/repmgr --config-file=/opt/mytool/programs/repmgr/repmgr.conf --host=myhost1 --username=repmgr --dbname=repmgr standby clone --force

As a workaround I changed "data_directory" in the repmgr settings to the real path e.g. /media/database/database . This prevented repmgr from cut my PGDATA in 2 parts.