Restore data fails with Permission Denied error

[nachiroy]

Hi All,

I am getting the following error when performing a restore operation:

2024/03/04 23:27:33.630300 debug done                      duration=2.567s operation=HardlinkBackupPartsToStorage
2024/03/04 23:27:33.630319 debug data to 'detached' copied backup=Demo_Backup operation=restore table=DemoDb.DemoTable_Local
2024/03/04 23:27:33.630355  info SELECT sum(log_pointer + log_max_index + absolute_delay + queue_size)  AS replication_in_progress FROM system.replicas WHERE database=? and table=? SETTINGS empty_result_for_aggregation_by_empty_set=0 with args [DemoDb DemoTable_Local] logger=clickhouse
2024/03/04 23:27:33.636787  info replication_in_progress status = [{InProgress:0}] logger=clickhouse
2024/03/04 23:27:33.636806  info ALTER TABLE `DemoDb`.`DemoTable_Local` ATTACH PART '1684432800_0_47_10_618' logger=clickhouse
2024/03/04 23:27:33.637572  info clickhouse connection closed logger=clickhouse
**2024/03/04 23:27:33.637590 error can't attach data parts for table 'DemoDb.DemoTable_Local': 
code: 1001, message: std::__1::__fs::filesystem::filesystem_error: filesystem error: 
in directory_iterator::directory_iterator(...): Permission denied
["/mnt/DemoProject/ClickHouse/store/ae1/ae1e2f25-0b2f-4616-91b4-0642391a2ea0/detached/attaching_1684432800_0_47_10_618"]**

We are running 'clickhouse' on physical linux machines. The clickhouse is running under the 'clickhouse' user. I downloaded clickhouse-backup from the releases page and extracted the tar.gz file.

These are the steps I took to perform a restore operation manually via the cli interface (config file is set via environment variable):

1. sudo su
2. mkdir /path/to/clickhousebackup
3. cd /path/to/clickhousebackup
4. ./clickhouse-backup download Demo_Backup --resumable
5. ./clickhouse-backup restore --data --tables="DemoDb.Demo_Backup" Demo_Backup (I am restoring data only as the table is created by a separate service'

I found another similar discussion here, however it looks like there they were running both clickhouse and clickhouse-backup as docker containers. The issue seems to be similar however.
If it is the same issue then how can I run clickhouse-backup as 'clickhouse' user?

[Slach]

sorry for late reply
could you provide more context?

which is your environment?
Is this standalone linux or docker or kubernetes?

[Slach]

could you share

df -h /mnt/DemoProject/

[nachiroy]

Hi @Slach!
No worries. Apologies for the late response as well. I forgot to update this thread. I have added the solution that worked for me. You can mark it as closed.

Thanks for your time!

[nachiroy]

I found the root cause of the issue. I was running 'clickhouse-backup' tool under the root user in our standalone Linux machines. As a result ClickHouse did not have the correct permissions to the parts file to re-attach them during restore process.

In order to successfully run under the clickhouse user (used by clickhouse-server process) I had to make the following modifications:

1. sudo su (I had root access to our Linux server)
2. vim /etc/passwd - This is the Linux file containing all the user accounts and their defaults.
3. Change the clickhouse user details. Modify \bin\false to \bin\bash. This allowed me to execute the sudo su - clickhouse command in bash (allows logging in as clickhouse user. Note this is a security risk as it allows users to login as clickhouse. In our setup we have controlled and audited access to our servers. Hence, it was not a security risk for us.)
4. Finally, execute sudo su - clickhouse which switches to the clickhouse user.

Now restore works properly and I no longer get the permission denied error.

PS: Thank you @hodgesrm for your clue/suggestion which helped me narrow down my search. The thread that helped me debug this was ClickHouse/ClickHouse#20280.