How to recover data if the simulated data disk is damaged?

[mxt-gith-qq]

目的：在服务器搭建RustFS系统，想模拟磁盘损坏恢复数据的过程。
服务器信息：操作系统为银河麒麟，有5块磁盘（4块挂载盘），Linux bogon 4.19.90-89.11.v2401.ky10.x86_64 #1 SMP Tue May 7 18:33:01 CST 2024 x86_64 x86_64 x86_64 GNU/Linux

以下是我的操作步骤：
1 格式化磁盘，并给磁盘设置标签，以便后续挂载

	sudo mkfs.xfs  -i size=512 -n ftype=1 -L RUSTFS0 /dev/sdb
	sudo mkfs.xfs  -i size=512 -n ftype=1 -L RUSTFS1 /dev/sdc
	sudo mkfs.xfs  -i size=512 -n ftype=1 -L RUSTFS2 /dev/sdd
	sudo mkfs.xfs  -i size=512 -n ftype=1 -L RUSTFS3 /dev/sde

2 创建目录并授权

	sudo mkdir -p /data/rustfs{0..3} && chown -R 10001:10001 /data/rustfs{0..3}
	sudo mkdir -p /var/logs/rustfs /opt/tls
	sudo chmod -R 750 /var/logs/rustfs

3 挂载

	vim /etc/fstab
		LABEL=RUSTFS0 /data/rustfs0   xfs   defaults,noatime,nodiratime   0   0
		LABEL=RUSTFS1 /data/rustfs1   xfs   defaults,noatime,nodiratime   0   0
		LABEL=RUSTFS2 /data/rustfs2   xfs   defaults,noatime,nodiratime   0   0
		LABEL=RUSTFS3 /data/rustfs3   xfs   defaults,noatime,nodiratime   0   0

	sudo mount -a

4 下载并安装RustFS

	wget https://dl.rustfs.com/artifacts/rustfs/release/rustfs-linux-x86_64-musl-latest.zip
	unzip rustfs-linux-x86_64-musl-latest.zip
	chmod +x rustfs
	mv rustfs /usr/local/bin/

5 配置环境变量（单机多盘模式）

	sudo tee /etc/default/rustfs <<EOF
	RUSTFS_ACCESS_KEY=rustfsadmin
	RUSTFS_SECRET_KEY=rustfsadmin123
	RUSTFS_VOLUMES="/data/rustfs{0...3}"
	RUSTFS_ADDRESS=":9000"
	RUSTFS_CONSOLE_ENABLE=true
	RUST_LOG=error
	RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
	EOF

6 配置系统服务

	sudo tee /etc/systemd/system/rustfs.service <<EOF
	[Unit]
	Description=RustFS Object Storage Server
	Documentation=https://rustfs.com/docs/
	After=network-online.target
	Wants=network-online.target

	[Service]
	Type=notify
	NotifyAccess=main
	User=root
	Group=root

	WorkingDirectory=/usr/local
	EnvironmentFile=-/etc/default/rustfs
	ExecStart=/usr/local/bin/rustfs $RUSTFS_VOLUMES

	LimitNOFILE=1048576
	LimitNPROC=32768
	TasksMax=infinity

	Restart=always
	RestartSec=10s

	OOMScoreAdjust=-1000
	SendSIGKILL=no

	TimeoutStartSec=30s
	TimeoutStopSec=30s

	NoNewPrivileges=true
	 
	ProtectHome=true
	PrivateTmp=true
	PrivateDevices=true
	ProtectClock=true
	ProtectKernelTunables=true
	ProtectKernelModules=true
	ProtectControlGroups=true
	RestrictSUIDSGID=true
	RestrictRealtime=true

	# service log configuration
	StandardOutput=append:/var/logs/rustfs/rustfs.log
	StandardError=append:/var/logs/rustfs/rustfs-err.log

	[Install]
	WantedBy=multi-user.target
	EOF

7 重新加载服务配置，防火墙开通相关端口略过

	sudo systemctl daemon-reload

8 启动rustfs服务

	systemctl start rustfs

9 访问文件系统

	http://192.168.1.101:9001/rustfs/console/browser

10 创建存储桶yjgs，并上传5张jpg图片，分别为1.jpg、2.jpg、3.jpg、4.jpg和5.jpg

执行上面的10步操作后，开始模拟磁盘/dev/sdb损坏

1）、关闭或不关闭Rustfs服务的情况下，将/data/rustfs0目录的文件全部清空；
2）、重启Rustfs服务，发现/data/rustfs0目录只恢复了2.jpg、3.jpg和5.jpg三个文件；

请问如何操作或配置能将/data/rustfs0目录下的文件全部恢复，麻烦给出详细的解决方案，谢谢！

[gbyangg]

我感觉rustfs还有一段时间得打磨，我这几天也在弄，但是发现了一些问题，而且排查起来还很麻烦。

[mxt-gith-qq]

有遇见过我这种情况吗？

[gbyangg]

rustfs-linux-x86_64-musl-latest.zip 这个最新版本你创建用户之后，能用新用户登录成功吗？

[mxt-gith-qq]

不能

[mxt-gith-qq]

对 yjgs 存储桶进行手动深度健康修复任务，提示运行成功，但是文件也没恢复
[root@localhost rustfs0]# rc admin heal start myrustfs --bucket yjgs
✓ Heal operation started successfully.
[root@localhost rustfs0]# rc admin heal start myrustfs --bucket yjgs --scan-mode deep
✓ Heal operation started successfully.
[root@localhost rustfs0]# rc admin heal start myrustfs
✓ Heal operation started successfully.
[root@localhost rustfs0]# rc admin heal status myrustfs
Heal Status: Idle

No heal operation currently running.

[weisd]

@mxt-gith-qq Does the 1.jpg/xl.meta file exist on other normal disks?

[mxt-gith-qq]

Yes, both 1.jpg and 4.jpg exist in the directories of the other three disks

[loverustfs]

It is recommended to refrain from writing data immediately and back up the data directories of the remaining three normal disks.

I verified the same scenario on the current RustFS main in a 4-node/4-disk distributed mode: after uploading 1.jpg to 5.jpg and then clearing one of the disks, restarting the cluster, all five objects were restored to the cleared disk, and S3 GET read verifications all passed. Therefore, the current version does not reproduce the "only partial object recovery" issue in a distributed 4-node/4-disk environment.

For your environment, it is recommended to follow the steps below:

1. Stop RustFS.

2. Back up the data directories of the remaining three normal disks.

3. Check if 1.jpg and 4.jpg have xl.meta and corresponding part.1 on the other three disks.

4. Upgrade to the latest RustFS.

5. Reformat the damaged disk and mount it back to the original /data/rustfs0 path, maintaining the RUSTFS_VOLUMES order.

6. Start RustFS.

7. Execute a deep heal:

rc admin heal start myrustfs --bucket yjgs --scan-mode deep

8. Then read 1.jpg and 4.jpg via S3/rc to confirm object readability and check if /data/rustfs0/yjgs/1.jpg and /data/rustfs0/yjgs/4.jpg have been recovered.

If the problem persists after a deep heal, please provide the following:

• The specific RustFS version;

• The RUSTFS_VOLUMES configuration in /etc/default/rustfs;

• The find output of yjgs/1.jpg and yjgs/4.jpg on the four disks;

• RustFS logs during the heal.