data consistency issues

[holygrolli]

Hi,

I did not create an issue yet. Perhaps some input of you might clear the fog around this:

I tried juicefs v1.0.0 in my home lab "production" scenario within kubernetes (via csi-plugin) for several days now. Now there are multiple observations:

1. I observed a huge spike in my metadata store over night, from 18250 to 147859 while not generating too much data. I would estimate about 600mb data being added.
2. During a juicefs fsck I encountered lots of file does not exist errors.

2022/09/08 06:48:09.706673 juicefs[2570293] <WARNING>: GetAttr inode 1000466: no such file or directory [base.go:931]                                                                 │root@vmd96188:~# keydb-cli -p 6379 -n 0 DBSIZE
2022/09/08 06:48:09.706795 juicefs[2570293] <ERROR>: can't find block 1/1212/1212430_0_646 for file inode:1000466: file does not exist [fsck.go:146]                                  │(integer) 23077
2022/09/08 06:48:09.725804 juicefs[2570293] <WARNING>: GetAttr inode 1003278: no such file or directory [base.go:931]                                                                 │root@vmd96188:~# keydb-cli -p 6379 -n 0 DBSIZE
2022/09/08 06:48:09.725908 juicefs[2570293] <ERROR>: can't find block 1/1215/1215859_0_128 for file inode:1003278: file does not exist [fsck.go:146]                                  │(integer) 23076
2022/09/08 06:48:09.742540 juicefs[2570293] <WARNING>: GetAttr inode 978744: no such file or directory [base.go:931]                                                                  │root@vmd96188:~# keydb-cli -p 6379 -n 0 DBSIZE
2022/09/08 06:48:09.742680 juicefs[2570293] <ERROR>: can't find block 1/1186/1186379_0_53610 for file inode:978744: file does not exist [fsck.go:146]                                 │(integer) 18250
2022/09/08 06:48:12.168363 juicefs[2570293] <WARNING>: GetAttr inode 993861: no such file or directory [base.go:931]                                                                  │root@vmd96188:~# keydb-cli -p 6379 -n 0 DBSIZE
2022/09/08 06:48:12.168454 juicefs[2570293] <ERROR>: can't find block 1/1204/1204485_0_160 for file inode:993861: file does not exist [fsck.go:146]                                   │(integer) 147859
  Found blocks count: 95853                                                                                                                                                           │root@vmd96188:~# keydb-cli -p 6379 -n 0 client list
  Found blocks bytes: 10.60 GiB (11378574159 Bytes)                                                                                                                                   │id=5 addr=185.207.106.150:6379 laddr=193.46.243.225:54772 fd=21 name= age=317928 idle=1 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=2
 Listed slices count: 158757                                                                                                                                                          │0528 events=r cmd=rreplay user=(superuser) redir=-1
Scanned slices count: 1132 / 158757 [--------------------------------------------------------------]   1 %8h1m0s                                                                      │id=14456 addr=10.42.0.46:59152 laddr=193.46.243.225:6379 fd=8654 name= age=16 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=2050
Scanned slices bytes: 25.94 MiB (27204449 Bytes)                                    ⠧                                                                                                 │4 events=r cmd=get user=default redir=-1
   Lost blocks count: 447                                                           ⠧                                                                                                 │id=14457 addr=10.42.0.46:59154 laddr=193.46.243.225:6379 fd=8655 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=204
   Lost blocks bytes: 3.73 MiB  (3914867 Bytes)

3. the juicefs-mount container logs a lot of WARNINGs (but no errors):

2022-09-07 22:33:56 | 2022/09/07 22:33:56.495191 juicefs[8] <WARNING>: GET chunks/0/715/715417_0_95201: NoSuchKey:
-- | --
  |   | 2022-09-07 22:33:56 | 2022/09/07 22:33:56.494472 juicefs[8] <WARNING>: GET chunks/0/715/715417_0_95201: NoSuchKey:
  |   | 2022-09-07 22:23:13 | 2022/09/07 22:23:13.776341 juicefs[8] <WARNING>: Upload chunks/0/702/702037_0_88: timeout after 1m0s (try 1) [cached_store.go:396]
  |   | 2022-09-07 22:23:02 | 2022/09/07 22:23:02.739859 juicefs[8] <WARNING>: Upload chunks/0/702/702037_0_88: timeout after 1m0s (try 1) [cached_store.go:396]
  |   | 2022-09-07 22:22:53 | 2022/09/07 22:22:53.027924 juicefs[8] <WARNING>: Upload chunks/0/702/702037_0_88: timeout after 1m0s (try 1) [cached_store.go:396]Show context
  |   | 2022-09-07 22:03:11 | 2022/09/07 22:03:11.882075 juicefs[8] <WARNING>: Secret key is removed for the sake of safety [redis.go:3241]
  |   | 2022-09-07 21:36:52 | 2022/09/07 21:36:52.318823 juicefs[8] <WARNING>: fail to read sliceId 652555 (off:365869, size:528, clen: 366397): get chunks/0/652/652555_0_366397: NoSuchKey:
  |   | 2022-09-07 21:36:52 | 2022/09/07 21:36:52.286568 juicefs[8] <WARNING>: GET chunks/0/652/652555_0_366397: NoSuchKey:
  |   | 2022-09-07 21:36:52 | 2022/09/07 21:36:52.286553 juicefs[8] <WARNING>: GET chunks/0/652/652555_0_366397: NoSuchKey:
  |   | 2022-09-07 21:20:16 | 2022/09/07 21:20:16.384710 juicefs[8] <WARNING>: fail to read sliceId 633991 (off:900647, size:528, clen: 901175): get chunks/0/633/633991_0_901175: NoSuchKey:
  |   | 2022-09-07 21:20:16 | 2022/09/07 21:20:16.363791 juicefs[8] <WARNING>: GET chunks/0/633/633991_0_901175: NoSuchKey:
  |   | 2022-09-07 21:20:16 | 2022/09/07 21:20:16.363791 juicefs[8] <WARNING>: GET chunks/0/633/633991_0_901175: NoSuchKey:
  |   | 2022-09-07 20:57:39 | 2022/09/07 20:57:39.267203 juicefs[8] <WARNING>: The entry of the inode was not found. inode: 501318 [redis.go:2971]

As I am using quite a cheap s3 storage provider I guess the performance is not optimal. This is why I use --writeback option to have everything cached locally first. My expectation would be juicefs is taking care of uploading everything in the background at whatever speed and whatever it takes. But also the massive increase in blocks concerns me as those small blocks are also not beneficial to my already slow s3 provider.

Do you have any advice for my home setup? Perhaps juicefs won't work in this environment. But when there are problems affecting my data integrity juicefs should at least print out some error messages.

And for completeness see attached a screenshot of my grafana dashboard.
Thanks a lot!

[vicaya]

juicefs appears to require a real-time object store to function properly. You probably want to set up a local minio cluster and use mc mirror to sync your local bucket to your remote provider.

[holygrolli]

if it would be like that then the warning logs should definitely be error logs. nothing during runtime lead me to believe there is data loss/corruption

[holygrolli]

and about minio mirror. I just tried to find docs about it. there would be client-side mirroring with mc mirror and server-side mirror which supports multiple scenarios. but from what I understand it's a real mirror and I would need to have the same storage locally. that is no option for me, as this is one of the benefits of juicefs (having a local small cache directory)

[vicaya]

mirror and replication are different concepts for minio. mirror works for any s3 compatible source and target. replication requires a minio source and target.

Checkout the examples in https://docs.min.io/minio/baremetal/reference/minio-mc/mc-mirror.html

[holygrolli]

as I already wrote this mirroring is nothing like the concept of infinite cloud storage. no option for me. but thanks for the idea

[vicaya]

I misunderstood your "the same storage locally" as the same storage type :). If your goal is infinite cloud storage, you can still achieve that with minio tiering if the backend is aws, google, and azure. cf. https://docs.min.io/minio/baremetal/reference/minio-mc-admin/mc-admin-tier.html

[xiaogaozi]

Here are a few questions for you to confirm:

1. What is the network latency from the machine where JuiceFS is mounted to the object storage? By default, the JuiceFS client uploads object storage with a timeout of 1 minute, which you can adjust with the --put-timeout option.
2. How much disk space does the JuiceFS mount option --cache-dir correspond to?
3. Assuming --cache-dir is /var/jfsCache, please check how much space is currently in the /var/jfsCache/rawstaging directory?

After enabling the client write cache, the following points need to be paid attention to:

1. If the asynchronous upload fails, the data blocks on the local disk will not be deleted, but will remain until the upload is successful.
2. The --cache-dir space will continue to grow because the blocks are kept until the upload is successful.
3. If the disk corresponding to --cache-dir is almost full, asynchronous upload will become synchronous upload.

[holygrolli]

Sorry for my long delay in responding. I had no time for this hobby project. So now I set up a new single node k3s cluster additionally with minio and keydb on the same host. So latency should not be a problem any more.
Regarding you cache-dir question this is the output of csi-controller:
Disk cache (/var/jfsCache/7d3408fb-28be-42ea-b3c8-b3676e3212e1/): capacity (102400 MB), free ratio (10%), max pending pages (15)
And last about rawstaging folder:

# du -sch /var/jfsCache/7d3408fb-28be-42ea-b3c8-b3676e3212e1/rawstaging/
648K	/var/jfsCache/7d3408fb-28be-42ea-b3c8-b3676e3212e1/rawstaging/
648K	total

And again I see popping up WARNINGs about The entry of the inode was not found. inode
How can this happen and more interestingly what does it mean? should I be worried?

And what also worries me is the fact the juicefs metrics state there are 8806 files in the moment, but keydb states 17122 keys in the DB. I know files will be splitted into chunks and being uploading while not using a whole chunk. So I guess it is kind of "normal" we have a files and actual keys diverge, but will there ever be any cleanup done automatically? Do I need to run gc on regular basis?

Sorry for bringing all these questions together here, but I really would like to understand if everything is alright with the data