How to enable only write cache?

[mangorilla]

I have an app that writes about 500G data (to about 10k files) each day. The app only writes (only append to existing files), never reads. It writes in small chunks ranging from 1k to 10m size continuously 24 hours.

I am trying to use JuiceFS + aws S3 to store such data. I used this set of parameters in fstab:

_netdev,writeback,cache-size=10240,buffer-size=1024,upload-delay=15s

I only give 10G cache size since my app do not reads. It worked great at first. However, after the cache is full (du /var/jfsCache > 10G), there appears to be a lot of read and write operations on the disk. I assume it is JuiceFS organizing cache? IOPS and iowait gets very high sometimes that even the ssh session becomes laggy.

I tried to disable read cache by setting cache-size=0, while keeping the writeback option:

_netdev,writeback,cache-size=0,buffer-size=1024,upload-delay=15s

However, after this there seems to be no write cache either. Directory /var/jfsCcahe is absent, and the write operations in the app become much slower (100x slower. I assume it's the app writing to S3 directly?).

Because the app is writing in small chunks, I am hoping to buffer the writes and upload them in batches. The read cache seems to be causing a lot of unnecessary IO operations. Is there a way to disable read cache but only enable write cache?

Environment:
juicefs version 1.0.0-beta2+2022-03-04T03:00:41Z.9e26080
Ubuntu 18.04.6 LTS

[davies]

When upload-delay is used, JuiceFS writes the data into disk, then read them back and upload them to S3 later. So there will be some read IO on the disk.

If you want to disable cache for read, please use a small cache-size, for example, 100 , which means 100MB.

Is swap enabled on your system? Can you check how much memory is used by JuiceFS? We have fixed an issue on memory consumption in writeback mode: #1558

[mangorilla]

Thank you for your reply.

The system has 16Gb RAM. Swap was disabled. Sorry I don't have an exact number of JuiceFS's memory usage. I estimated JuiceFS used about 4~6Gb RAM. Total RAM usage never exceeds 90%, and was 50%~80% most of the time.

I have decided to switch to storing directly to S3. I.e. Write to local disk first, and upload the files of previous day to S3 after 24:00.

Maybe it is not the right way to use JuiceFS for this type of workload (as well as other filesystem mounted as a layer upon S3, like AWS File Gateway). Because the objects stored in S3 cannot be modified, appending a file to S3 means discarding the object and creating a new one. After a lot of appending, there will be enormous waste on object storage. After just one day's test, there were more than 20m PUT/COPY/POST/LIST requests and 10m GET requests. As comparison, only about 10k files were written in total. After switching to S3 directly, number of requests reduced to about 20k each day, which is normal. For this type of workload, maybe it is better to write to local disk first, and batch upload after all the writing is done. Then JuiceFS can be used to read all the data stored in S3.

[davies]

JuiceFS Cloud can combine the slices from multiple files into single PUT request, but JuiceFS community can't. Can you try the JuiceFS cloud [1] (hosted), and check whether it works for you?

https://juicefs.com/