[Core] Making Object Store Fallback Directory Configurable

[MengjinYan]

Why are these changes needed?

Recently we investigated issue where a job ran out of disk because the fallback directory is full. Currently, the fallback directory is set to /tmp folder which might have a size constraint and is not able to let the job to use the full disk when it is necessary.

This PR make the fallback directory configurable so that it can be configured to a larger disk space when it is needed. The configuration is done by setting the --fallback-directory option in the ray start command.

The PR is manually tested:

1. When running ray start without the config:

ray start --head

The fallback directory is set to /tmp in raylet.out:

[2025-03-08 19:50:31,664 I 31368 35096955] (raylet) dlmalloc.cc:321: Setting dlmalloc config: plasma_directory=/tmp, fallback_directory=/tmp, hugepage_enabled=0, fallback_enabled=1

2. When running ray start with the config:

ray start --head --fallback-directory="/tmp/fallback"

The fallback directory is set to /tmp/fallback in raylet.out:

[2025-03-08 20:23:25,279 I 61733 35267407] (raylet) dlmalloc.cc:321: Setting dlmalloc config: plasma_directory=/tmp, fallback_directory=/tmp/fallback, hugepage_enabled=0, fallback_enabled=1

Related issue number

N/A

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(