-
Notifications
You must be signed in to change notification settings - Fork 0
Phone Conference, November 10th 2011
The discussion started the week before continued. Behavior on NFS client/server in HPC environment was discussed, with focus on "what happens if a server restarts ?". Making kind of "internal restart" (by pausing/restarting the FSAL) may lead to a high level of complexity, making the code unstable. On the other hand, clients do have retry logic (stateless NFSv3) and recovery mechanism (NLMv4+NSM logic in NFSv3 and pure NFSv4 recovery mechanism). So, as a first approach, it's probably wiser to simply use a "restart" on the server, making the assumption that the client will handle this properly, for example in the case the configuration file for ganesha has changed a lot. For minor changes (more clients added to an export entry, or clients removed for example) that do not impact the cache or the FSAL (typically changes to the export entries), signal SIGHUP can be sent to GANESHA,forcing it to reread part of its configuration. This feature (added by IBM) works pretty well. Anyway, a "restart" logic must be based on a clean shutdown, so 'kill -KILL' should be avoided. Instead of this, SIGTERM should be preferred, causing the daemon to quiesce gently. To be noted: there is no way, in Ganesha, to detect if a change to the configuration requires the daemon to restart or if sending SIGHUP is enough.
Adam now uses IBM's state manager to deal with layout, in correlation with his FSAL_CEPH. He will use an API similar to the one suggested by Boaz at BAT. Code should be available for comments/reviews/tests in a few weeks. Tests will be made on it for GPFS (by IBM) and LUSTRE (by CEA).
The core question was "what if I invalidate a file which has associated states like opened FDs or locks ?". Various strategies exists, all having advantages and inconveniences : keeping it in cache, knowing it's no more coherent (with the use of a refcounter to get ridd of it as the last state is revoked), repopulate it immediately and attached formerly acquired states to the new version. Question is still opened, but it clearly appears that invalidating a directory or symlink and invalidating a file are quite different jobs. LinuxBox stated that Cache Inode GC was becoming an issue when running on files with states (that should not be GCed until the last state is revoked). They will work on not perform GC on this entries, making the garbagge collector aware of open states by looking at the cache_entry internals.
To be keep in mind: invalidation may occur when the file is removed (it's unlinked), but as well when one of its metadata is updated (setattr). This second case if trickier because the entry will come back in the cache and a clear strategy needs to be defined to handle this "transition".
Decision was taken to work on a state diagram describing all cases for a cached file with/without attached states and showing how states transition are to be handled. Someone (I think it was Frank (to be confirmed)) said he found kind of document that brings elements on this.
- tests to be used to defined a common test platform.
- NFSv4 lock#2 test
- State Diagram
- IBM thread refactoring