Graceful restart using separate FUSE manager daemon

[ilyee]

As mentioned in #151, if we restart the stargz-snapshotter process, all the related containers need to be restarted because the fuse goroutine is killed.

Some Ideas

For solving this we can use a separate FUSE manager daemon to handle all FUSE servers.

An new implementation of FileSystem interface in snapshot/snapshot.go can be made, which calls FUSE manager daemon through gRPC, so FUSE servers will be managed by manager daemon. Then after snapshotter's restart, containers don't need to be restarted, and stargz images don't need to be restored.

Some Worries

Same issue will happen again when we want to restart the FUSE manager

cc: @ktock

[ktock]

@ilyee What is your use-case of restart? Reloading configuration?

[ilyee]

@ilyee What is your use-case of restart? Reloading configuration?

yes, reloading configuration for current, but no guarantee that there won't be more use-case

[ktock]

@ilyee Having a separate FUSE manager daemon sounds good to me.

We need to make sure that all filesystem configuration are passed to the FUSE manager on each call of mount (or on each restart of snapshotter) for avoiding the necessity of restarting FUSE manage for config reload. One possible way is sending all configuration through gRPC and another one is creating a temporary config file and send the reference to it (similar to config.json of filesystem bundle)

[ilyee]

We need to make sure that all filesystem configuration are passed to the FUSE manager on each call of mount (or on each restart of snapshotter) for avoiding the necessity of restarting FUSE manage for config reload. One possible way is sending all configuration through gRPC and another one is creating a temporary config file and send the reference to it (similar to config.json of filesystem bundle)

Agree with that, I think both way are applyable, I'll make a pr for this soon

[chenk008]

@ilyee @ktock How about saving the fuse pipe FD in the FUSE manager? When we need to upgrade stargz-snapshotter, send this pipe FD to the new stargz-snapshotter process and let the old stargz-snapshotter process quit, so we don't need to remount on upgrading.

[ilyee]

due to completion, the issue is closed now