file access conflict for shim binaries

[phyrog]

In some cases, the node installer exits with this error:

ERROR failed to install error="failed to install shim 'spin-v2': open /mnt/node-root/opt/kwasm/bin/containerd-shim-spin-v2: text file busy"

This happens when containerd has already started a shim process with that binary and the installer wants to overwrite the binary file.

I believe we need to change the behavior of always copying the shim binaries and then detecting if something changed to first detect a change and then copy the binary:

runtime-class-manager/cmd/node-installer/install.go

Line 81 in 6f849a8

binPath, changed, err := shimConfig.Install(fileName)

[phyrog]

Generally this begs the question if we even can update shim binaries with the same name (i.e. same major version)

[tillknuesting]

Can we rename the old binary and place the new binary after it?

[phyrog]

Maybe, I'll try it later. Alternatively we could include a hash of the file in the name and only need to update the containerd config with the new binary path.

[toksikk]

@phyrog and I discussed the problem and decided to append a hash to the binary as an interim solution. Unfortunately, we couldn't come up with a good solution to avoid accumulating trash binaries after multiple updates.

According to https://iximiuz.com/en/posts/implementing-container-runtime-shim/ and some tests using the test manifests from #65 (comment), the shim process only lives if there is a workload, we can't just delete any unused binary by simply using lsof to check if there is a process still accessing it.

This issue still needs to be addressed.

[rajatjindal]

when we update the shim-binary, what is the expected (and current) behavior of workloads which are currently running?

• should they be restarted? (because we have new shim version) or
• should they continue to use current shim version until restarted by an external event (e.g. k8s pod restart/reschdule etc).

in my opinion, we should force the workloads to restart somehow when we have new version. but also look forward to hearing what others think about this and what is the current behavior.

[phyrog]

For major version updates, existing workloads should be unaffected (runtime class name includes the major version), minor/patch version update handling is currently undefined, afaik.

As for what the handling should be: I don't know, but I suppose we could make the behavior configurable via the CRD.

[devigned]

We had considered this question a while back in https://github.com/deislabs/containerd-wasm-shims/blob/main/proposals/2022-09-14-kubernetes-versioning.md. The bin names should probably have the version encoded in the file name. Then the only concerns are how to clean up old versions and when to update the containerd config mapping the runtime handler to the version specific file name.

Replacing the same file name shim-{version} with another of the same name, but containing different compiled code seems misleading. If that is a goal, then shim-{sha} seems more appropriate.

[voigt]

I think I "accidentally" addressed the issue in #68.

I'm renaming the shim binary to containerd-shim-<shim-resource-name>. As the shim resource in Kubernetes is unique, the binary name will be unique as well.

IMO this approach makes debugging easier, as the relation shim-resource-in-cluster <-> binary-on-node is clear.

This technically allows to install the same shim twice on the same node, but address it with a different runtimeclass (cool if you want to run a debug build of the same version).

---

Edit: Argh, it is too late ;) I'm rereading the conversation and realize, that it does not actually fix the "update" scenario:

Shim Resource installed:

name: spin-v2
source: https://github.com/spinkube/containerd-shim-spin/releases/download/v0.14.1/containerd-shim-spin-v2-linux-aarch64.tar.gz

Updating Shim Resource with new version of shim:

name: spin-v2
https://github.com/spinkube/containerd-shim-spin/releases/download/**v0.15.0**/containerd-shim-spin-v2-linux-aarch64.tar.gz

Will then lead to the mentioned error.

However, in a prod scenario, I'd rather run two versions of the same runtime and switch Apps one by one to the new version, as this leaves me with a roll-back option.