Add support for fully hibernate-able Wasm instances

[mamidon]

I'm looking to build a runtime, based on WebAssembly, which provides image-like semantics. This requires persisting the state of WASM instance(s) across process boundaries.

If wasmi_cli were to support this, it might look sort of like this:

wasmi_cli counter_module.wasm --instantiate counter_instance.bin ## all runtime state exists on file system
wasmi_cli counter_instance.bin --invoke increment_and_print
> 1
wasmi_cli counter_instance.bin --invoke increment_and_print
> 2

whereas today, without persisting runtime state you'd get something like this:

wasmi_cli counter_module.wasm --invoke increment_and_print
> 1
wasmi_cli counter_module.wasm --invoke increment_and_print
> 1

This doesn't actually require pre-emption, which seems to be blocked, 'just' the ability to serialize instance runtime state & create an instance via that state vs. the usual module -> link -> instantiate flow.

Is this something WASMI would be interested in supporting? If so, I'd be happy to contribute it myself.

[Robbepop]

Hi @mamidon ,

thank you for the feature request.

Wasmi currently supports resumable functions to a limited extend when used as library. It should be possible to support fuel metering for resumable functions in Wasmi in order to make it possible to use fuel as a kind of ticker and refuel the process continuously.

Would this satisfy your use case?
I am very willing to add this feature or accept contributions that do so as long as it does not regress performance of Wasmi in any way significantly.

I am hesistant to add this functionality to the Wasmi CLI though since that would require to come up with binary formats and (de)serialisation support which is hard to get right and a huge amount of work imo.

[mamidon]

Howdy @Robbepop,

I have no particular requests for Wasmi's CLI, the above was just an example of what would be possible.

My only requirement (spec wise, obviously this is your project!) is the ability to serialize state between calls. This implies storing all abstract WebAssembly state (Store, Instance memory, etc) plus enough Wasmi specific state to rehydrate. Supporting resumable functions 'just' means more state to persist & restore.

Using the library as an example, loosely this might look like:

let engine = Engine::default();
let module = Module::new(&engine, &wat::parse_str(...)?)?;
let mut store = Store::new(&engine, 42);
let mut linker = <Linker<HostState>>::new(&engine);

configure_imports(&mut linker);

let instance = linker
  .rehydrate("some_file.bin", &module)?; // do not call start!

rehydrate cracks open a Wasmi specific file, restores relevant state into the Store, and instantiates the WASM instance (skipping most WASM startup logic, this should be transparent to the instance), restores linear memory seamlessly.

You're correct that figuring out exactly what all this state is & how to serialize it is a heavy lift. I'd be happy to work out a detailed design (including what supporting resumable functions requires) and with your sign off, subsequently contribute the implementation.

WRT preformance regressions, I don't think this should be a problem -- serializing & deserializing should happen out of band to WASM execution, although I'm not so certain that support for resumable functions won't require runtime book keeping.

[Robbepop]

Okay, I see the difference is that resumable functions only persist state of a single execution and what you want is to persist the state of an entire Wasm instance.
This is going to be a huge amount of work from what I can tell due to the aforementioned work items.

Also this is probably not going to support resumable functions as to not introduce inefficiencies in the executor. Also I think it is fine not to support resumable function calls as a resumable function call is simply one that was stopped but all of its altered state has been written down when it did. And this state is what we capture. Maybe it is possible to support resumable functions without introducing performance regressions.

At this point though I would not want to promise you to merge any of that. So in the worst case you'd end up with a fork of Wasmi that supports hibernatable Wasm instances and I am sure it would work. In the best case the work is great, docs are fine, it does not introduce perf regressions and we find it useful enough for the greater community to merge it to Wasmi.

Currently Wasmi master is in beta phase and I do not want to introduce any major features at the moment and just focus on bug fixes and stabilizations until the beta phase is over which should hopefully happen soon(TM).

So under these conditions feel free to do the work and let's see where this is heading. :)

[mamidon]

Outstanding, I hope you've had a wonderful Easter weekend! I got the memo last week that I'm going to be laid off, so I'll sort out my $day_job$ situation first. But after that, I'll whip up a prototype of this. Thanks!

[Robbepop]

Outstanding, I hope you've had a wonderful Easter weekend! I got the memo last week that I'm going to be laid off, so I'll sort out my dayjob situation first. But after that, I'll whip up a prototype of this. Thanks!

I am sorry to hear about your job situation and wish you all the best to sort it out. Feel free to re-open when you are ready again.

[mamidon]

Don't worry, I've already got a few interview loops going. I'm just a stickler for remote, and refused to return to office lol.

[mamidon]

'Right, I'm back on this. I think I've got a reasonable approach, from the POV of WASM standards.

Persisting a WASM Instance, between function calls -- no state on the implicit stack, is equivalent to taking the module definition and changing it as follows:

1. remove any imported memory segment definition
2. remove any original passive or active data segments
3. remove any original element segments
4. any dirty memory pages are replaced with active data segments
5. export a memory with required size characteristics
6. add an elemns segment which holds entries for any runtime items, e.g. tables

I think that's about it.

Roughly, the public API in WASMI might look like:

1. Add to_module(&self) on Instance.. I think I can do this without requiring a reference to the original Module
2. Add to_segments(&self) on Module

Actually persisting the module to disk is left as an exercise to the reader, but it should be straightforward to create a fully valid WASM file with all the segments, or one which doesn't have redundant data (like code) if you have the original module file.

[kajacx]

I am trying to use WASM for video game modding, so I will have to persist instance state across save&load sooner or later. But I'm not sure I understand what the problem is, can't you just save the instance's memory as bytes and then load those bytes later? You can do that right now with the existing memory API (assuming the host can (re)allocate memory of arbitrary size).

I have no idea how resumable functions work, though. I have thought about two use cases: first, host imported function could "return early", returning the execution flow directly to the caller without going through the wasm code, while informing the caller that this has happened. The caller could then resume from that point at a later time. This could be very useful for setting debugging breakpoints.

Second, when running out out of fuel, the caller would once again be infromed that this has heppened and could resume from that point after inserting more fuel. I have no idea if fuel works like this, I have never used it yet.

If I understand this correctly, the issue is then to correctly persist these "save points" from which the host can resume later?