Create an API for Pre-processing layers

[jsturtevant]

We've added the ability to pre-compile layers in #405 and followed up with some improvements in #492.

During the iterations it was identified that a similar API could be used to do other pre-initialization work as well:

• composing multiple layers into a single layer
• pre-compiling (as is already done but could be done in this api instead along with other processing)
• using tools like Wizer to do other pre-initialization optimizations

I would like to propose we create an API similar to the precompile but would replace it with something like process_layers or initialize_layers . This should also allow the shim to give us processed layers that are combined and the id's of ones the want to remain untouched.

One possible way to do this would look something like the precompile layer (open to suggestions):

struct ProcessedLayers {
  layers: Vec<Layer>
  unique_cache_id: Option<String> // return None if nothing to do
}

enum Layer {
 Original(digest)  // for files we want untouched
 Processed(Vec<u8>)
}

fn precompile(&self, _layers: &[WasmLayer]) -> Result<ProcessedLayers> {
        // do work
}

On the shim side we could take the layers create a new file that stores these digests and link it to the original image. Upon reloading the image we could look up the file, load those digests.

Questions:

• in Pre-compile each layer individually #492 (review) we add ability to re-use layers that where already processed. How to accomplish that in this scenario?
• is this an evolution of precompile api, do we add new and depreciate, or keep both?