Possible to open virtual stores made with 2 different icechunk versions (0.2.15 and >1.0.0)?

[abarciauskas-bgse]

👋🏽 @jbusecke and I have been working on 2 icechunk stores for the MUR SST dataset, as documented here (historical) and here (ongoing). I'm wondering if it is possible to open both stores in a single python environment? Here is a notebook where I try to do that with icechunk 1.0.3, but the historical store (created with icechunk 0.2.15) fails with the following error:

IcechunkError: Failed to decode variable 'time':   x no virtual chunk container can handle the chunk location (s3://podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20020601090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc), edit the
  | repository configuration adding a virtual chunk container for the chunk references, see https://icechunk.io/en/stable/virtual/
  | 
  | context:
  |    0: icechunk::store::get
  |            with key="time/c/0" byte_range=From(0)
  |              at icechunk/src/store.rs:198
  | 
  `-> no virtual chunk container can handle the chunk location (s3://podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20020601090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc), edit the
      repository configuration adding a virtual chunk container for the chunk references, see https://icechunk.io/en/stable/virtual/

@paraseba is it possible to open both stores in one python environment or do these stores need to be used separately until we re-generate the historical store with icechunk>=1.0.0?

[TomNicholas]

Have you retrospectively set the necessary virtual chunk container on the older store?

0.2.15 predates #1039, so when that store was originally created, I believe no virtual container was needed.

[abarciauskas-bgse]

Thanks @TomNicholas !

Do you have advice for retrospectively setting the necessary virtual chunk container on the older store? Right now my code:

def open_repo(store_name: str) -> Repository:    
    storage = s3_storage(
        bucket=bucket,
        prefix=f"icechunk/{store_name}",
        anonymous=True
    )

    # we create a container by giving it the url prefix and the store config
    container = VirtualChunkContainer(f"s3://{bucket}/icechunk/{store_name}", store_config)
    
    # we add it to the repo config
    config = RepositoryConfig.default()
    config.set_virtual_chunk_container(container)
    
    # we set credentials for the virtual chunk container
    # repo readers will also need this to be able to resolve the virtual chunks
    credentials = containers_credentials(
        {
            # we identify for which container we are passing credentials
            # by using its url_prefx
            # If the value in the map is None, Icechunk will use the "natural" credential
            # type, usually fetching them from the process environment
            f"s3://{bucket}/{store_name}": s3_credentials(
                access_key_id=ea_creds['accessKeyId'],
                secret_access_key=ea_creds['secretAccessKey'],
                session_token=ea_creds['sessionToken']
            )
        }
    )
    
    # When we create the repo, its configuration will be saved to disk
    # including its virtual chunk containers
    repo = Repository.open(
        storage=storage,
        config=config,
        authorize_virtual_chunk_access=credentials,
    )
    return repo

includes config.set_virtual_chunk_container(container)

Also, would it be an issue that the original store includes both native zarr chunks and virtual chunks?

[paraseba]

No issues with native + virtual chunks, you can even mix them on the same array.

What I recommend is, once you find the correct set of VirtualChunkContainers, you repo.save_config() so you no longer need to pass them again every time you open the repo.

[paraseba]

It may take some tweaking as @TomNicholas mentions, maybe even some config changes, but, it should be possibe

[abarciauskas-bgse]

This is resolved and the issue was user error that @jbusecke helped me identify and resolve. I was setting the VirtualChunkContainer and container credentials on the icechunk store location and not the data store location. Which in retrospect makes a lot of sense 😅 and is described in the Take note of the [VirtualChunkContainer](https://icechunk.io/en/stable/reference/#icechunk.VirtualChunkContainer) Note on the virtual datasets documentation page.