IPCC region extraction for multiple models

[rswamina]

I am writing a recipe to extract all IPCC land regions (around 44 region extractions) and perform some basic preprocessing on them - regridding, temporal statistics. I need this applied for different seasons and multiple models. What would be the best way to do this and avoid memory issues? I find that including 10 models with 3 such preprocessors on JASMIN's sci machines makes the program hang. I am able to configure dask options but am wondering if I should split the region extractions across recipes or do fewer models or such as well ?
An example preprocessor function with just one region would be:

preprocessors:
  preproc_extract_region_land_annual_mean:
    mask_landsea:
      mask_out : sea
    regrid:
      target_grid: 1.0x1.0
      scheme: linear
    annual_statistics:
      operator: mean
    extract_shape:
      shapefile: ar6
      decomposed : True
      method: contains
      crop: True
      ids:
        Acronym:
          - NSA

@bouweandela - could you suggest how best to do this?

[bouweandela]

Could you please share the complete recipe (except the diagnostic scripts if those need to stay confidential) so I can try to reproduce the issue?

[valeriupredoi]

careful extracting NSA 😁 🇺🇲

[rswamina]

@bouweandela - Oddly enough this is working now. I don't know if this failed due to issues with JASMIN previously. I want to try a couple more things before closing this issue but will post back if I find a memory issue. Somehow I was sure I would need to use a dask configuration for this. If all works well, will close this issue shortly.