Stac MlM metadata drafting

[kshitijrajsharma]

Setup a minimal stac mlm item for the models that will be integrated, at least minimal compulsory field should be defined to fully support Current UI as well as upcoming implementation including field like categories , mlflow specific entry points , dockerfiles etc etc . It should be researched and documented well enough for someone else to fill those out !

[kshitijrajsharma]

We need to check what is available currently : https://stac-extensions.github.io/mlm/v1.5.1/schema.json
& we need to compile with what fair uses : https://fair.hotosm.org/ , so we can migrate !

[kshitijrajsharma]

Supported categories for the model :

• Instance segmentation ( mAP @ 50 )
• Semantic segmentation ( IOU , f1 , if it is binary or multiclass )
• Object detection ( for eg : mAP@50 )

We need to define the common metrics for each of them !
We also need to define how confusion metrics would be computed !

We need to setup the standards for each of those categories !

[AbdelrahmanKatkat]

Green Box = Must Add
Yellow Box = Optional
Blue Box = Custom (Not MLM element to begin with )

[AbdelrahmanKatkat]

Metadata Table:

Parameter	Category	Example (schema)	Why
stac_extensions (root)	Must add	["https://stac-extensions.github.io/mlm/v1.5.1/schema.json"]	Schema requires stac_extensions to contain the MLM schema URL so validators apply MLM rules.
mlm:name	Must add	"building_seg_v1" (pattern: ^[a-zA-Z][a-zA-Z0-9_.\-\\s]+[a-zA-Z0-9]$)	Required by MLM schema. Human-readable model identity for governance, UI, and search. Map: model identity in STAC DB.
mlm:architecture	Must add	"Vision Transformer" or "ResNet"	Required by MLM schema. Describes model structure for governance and runtime expectations. Map: “model code structure” validation.
mlm:tasks	Must add	["semantic-segmentation"] (enum: regression, classification, detection, segmentation, etc.)	Required by MLM schema. Defines model category (e.g. semantic-segmentation). Map: “model category” in governance and downstream routing.
mlm:input	Must add	[{"name":"imagery","input":{"shape":[1,4,256,256],"dim_order":["batch","channel","height","width"],"data_type":"uint16"}}]	Required by MLM schema. Defines input contract (shape, dim_order, data_type). Map: “input/output interface” validation and contract tests.
mlm:output	Must add	[{"name":"mask","tasks":["semantic-segmentation"],"result":{"shape":[1,1,256,256],"dim_order":["batch","channel","height","width"],"data_type":"uint8"}}]	Required by MLM schema. Defines output contract. Map: “input/output interface” and interpretable outputs for data registration.
Asset: roles (model asset)	Must add	["mlm:model"] (at least one asset must have roles containing "mlm:model")	Schema requires at least one asset to define the model role; that asset must also have mlm:artifact_type.
mlm:artifact_type (on model asset)	Must add	"torch.save" or "safetensors.torch.save_file"	Required on any asset with role mlm:model. Tells how to load the artifact (e.g. torch.save). Map: reproducible build and registration.
ModelInput: name	Must add	"imagery" (string, minLength 1)	Required by schema per input. Identifies the input in the contract.
ModelInput: input (shape, dim_order, data_type)	Must add	{"shape":[1,4,256,256],"dim_order":["batch","channel","height","width"],"data_type":"uint16"}	Required by schema. Defines the tensor/array contract for validation and testing.
ModelOutput: name	Must add	"mask" (string, minLength 1)	Required by schema per output. Identifies the output.
ModelOutput: tasks	Must add	["semantic-segmentation"] (same enum as mlm:tasks)	Required by schema per output. Links output to task type.
ModelOutput: result (shape, dim_order, data_type)	Must add	{"shape":[1,1,256,256],"dim_order":["batch","channel","height","width"],"data_type":"uint8"}	Required by schema. Defines output structure for tests and downstream use.
mlm:framework	Optional add	"PyTorch" or "TensorFlow" (enum) or any string matching pattern	Not required by schema. Clarifies runtime (PyTorch/TensorFlow/etc.) and improves reproducibility. Map: build/test and deployment.
mlm:framework_version	Optional add	"2.1.0" (semver pattern)	Not required. Pins framework version; avoids “works locally, fails in CI/prod”. Map: reproducibility.
mlm:total_parameters	Optional add	2500000 (integer, minimum 0)	Not required. Helps capacity planning and cost. Map: ops and scaling.
mlm:memory_size	Optional add	1073741824 (integer bytes, minimum 0)	Not required. Helps sizing and avoiding OOM. Map: contract tests and production.
mlm:pretrained / mlm:pretrained_source	Optional add	true and "ImageNet" or null (if trained from scratch)	Not required. Documents training lineage for governance and transparency.
mlm:accelerator	Optional add	"cuda" or null (enum: amd64, cuda, xla, amd-rocm, intel-ipex-cpu, intel-ipex-gpu, macos-arm)	Not required. Ensures contract tests and prod run on correct hardware. Map: build/test and promotion.
mlm:accelerator_constrained	Optional add	false (boolean, default false)	Not required. Makes it explicit if model is GPU-only (or similar).
mlm:accelerator_summary / mlm:accelerator_count	Optional add	"1x NVIDIA A100" (string) / 1 (integer, minimum 1)	Not required. Extra hardware detail for ops.
mlm:batch_size_suggestion	Optional add	8 (integer, minimum 0)	Not required. Suggests batch size for the chosen accelerator.
mlm:compile_method (asset)	Optional add	"aot" or "jit" (string, minLength 1)	Not required. Documents AOT/JIT etc. for advanced runtimes.
mlm:entrypoint (asset)	Optional add	"my_package.my_module:inference_function" or "inference.py"	Not required. Points to inference entry (e.g. script:function). Map: reproducible inference. Must be on an asset with role code if present.
ModelInput: description	Optional add	"4-band Sentinel-2 tile" (string, minLength 1)	Not required. Human-readable description of the input.
ModelInput: bands / variables	Optional add	["B02","B03","B04","B08"] or [{"name":"B02"}]; if used, dim_order must include bands or variables	Not required. Clarifies channel meaning; needed if you use EO/Raster/Datacube. Map: data registration and interop. If used, dim_order must include bands or variables and you need band/variable definitions elsewhere.
ModelInput: value_scaling	Optional add	[{"type":"min-max","minimum":0,"maximum":255}] or [{"type":"z-score","mean":0.5,"stddev":0.2}] (also clip, offset, scale, processing)	Not required. Describes normalization (min-max, z-score, etc.). Strongly recommended so inference matches training. Map: contract tests.
ModelInput: resize_type / pre_processing_function	Optional add	"interpolation-linear" (enum: crop, pad, interpolation-, wrap-); processing:expression or array	Not required. Describes resize/preprocessing for reproducible input.
ModelOutput: description	Optional add	"Binary building mask" (string, minLength 1)	Not required. Human-readable description of the output.
ModelOutput: bands / variables	Optional add	Same shape as ModelInput (array of string or {name, format?, expression?})	Not required. Useful when output has band/variable structure.
ModelOutput: classification:classes	Optional add	Ref to classification extension schema; or [] for non-class outputs	Not required. Maps class indices to names (e.g. 1 = building). Map: interpretable outputs and data registration.
ModelOutput: post_processing_function	Optional add	processing:expression or array of expressions (or null)	Not required. Documents NMS, thresholding, etc.
mlm:hyperparameters	Optional add	{"learning_rate":0.001,"epochs":50} (object, keys: ^[0-9a-zA-Z_.-]+$)	Not required. Key training/inference knobs for reproducibility.
eval_metrics (custom)	Optional add (custom)	e.g. {"mIoU":0.82,"per_class":{"building":0.85}} (your structure)	Not in MLM schema. Use for evaluation metrics. Put in properties; do not use mlm:metrics. Map: model quality and promotion.
archived (custom)	Optional add (custom)	true (boolean)	Not in MLM schema. Use to mark deprecated models. Put in properties; do not use mlm:archived.
superseded_by (custom)	Optional add (custom)	"model-item-id-v2" (STAC Item id string)	Not in MLM schema. STAC Item id of replacement model.
archived_reason (custom)	Optional add (custom)	"Replaced by v2" (string)	Not in MLM schema. Short reason for deprecation.
mlm:archived	Don't include	—	Not defined in MLM schema. Unknown mlm:* keys are disallowed. Use custom key archived without mlm: prefix.
mlm:metrics / mlm:eval_metrics	Don't include	—	Not in MLM schema. Use custom key eval_metrics in properties instead.
Any other unknown mlm:* key	Don't include	—	Schema allows only defined mlm:* properties. Custom fields must not start with mlm:.
mlm:artifact_type in Item properties	Don't include	—	Schema allows mlm:artifact_type only on assets, not on the Item. Put it on the asset with role mlm:model.
mlm:input / mlm:output on an asset	Don't include	—	Schema allows these only on Item properties, not on assets.
mlm:name / mlm:hyperparameters on an asset	Don't include	—	Schema disallows these on Asset properties (mlmAssetFields).

---

[kshitijrajsharma]

Few points :

On the table :

• lets add one more column called constants or defaults, as some of the stac items would be constraints for eg input output for now !
• lets add one more column for the validation ( that should be listed )
• may be Shrink down table in two categories : Required and Optional !

On the metadata :

Constants with defaults :

• stac_extensions ; provide constant and defaults ( user doesn't need to supply this )
• mlm:input -> constant
• mlm:output - > constant

Required :

• mlm:name ->
• mlm:architecture -> validation
• mlm:tasks -> validation
• model:input - >
  -model:output ->
• mlm:framework -> validation
• mlm:total_parameters
• mlm:memory_size
• mlm:accelerator -> validation
• mlm:accelerator_contrained -> validation
• mlm:hyperparameters -> validation for default values
• mlm:batch_size_suggestion -
• mlm:entrypoints - > validation for training and inference ( ideally zenml function names on submitted python file )
• all assest level keys - > validation for model checkpoint , training data source (optional) if added
• eval_metrics ( key and definition should be included for each metric )
• status : draft , published , archieved -> validation
• version -> validation ( semantic versioning ? )

Optional

• mlm:framework_version
• mlm:total_parameters
• mlm:pretrained
• mlm:pretrained_source
• mlm:accelerator_summary
• preprocessing function
• postprocessing function

Not needed

• mlm : accelerator_count ( only 1 is supported atm )

Missing :

• tags ( requires validation )
• license ( requires validation )
• training_time ( with details of area / per image or some metric )
• inference_time ( per one tile ) with hardware details
• model provider, org details
• contact / paper info
• dockerfile
• container image (if available)
• thumbnail ( can be optional )
• loss function
• training_dataset ( info ) , perhaps also linked to asset ( optional )
• basemodel, backbone info ( frozen ? not frozen )
• total_trainable_params

Discussion :

• may be documentation link ???
• model_release_date
• how do we handle model description ? risks , limitation to be provided
• Inference would be done via zenml , how to handle live prediction though ? offline prediction and training can be done
• Should we include the zenml config yaml in the metadata ? along with zenml python file , that will help us pinpoint which file to pull which dockerfile to swamp up and which config to load ! (https://docs.zenml.io/concepts/steps_and_pipelines/yaml_configuration) or it can be another way around we fill up the stac based on the config
• After this , how would visual of the model card would look like ? what kind of model card we can build ?

[kshitijrajsharma]

One more missing item :

• Location ( optional ) for the model , geojson ( point / polygon or bbox )