[WIP] Introduce Iceberg management REST API into Kyuubi

[a49a]

Motivation

Since the iceberg has some spark procedures to manage the iceberg table and KPIP-1 has introduced REST API into the Kyuubi. So we can add some iceberg management REST API into Kyuubi. If so users can only use REST API to manage the iceberg table via Kyuubi, and some users also can config iceberg management jobs in their big-data web platform via this REST API.

Design

The KyuubiServer converts JSON parameters of REST API to a Spark SQL of Iceberg procedure, and then it will submit this SQL job.

Need to implement a total of 12 procedures:

• Snapshot management
  • rollback_to_snapshot
  • rollback_to_timestamp
  • set_current_snapshot
  • cherrypick_snapshot
• Metadata management
  • expire_snapshots
  • remove_orphan_files
  • rewrite_data_files
  • rewrite_manifests
• Table migration
  • snapshot
  • migrate
  • add_files
• Metadata information
  • ancestors_of

Implementation

REST API path sample

POST /${version}/sessions/${sessionHandle}/operations/iceberg/procedure/${procedure_name}
Example: /v1/sessions/${sessionHandle}/operations/iceberg/procedure/rewriteDataFiles

REST API JSON example (rewrite_data_files)

{
  "catalog": "spark_catalog",
  "namespace": "db",
  "table": "sample",
  "strategy": "binpack/sort",
  "sort_order_type": "zorder/ordinary",
  "ordinary_order_options": [
    {
      "col_name": "c1",
      "order": "ASC/DESC",
      "null_position": "FISRT/LAST"
    },
    {      
      "col_name": "c2",
      "order": "ASC/DESC",
      "null_position": "FISRT/LAST"
    },
  ],
  "zorder_col_names": [
    "c1",
    "c2"
  ],
  "options": {
      "min-input-files": 2,
      "min-file-size-bytes": 1024
  },
  "where": "id = 3 and name = \"foo\""
}

converted Spark SQL sample

CALL catalog_name.system.rewrite_data_files(
  table => 'db.sample',
  strategy => 'sort', 
  sort_order => 'id DESC NULLS LAST,name ASC NULLS FIRST')

[yaooqinn]

It is better to discussion in the dev mailing list

[a49a]

After I accomplish this draft, I will send a dev mail and refer to the draft in the dev mailing list.

[yaooqinn]

thank you @deadwind4