[FEATURE] Write metadata to index mappings _meta as read cache for frontend user to access

[seankao-az]

Is your feature request related to a problem?
Write the following to Flint index mappings _meta field so that frontend user can use it as a read cache for crucial metadata access.

• refresh interval
• source table names
• last refresh time

What solution would you like?
There’ll be a _meta field in the index mappings, within it will be a properties field, which contain the following fields:

• metadataCacheVersion
  • type: String
  • value will be constant "1.0"
• refreshInterval
  • type: Int
  • unit: seconds
  • this field is added only if index refresh type is auto refresh and refresh_interval option is set
• sourceTables
  • Array of String (table names)
• lastRefreshTime
  • type: Long
  • unit: milliseconds
  • this field is added only if index already gets refreshed at least once

Example:

GET myindex/_mappings

{
  "myindex": {
    "mappings": {
      "_meta": {
        "properties": {
          "metadataCacheVersion": "1.0",
          "refreshInterval": 900,
          "sourceTables": [
            "mys3.default.table1",
            "mys3.default.table2"
          ],
          "lastRefreshTime": 1727395328283
        }
      }
      "properties": { // existing OpenSearch index mapping
        "field1": ...
        "field2": ...
      }
    }
  }
}

Incoming changes

1. Writer class to handle the metadata cache writing; flow control
2. Add source tables to index metadata for materialized view
3. Track last refresh time for index
   1. Out of Scope: track each micro batch execution time for internal scheduler (spark streaming query)