[DATATYPE] Refactor data-types

[blythed]

Issue is that not all output types are supported by every database, and to support certain operations, the data needs to be in the correct format.

@kartik4949 @jieguangzhou to provide input.

• [DATATYPE] Auto-infer _fields based on annotations File, Blob, etc. #2649
• [DATATYPE] Create mapped datatype for flattened Listener #2499
• [DATATYPE] Migrate CFG.bytes_encoding to databackend #2628
• [DATATYPE] Create base DataType without all parameters and use it for "switch" Vector type #2629
• [DATATYPE] Create input-built datatype to handle pickle, dill etc.. #2634
• [DATATYPE] Bring _artifact_schema inline with _schema #2635
• [DATATYPE] Deprecate the concept of lazy in encodables #2636
• [DATATYPE] Make bytes_encoding databackend dependent and only for inline data #2641

[blythed]

1. Artifact and File should not change depending on CFG.bytes_encoding. This is because they are writing to file.
2. ...

[jieguangzhou]

The superduper-framework only provides the most basic datatypes:

• Native
  • int
  • str
  • bytes
  • dict (json)
  • …
• Vector
• python_obj (default: pickle)

Plan 1

For different databackends, we can configure datatype conversion relationships, such as:

• ibis:

  • dict → json
  • vector → sqlvector

• postgres:

  • vector → pgvector

Thus, datatype conversions will occur in the following areas:

• create_table
• insert data
• query data

Example: JSON

insert data

input_data = {"data": {"a": "b"}}
input_schema = Schema({"data": "dict"})

schema = _convert_schema(input_schema)
## schema = Schema({"data": "json"})

encode_data = schema.encode(input_data)
encode_data = {"data": '{"a": "b"}'}

query data

input_data = {"data": '{"a": "b"}'}
input_schema = Schema({"data": "dict"})

schema = _convert_schema(input_schema)
## schema = Schema({"data": "json"})

decode_data = schema.decode(input_data)
decode_data = {"data": {"a": "b"}}

Plan 2:

We also define this through configuration files within the preset datatype definitions.

datatypes:
  vector: ibis.datatype.sql_datatype
  # or vector: postgresql.datatype.sqldatatype
  dict: json

class Vector:
    def __post__init(self):
        datatype_config = CFG.xxxx
        obj = cls(...)
        if self.class .__name__ in datatype_config:
            cls_real_datatype =  # import the real_datatype class
            self.real_datatype = cls_real_datatype(...)

        else:
            self.real_datatype = None

    def encode_data(...):
        return (self.real_datatype or self).encode_data()

    def decode_data(...):
        return (self.real_datatype or self).encode_data()

[jieguangzhou]

Regarding CFG.bytes_encoding, if it’s set to base64, then for all datatype.encode_data output:

If the data is bytes, convert it to base64 and add the prefix BASE64:.

Then, during decode_data:

If the input data is a str with the BASE64: prefix, convert it back to bytes.