Use auto DataType instead of Encoder

[jieguangzhou]

Currently, we often need to explicitly specify an Encoder to address the storage of certain data types in a database. However, for most users' needs, they may not need to understand how to set this Encoder. We can set preset DataTypes for common data types in the machine learning field, to automatically handle these logics.

Users no longer need to explicitly worry about how data is converted and stored in the database, as all of this is automatically completed by SuperDuperDB.

For example, when reading data, do not use unpack, because the data users get should be post-unpack. Keep the data the same before insertion and after reading, so users should not need additional operations.

Data Interaction

Inserting data

insert(
	{
		'embedding': np.array(xxxx),
		'label': 1,
		'text': 'xxxxx'
	}
)

Reading data

doc: Document = query(xxxxxx)
doc.data
>>  {
		'embedding': np.array(xxxx),
		'label': 1,
		'text': 'xxxxx'
	}

Model Output Interaction

When inserting model output data

class FakeEmbeddingModel(Model):
	def predict(self, x, **kwargs):
		output = np.array(xxx)
		return output

class FakeClassificationModel(Model):
	def predict(self, x, **kwargs):
		label_id = 1
		prob = 0.7
		probs = np.array([0.1, 0.7, 0.2])
		return {'label': label_id, 'prob': 0.7, 'probs': probs}

We need to automatically detect the model's results and convert them into specific data that the database can store.

When reading model output data

doc: Document = query_outputs('FakeEmbeddingModel')
doc.output('x', 'FakeEmbeddingModel')
>> np.array(xxx)


doc: Document = query_outputs('FakeClassificationModel')
doc.output('x', 'FakeClassificationModel')
>> {'label': 1, 'prob': 0.7, 'probs': np.array([0.1, 0.7, 0.2])}

We need to automatically restore the data.

DataType (Currently Known as Encoder)

Preset DataType

Class DataType: 
	def encode(self, data):
		return data
	def decode(self, data):
		return data
	def identify(self, data):
		# return True or False

Class Int: ...
	def identify(self, data):
		return isinstance(data, int)
		
Class Float: ...
Class String: ...
Class Dict: ...
Class Numpy: ...
Class Torch: ...
Class DataFrame: ...
Class URL: ...

Custom DataType

This is similar to our current Encoder. Users can set DataType to convert data according to their preference. If set, the user-defined DataType will be used; otherwise, our preset DataTypes will be used.

[blythed]

@jieguangzhou agreed. However I have concerns about performance. Suppose that I have a very large update. Then I have
to do a lot of work to inspect all of the objects in it and find their types.