Idiomatic DML/CRUD?

[bellwether-softworks]

Sorry, I'm not quite sure how to frame that title, but in a nutshell, as my data changes I frequently find myself having to tear down and replace data wholesale. My current pattern is basically:

1. ctx.deregister_table("my_first_table")
2. Create new MemTable using latest dataset
3. ctx.register_table("my_first_table", Arc::new(mem_table))?

Incidentally, this works for me for most cases, since a lot of my data is static, but there are some datasets I am frequently modifying, and it feels "wasteful" to have to scrap entire tables if all I'm doing is deleting or modifying the occasional record.

Am I looking at this wrong? Is there a "better" way to handle basic CRUD operations (including using the SQL interface); if not now, is it planned, or am I misusing/abusing DF by trying to do this in the first place?

[alamb]

Here are some ideas

Insert into the MemTable

If you have a MemTable already, you can use the MemTable::insert_into API to add new batches

This is the API that gets used to implement SQL INSERT INTO <table> SELECT .... type queries

Directory of Files

If your data is in files in a directory, you can write new files into that same directory and they will be picked up on the next read (aka "hive" style) using ListingTable (e.g. read_csv with a path to a directory)

Make your own TableProvider

Depending on where your data is coming from, rather than creating a new MemTable for it, you could potentially implement your own TableProvider that would return the data on demand -- depending on what you are doing you could also implement predicate / projection pushdown too.