feat: add optional parameters for copy table from command

[Standing-Man]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

• Closes Enhance copy from csv file command #6989.

What's changed and what's your intention?

• Add optional parameters for copy table from command
  • headers(default: false): Enforce validation to ensure that the imported CSV file matches the table’s schema, including column names and types.
  • continue_on_error (default:true): If a mismatch is detected between the imported csv file and the table schema, and continue_on_error is set to true, the file will be skipped and the next CSV file will be processed. Otherwise, the process will stop.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[WenyXu]

Hi @Standing-Man, Thank you for your contribution! Could you add some SQLness tests or integration tests under the tests-integration directory? You can find an example in tests-integration/src/test/instance_test.rs.

[Standing-Man]

Hi @WenyXu, I’m a bit confused — should header and has_header be consistent? Once the header exists, can we directly validate it? Also, how do we determine if a record is corrupted?

https://github.com/GreptimeTeam/greptimedb/blob/main/src/operator/src/statement/copy_table_from.rs#L234

So far, I see how a stream connects to a CSV source, but I’m unsure where the CSV reading and processing logic happens. Could you point me in the right direction?

[WenyXu]

should header and has_header be consistent?

Yes

how do we determine if a record is corrupted?

It depends on the CSV reader’s capabilities. If users attempt to copy data from multiple files and one of them has a schema that differs from the target table, we should consider skipping data from that file. It would be best to review the CSV reader’s source code.

[Standing-Man]

Hi @WenyXu, no matter whether has_header is true or false, the schema compatibility of the target table will be checked.

greptimedb/src/operator/src/statement/copy_table_from.rs

Line 405 in d8563ba

ensure_schema_compatible(&projected_file_schema, &projected_table_schema)?;

Also, the CSV reader reads rows from the source in batches, with default batch size of DEFAULT_BATCH_SIZE.

greptimedb/src/operator/src/statement/copy_table_from.rs

Line 247 in d8563ba

.with_batch_size(DEFAULT_BATCH_SIZE);

I don’t think it’s necessary to add these two optional parameters to enhance the COPY TABLE FROM FILE (FORMAT='CSV') command.

[WenyXu]

Hi @WenyXu, no matter whether has_header is true or false, the schema compatibility of the target table will be checked.

greptimedb/src/operator/src/statement/copy_table_from.rs

Line 405 in d8563ba

ensure_schema_compatible(&projected_file_schema, &projected_table_schema)?;

Yes, this is the expected behavior. The underlying CSV reader supports reading files without a header line — in such cases, the fields are named field_0, field_1, and so on. Therefore, we assume that the column order in the input CSV matches that of the target table, and align the CSV schema with the table schema before performing the compatibility check.

[Standing-Man]

@killme2008 and @WenyXu PTAL when you have time.

[WenyXu]

We can’t simply determine schema incompatibility based on field length. If the source schema is a subset of the target schema, they should be considered compatible.

[Standing-Man]

Should we implement a separate header checker method to verify whether the schema lengths of the file and the table match, and whether their names and types are consistent?

[WenyXu]

Yes, and I suggest adding more tests to cover this feature, as there are many cases we need to handle.

[WenyXu]

I think it would be better to add some test cases in tests-integration to cover the scenarios when header=false, e.g.,:

• The order of CSV fields differs from the table’s.
• The number of CSV fields is fewer or greater than the table’s.

BTW, when testing with header=false, please make sure to use a CSV file without a header row.

[Standing-Man]

Good suggestions. If a CSV file doesn’t contain a header, can I skip checking the header?

[WenyXu]

If a CSV file doesn’t contain a header, can I skip checking the header?

I think we should assume that the data fields in the CSV are in the same order as the table columns.