diff --git a/docs/en/developer/00-drivers/01-python.md b/docs/en/developer/00-drivers/01-python.md index b9a8c9aa43..b2ada984f0 100644 --- a/docs/en/developer/00-drivers/01-python.md +++ b/docs/en/developer/00-drivers/01-python.md @@ -124,4 +124,4 @@ with engine.connect() as conn: ## Resources - **PyPI**: [databend-driver](https://pypi.org/project/databend-driver/) • [databend-sqlalchemy](https://pypi.org/project/databend-sqlalchemy/) -- **GitHub**: [databend-driver](https://github.com/databendlabs/databend-py) • [databend-sqlalchemy](https://github.com/databendlabs/databend-sqlalchemy) \ No newline at end of file +- **GitHub**: [databend-driver](https://github.com/databendlabs/bendsql/tree/main/bindings/python) • [databend-sqlalchemy](https://github.com/databendlabs/databend-sqlalchemy) \ No newline at end of file diff --git a/docs/en/developer/00-drivers/02-nodejs.md b/docs/en/developer/00-drivers/02-nodejs.md index a0a9c4760b..e3586f54ff 100644 --- a/docs/en/developer/00-drivers/02-nodejs.md +++ b/docs/en/developer/00-drivers/02-nodejs.md @@ -81,5 +81,5 @@ conn.close(); ## Resources - **NPM Package**: [databend-driver](https://www.npmjs.com/package/databend-driver) -- **GitHub Repository**: [databend-js](https://github.com/databendlabs/databend-js) +- **GitHub Repository**: [databend-driver](https://github.com/databendlabs/bendsql/tree/main/bindings/nodejs) - **TypeScript Definitions**: Included in package diff --git a/docs/en/developer/00-drivers/04-rust.md b/docs/en/developer/00-drivers/04-rust.md index 1e325eee65..8db478b6bc 100644 --- a/docs/en/developer/00-drivers/04-rust.md +++ b/docs/en/developer/00-drivers/04-rust.md @@ -12,7 +12,7 @@ Add the driver to your `Cargo.toml`: ```toml [dependencies] -databend-driver = "0.7" +databend-driver = "0.30" tokio = { version = "1", features = ["full"] } ``` diff --git a/docs/en/developer/10-apis/http.md b/docs/en/developer/10-apis/http.md index fec91f9dd0..b3cb85d8c9 100644 --- a/docs/en/developer/10-apis/http.md +++ b/docs/en/developer/10-apis/http.md @@ -19,8 +19,7 @@ This handler return results in `pages` with long-polling. 1. A `GET` to the `next_uri` returns the next `page` of query results. It returns `QueryResponse` too, processing it the same way until `next_uri` is null. 2. (optional) A `GET` to the `kill_uri` to kill the query. Return empty body. - 3. (optional) A `GET` to the `stats_uri` to get stats only at once (without long-polling), return `QueryResponse` - with empty `data` field. + Please note that you should keep using the latest `next_uri` to get the next page of results before the query is finished, otherwise you may miss some results or leak session resources until session timeout. The `next_uri` will be null when you have received all the results of the query. diff --git a/docs/en/guides/40-load-data/01-load/02-local.md b/docs/en/guides/40-load-data/01-load/02-local.md index 98e5f04ecf..c44ac5df39 100644 --- a/docs/en/guides/40-load-data/01-load/02-local.md +++ b/docs/en/guides/40-load-data/01-load/02-local.md @@ -7,6 +7,16 @@ Uploading your local data files to a stage or bucket before loading them into Da Please note that the files must be in a format supported by Databend, otherwise the data cannot be imported. For more information on the file formats supported by Databend, see [Input & Output File Formats](/sql/sql-reference/file-format-options). +You can also load local files into tables programmatically using JDBC or Python drivers. + +## Load Methods + +There are two methods to load data from local files: + +1. **Stage**: Upload the local file to an internal stage, then copy data from the staged file into the table. File upload occurs either through databend-query or using a presigned URL, depending on the `presigned_url_disabled` connection option (default: `false`). +2. **Streaming**: Load the file directly into the table during upload. Use this method when the file is too large to store as a single object in your object storage. + + ## Tutorial 1 - Load from a Local File This tutorial uses a CSV file as an example to demonstrate how to import data into Databend using [BendSQL](../../30-sql-clients/00-bendsql/index.md) from a local source. @@ -47,14 +57,48 @@ CREATE TABLE books ( Send loading data request with the following command: ```shell -❯ bendsql --query='INSERT INTO book_db.books VALUES;' --format=csv --data=@books.csv +❯ bendsql --query='INSERT INTO book_db.books from @_databend_load file_format=(type=csv)' --data=@books.csv +``` + +- The `@_databend_load` is a placeholder representing local file data. +- The [file_format clause](/sql/sql-reference/file-format-options/) uses the same syntax as the COPY command. + +Alternatively, use a Python script: + +```python + import databend_driver + dsn = "databend://root:@localhost:8000/?sslmode=disable", + client = databend_driver.BlockingDatabendClient(dsn) + conn = client.get_conn() + query = "INSERT INTO book_db.books from @_databend_load file_format=(type=csv)" + progress = conn.load_file(query, "book.csv") + conn.close() +``` + +Or use Java code: + +```java +import java.sql.Connection; +import java.sql.Statement; +import java.io.FileInputStream; +import java.nio.file.Files; +import com.databend.jdbc.DatabendConnection; +String url = "jdbc:databend://localhost:8000"; +try (FileInputStream fileInputStream = new FileInputStream(new File("book.csv"))); + Connection connection = DriverManager.getConnection(url, "databend", "databend"); + Statement statement = connection.createStatement()) { + DatabendConnection databendConnection = connection.unwrap(DatabendConnection.class); + String sql = "insert into book_db.books from @_databend_load file_format=(type=csv)"; + int nUpdate = databendConnection.loadStreamToTable(sql, fileInputStream, f.length(), DatabendConnection.LoadMethod.Stage); +} ``` :::note Be sure that you are able to connect to the backend object storage for Databend from local BendSQL directly. -If not, then you need to specify the `--set presigned_url_disabled=1` option to disable the presigned url feature. +If not, you need to specify the `--set presigned_url_disabled=1` option to disable the presigned url feature. ::: + ### Step 3. Verify Loaded Data ```shell @@ -96,7 +140,6 @@ CREATE TABLE bookcomments ( comments VARCHAR, date VARCHAR ) - ``` ### Step 2. Load Data into Table @@ -104,7 +147,7 @@ CREATE TABLE bookcomments ( Send loading data request with the following command: ```shell -❯ bendsql --query='INSERT INTO book_db.bookcomments(title,author,date) VALUES;' --format=csv --data=@books.csv +❯ bendsql --query='INSERT INTO book_db.bookcomments(title,author,date) file_format=(type=csv)' --data=@books.csv ``` Notice that the `query` part above specifies the columns (title, author, and date) to match the loaded data.