Skip to content

Commit 94c8e3f

Browse files
authored
update docs about streaming load (#2874)
* fix links * update version of databend-dirver * http handler: rm doc about stats_uri * update docs about streaming load
1 parent f052ef0 commit 94c8e3f

File tree

5 files changed

+51
-9
lines changed

5 files changed

+51
-9
lines changed

docs/en/developer/00-drivers/01-python.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,4 +124,4 @@ with engine.connect() as conn:
124124
## Resources
125125

126126
- **PyPI**: [databend-driver](https://pypi.org/project/databend-driver/)[databend-sqlalchemy](https://pypi.org/project/databend-sqlalchemy/)
127-
- **GitHub**: [databend-driver](https://github.com/databendlabs/databend-py)[databend-sqlalchemy](https://github.com/databendlabs/databend-sqlalchemy)
127+
- **GitHub**: [databend-driver](https://github.com/databendlabs/bendsql/tree/main/bindings/python)[databend-sqlalchemy](https://github.com/databendlabs/databend-sqlalchemy)

docs/en/developer/00-drivers/02-nodejs.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,5 +81,5 @@ conn.close();
8181
## Resources
8282

8383
- **NPM Package**: [databend-driver](https://www.npmjs.com/package/databend-driver)
84-
- **GitHub Repository**: [databend-js](https://github.com/databendlabs/databend-js)
84+
- **GitHub Repository**: [databend-driver](https://github.com/databendlabs/bendsql/tree/main/bindings/nodejs)
8585
- **TypeScript Definitions**: Included in package

docs/en/developer/00-drivers/04-rust.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Add the driver to your `Cargo.toml`:
1212

1313
```toml
1414
[dependencies]
15-
databend-driver = "0.7"
15+
databend-driver = "0.30"
1616
tokio = { version = "1", features = ["full"] }
1717
```
1818

docs/en/developer/10-apis/http.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,7 @@ This handler return results in `pages` with long-polling.
1919
1. A `GET` to the `next_uri` returns the next `page` of query results. It returns `QueryResponse` too, processing it
2020
the same way until `next_uri` is null.
2121
2. (optional) A `GET` to the `kill_uri` to kill the query. Return empty body.
22-
3. (optional) A `GET` to the `stats_uri` to get stats only at once (without long-polling), return `QueryResponse`
23-
with empty `data` field.
22+
2423

2524
Please note that you should keep using the latest `next_uri` to get the next page of results before the query is finished, otherwise you may miss some results or leak session resources until session timeout. The `next_uri` will be null when you have received all the results of the query.
2625

docs/en/guides/40-load-data/01-load/02-local.md

Lines changed: 47 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,16 @@ Uploading your local data files to a stage or bucket before loading them into Da
77

88
Please note that the files must be in a format supported by Databend, otherwise the data cannot be imported. For more information on the file formats supported by Databend, see [Input & Output File Formats](/sql/sql-reference/file-format-options).
99

10+
You can also load local files into tables programmatically using JDBC or Python drivers.
11+
12+
## Load Methods
13+
14+
There are two methods to load data from local files:
15+
16+
1. **Stage**: Upload the local file to an internal stage, then copy data from the staged file into the table. File upload occurs either through databend-query or using a presigned URL, depending on the `presigned_url_disabled` connection option (default: `false`).
17+
2. **Streaming**: Load the file directly into the table during upload. Use this method when the file is too large to store as a single object in your object storage.
18+
19+
1020
## Tutorial 1 - Load from a Local File
1121

1222
This tutorial uses a CSV file as an example to demonstrate how to import data into Databend using [BendSQL](../../30-sql-clients/00-bendsql/index.md) from a local source.
@@ -47,14 +57,48 @@ CREATE TABLE books (
4757
Send loading data request with the following command:
4858

4959
```shell
50-
❯ bendsql --query='INSERT INTO book_db.books VALUES;' --format=csv [email protected]
60+
❯ bendsql --query='INSERT INTO book_db.books from @_databend_load file_format=(type=csv)' [email protected]
61+
```
62+
63+
- The `@_databend_load` is a placeholder representing local file data.
64+
- The [file_format clause](/sql/sql-reference/file-format-options/) uses the same syntax as the COPY command.
65+
66+
Alternatively, use a Python script:
67+
68+
```python
69+
import databend_driver
70+
dsn = "databend://root:@localhost:8000/?sslmode=disable",
71+
client = databend_driver.BlockingDatabendClient(dsn)
72+
conn = client.get_conn()
73+
query = "INSERT INTO book_db.books from @_databend_load file_format=(type=csv)"
74+
progress = conn.load_file(query, "book.csv")
75+
conn.close()
76+
```
77+
78+
Or use Java code:
79+
80+
```java
81+
import java.sql.Connection;
82+
import java.sql.Statement;
83+
import java.io.FileInputStream;
84+
import java.nio.file.Files;
85+
import com.databend.jdbc.DatabendConnection;
86+
String url = "jdbc:databend://localhost:8000";
87+
try (FileInputStream fileInputStream = new FileInputStream(new File("book.csv")));
88+
Connection connection = DriverManager.getConnection(url, "databend", "databend");
89+
Statement statement = connection.createStatement()) {
90+
DatabendConnection databendConnection = connection.unwrap(DatabendConnection.class);
91+
String sql = "insert into book_db.books from @_databend_load file_format=(type=csv)";
92+
int nUpdate = databendConnection.loadStreamToTable(sql, fileInputStream, f.length(), DatabendConnection.LoadMethod.Stage);
93+
}
5194
```
5295

5396
:::note
5497
Be sure that you are able to connect to the backend object storage for Databend from local BendSQL directly.
55-
If not, then you need to specify the `--set presigned_url_disabled=1` option to disable the presigned url feature.
98+
If not, you need to specify the `--set presigned_url_disabled=1` option to disable the presigned url feature.
5699
:::
57100

101+
58102
### Step 3. Verify Loaded Data
59103

60104
```shell
@@ -96,15 +140,14 @@ CREATE TABLE bookcomments (
96140
comments VARCHAR,
97141
date VARCHAR
98142
)
99-
100143
```
101144

102145
### Step 2. Load Data into Table
103146

104147
Send loading data request with the following command:
105148

106149
```shell
107-
❯ bendsql --query='INSERT INTO book_db.bookcomments(title,author,date) VALUES;' --format=csv [email protected]
150+
❯ bendsql --query='INSERT INTO book_db.bookcomments(title,author,date) file_format=(type=csv)' [email protected]
108151
```
109152

110153
Notice that the `query` part above specifies the columns (title, author, and date) to match the loaded data.

0 commit comments

Comments
 (0)