SNOW-692968: Async queries support #787

sfc-gh-ext-simba-nl · 2024-11-28T04:12:25Z

Added snowflake_async_execute to execute async queries and snowflake_async_stmt to get previously run async queries using query id.

sfc-gh-ext-simba-hx · 2024-12-02T20:08:10Z

include/snowflake/client.h

+ *
+ * @return sfstmt SNOWFLAKE_STMT context for async queries.
+ */
+SF_STMT* STDCALL snowflake_async_stmt(SF_CONNECT *sf, const char *query_id);


The function name sounds confusing since it seems to be an equivalent of snowflake_stmt. Maybe use the similar function name as JDBC, create_resultset or to be more specific, create_async_query_result? Still return a SF_STMT instance since libsfclient doesn't have an struct specifically for result set, but would be easier to understand the functionality of the function.

lib/client.c

…ng results to when user wants to actually get results

sfc-gh-ext-simba-hx · 2024-12-03T18:05:42Z

lib/client.c

      char* queryStatus = snowflake_cJSON_GetStringValue(status);
      ret = get_status_from_string(queryStatus);
    }
+    else {


After seeing the logic in get_real_results I'm not quite sure whether we should set error here. Seems no data is acceptable, the error log in get_query_metadata would be the same.

sfc-gh-ext-simba-hx · 2024-12-03T18:07:51Z

lib/client.c

 #ifdef _WIN32
-  Sleep(sleep_time);


define this as a common function in platform.c? I think same logic being used in http_perform.c as well.

sfc-gh-ext-simba-hx · 2024-12-03T18:22:28Z

tests/test_async.c

Please add test cases to cover typical use case, checking query status before calling any fetch/metadata API. Take JDBC doc as example, https://docs.snowflake.com/en/developer-guide/jdbc/jdbc-using#examples-of-asynchronous-queries
Add usage example in PR description as well (could be reference to test case if the code is clean enough).
Calling fetch/metadata function without checking query status also needed. Please test with queries would take some time.

sfc-gh-jszczerbinski

Please remove unnecessary formatting changes. It's hard to review with so much (unnecessary?) changes.

sfc-gh-jszczerbinski · 2024-12-12T15:48:44Z

lib/connection.c

            // Remove old result URL and query code if this isn't our first rodeo
            SF_FREE(result_url);
            memset(query_code, 0, QUERYCODE_LEN);
            data = snowflake_cJSON_GetObjectItem(*json, "data");
            if (json_copy_string(&result_url, data, "getResultUrl") !=
-                SF_JSON_ERROR_NONE) {


No changes? No reason to reformat.

I think VS automatically did that accidentally when I wrapped it in an if clause. I'll revert these

sfc-gh-jszczerbinski · 2024-12-12T15:49:52Z

lib/connection.c

-                // Error came from request up, just break
-                stop = SF_BOOLEAN_TRUE;
-                break;
+              GET_REQUEST_TYPE, error, SF_BOOLEAN_FALSE,


sfc-gh-jszczerbinski · 2024-12-12T15:50:01Z

lib/connection.c

-                SET_SNOWFLAKE_ERROR(error, SF_STATUS_ERROR_BAD_JSON, error_msg,
-                                    SF_SQLSTATE_UNABLE_TO_CONNECT);
-                break;
+              stop = SF_BOOLEAN_TRUE;


sfc-gh-jszczerbinski · 2024-12-12T15:54:50Z

lib/client.c

+  // Get query stats
+  char* metadata_str = get_query_metadata(sfstmt);
+  if (metadata_str) {
+    cJSON* metadata = snowflake_cJSON_Parse(metadata_str);


Error handling?

sfc-gh-jszczerbinski · 2024-12-12T15:55:10Z

lib/client.c

+  char* metadata_str = get_query_metadata(sfstmt);
+  if (metadata_str) {
+    cJSON* metadata = snowflake_cJSON_Parse(metadata_str);
+    cJSON* stats = snowflake_cJSON_GetObjectItem(metadata, "stats");


We assume it's an object?

sfc-gh-jszczerbinski · 2024-12-12T15:56:29Z

lib/client.c

@@ -2073,6 +2284,13 @@ SF_STATUS STDCALL _snowflake_execute_ex(SF_STMT *sfstmt,
    body = create_query_json_body(sfstmt->sql_text, sfstmt->sequence_counter,
                                  is_string_empty(sfstmt->connection->directURL) ?
                                  NULL : sfstmt->request_id, is_describe_only);
+
+    if (is_async_exec) {
+      snowflake_cJSON_AddBoolToObject(body, "asyncExec", SF_BOOLEAN_TRUE);


snowflake_cJSON_AddBoolToObject(body, "asyncExec", is_async_exec); ?

sfc-gh-jszczerbinski · 2024-12-12T16:03:47Z

lib/connection.c

-        while (strcmp(query_code, QUERY_IN_PROGRESS_CODE) == 0 ||
-               strcmp(query_code, QUERY_IN_PROGRESS_ASYNC_CODE) == 0) {
+        sf_bool isAsyncExec = SF_BOOLEAN_FALSE;
+        cJSON *json_body = snowflake_cJSON_Parse(body);


Error handling?

…m/snowflakedb/libsnowflakeclient into SNOW-692968-async-queries-support

tests/test_async.c

sfc-gh-dprzybysz · 2024-12-20T10:33:22Z

include/snowflake/client.h

+ * The query status
+ */
+typedef enum SF_QUERY_STATUS {
+    SF_QUERY_STATUS_RUNNING,


let's sort statuses alphabetically

sfc-gh-dprzybysz · 2024-12-20T10:34:25Z

include/snowflake/platform.h

@@ -142,6 +142,9 @@ void STDCALL sf_memory_error_handler();
 // this should be called by application before any calls of sfclient
 void STDCALL sf_exception_on_memory_failure();

+void STDCALL sf_sleep_ms(int sleep_ms);


should it be in out public api?

sfc-gh-dprzybysz · 2024-12-20T10:34:56Z

lib/client.c

@@ -54,12 +56,193 @@ static SF_STATUS STDCALL
 _reset_connection_parameters(SF_CONNECT *sf, cJSON *parameters,
                             cJSON *session_info, sf_bool do_validate);

+static const char* query_status_names[] = {
+    "RUNNING",


can we keep the statuses sorted?

sfc-gh-dprzybysz · 2024-12-20T10:35:34Z

lib/client.c

+ */
+SF_QUERY_STATUS get_status_from_string(const char *query_status) {
+  if (query_status == NULL) {
+    return SF_QUERY_STATUS_NO_DATA;


is it no data or more UNKNOWN?

sfc-gh-dprzybysz · 2024-12-20T10:36:46Z

lib/client.c

+  cJSON *data = NULL;
+  cJSON *queries = NULL;
+  char *s_resp = NULL;
+  size_t url_size = strlen(QUERY_MONITOR_URL) -2 + strlen(sfstmt->sfqid) + 1;


let's apply code formatter on changed files

sfc-gh-dprzybysz · 2024-12-20T10:46:49Z

tests/test_async.c

+      cmocka_unit_test(test_premature_fetch),
+      cmocka_unit_test(test_new_connection),
+      cmocka_unit_test(test_fake_table),
+      cmocka_unit_test(test_invalid_query_id),


let's add a test that fetches chunks

sfc-gh-ext-simba-nl added 4 commits November 27, 2024 20:08

Add async query support

763153f

Add async test file

974613b

Fix linux build by including unistd.h

a2f2701

Remove unnecessary test

3829b60

sfc-gh-ext-simba-hx reviewed Dec 2, 2024

View reviewed changes