Skip to content

Conversation

@DinosL
Copy link
Contributor

@DinosL DinosL commented Jan 14, 2026

Curl was not capturing the response from GET requests, resulting in missing pagination metadata (e.g., cursor) required for paginated queries.

built on top of #205

Comment on lines +226 to +232
if (info.response_handler) {
auto response = TransformResponseCurl(res);
if (!info.response_handler(*response)) {
return response;
}
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you possibly check the similar code in httpfs_httplib_client.cpp:

        unique_ptr<HTTPResponse> Get(GetRequestInfo &info) override {
                if (state) {
                        state->get_count++;
                }
                auto headers = TransformHeaders(info.headers, info.params);
                if (!info.response_handler && !info.content_handler) {
                        return TransformResult(client->Get(info.path, headers));
                } else {
                        return TransformResult(client->Get(
                            info.path.c_str(), headers,
                            [&](const duckdb_httplib_openssl::Response &response) {
                                    auto http_response = TransformResponse(response);
                                    return info.response_handler(*http_response);
                            },
                            [&](const char *data, size_t data_length) {
                                    if (state) {
                                            state->total_bytes_received += data_length;
                                    }
                                    return info.content_handler(const_data_ptr_cast(data), data_length);
                            }));
                }
        }

In particular, I am not sure if they can be made more similar to each other the GET handling.

@carlopi
Copy link
Collaborator

carlopi commented Jan 15, 2026

Thanks!

@carlopi carlopi merged commit add35a0 into duckdb:main Jan 15, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants