Reading remote (az storage) delta tables causes segfault #154
Comments
|
Thanks for the detailed report. Unfortunately, since I don't have access to an Azure environment, this will be difficult for me to reproduce. You mention that you also tested this using the "classic" Node client. What version of that client did you use? It was only very recently upgraded to DuckDB 1.2.0. It would be useful information if you could report whether this happens in your environment with other ways of using DuckDB, such as the CLI or another up-to-date client, such as Python. |
|
Thanks for the swift response @jraymakers Yes, can confirm that the most recent 1.2.0 version of the classic "duckdb" package does not have this issue! |
|
I see, thanks for the additional info. I'm puzzled by what could be causing this, given the very specific platform & environment, and the difficulty in reproducing it on my own will make it hard for me to learn more by myself. I may need to ask for assistance. In the meantime, if you are able to further reduce the repro case, or reproduce in any other environment, that would likely help. For example, it could be helpful to know which part of |
|
I also have a weird(-ish) question that might help track this down, what are the platforms reported by DuckDB for the CLI and the Node package? And if they were to be different, could you possibly check also the Python package? (both checking the platform and whether that works on DELTA or not?) Thanks! |
|
Before going deeper on both of your comments, I installed the segfault-handler package & registered it as a listener. Maybe the output below helps. Another interesting observation is that the result varies from run to run, it's either causing the segfault as shown below or just showing "killed" indicating a OOM situation |
Just awaiting a ".run" call throws the issue already |
|
Given the stack trace above, I'd be curious if just awaiting |
Node Package: linux_amd64_gcc4 So both, are equal! Can check the python lib as well though :) 
|
Yes, can confirm that |
|
@LucaDe, thanks for checking! |
|
Unfortunately even that simple example seems to require an Azure account (which I don't have). |
Sure, checked python as well. Same platform (linux_amd64_gcc4) & reading the delta table from Azure works as expected. |
Yeah, super annoying. Can check if same behavior happens with azurite (never used it before) |
|
Given the stack trace above, the most likely causes are:
I'm not sure what to make of the apparent platform-specificity. There might be some memory corruption that's just more likely to result in an error on some platforms (e.g. Linux AMD64), or it might be somehow specific to the platform, perhaps because of some conditional compilation or compiler-specific behavior. The fact that this repros by just calling So, although we've only seen a repro on Node Neo so far, I tentatively suspect the cause is elsewhere (such as the Azure or Delta extension) and something about Node Neo on Linux x64 makes the problem more likely to surface. It would be interesting to try to make a pure C program that makes the same calls as Node Neo in this case. |
|
Yeah, it's somehow at the intersection of Neo, Azure & Delta. DockerfileFROM node:22-bookworm-slim
WORKDIR /app
COPY index.js .
RUN npm init -y && npm i @duckdb/node-api
ENTRYPOINT ["node", "index.js"]index.jsimport { DuckDBInstance } from "@duckdb/node-api";
const main = async () => {
const instance = await DuckDBInstance.create(":memory:");
const connection = await instance.connect();
await connection.run(`CREATE SECRET secret1 (TYPE AZURE,CONNECTION_STRING 'ABC');`);
console.log("Running prepare");
// This step fails when running on an AMD64 arch & succeeds on ARM64
await connection.prepare(`SELECT * FROM delta_scan('az://testing/data')`);
console.log("Prepare done");
};
await main();ARM64AMD64 |
|
Thanks for the slimmed-down repro above. With those steps, I was able to reproduce locally. I tried reproducing in a pure C program, but I was unable to. However, I was able to reproduce by adding a minimal test hardness function to Node Neo: Napi::Value test_issue154(const Napi::CallbackInfo& info) {
auto env = info.Env();
std::cout << "start" << std::endl;
duckdb_database db;
std::cout << "open" << std::endl;
duckdb_open(":memory:", &db);
duckdb_connection conn;
std::cout << "connect" << std::endl;
duckdb_connect(db, &conn);
duckdb_result res;
std::cout << "query" << std::endl;
duckdb_query(conn, "CREATE SECRET secret1 (TYPE AZURE,CONNECTION_STRING 'ABC');", &res);
duckdb_prepared_statement prepared;
std::cout << "prepare" << std::endl;
duckdb_prepare(conn, "SELECT * FROM delta_scan('az://testing/data')", &prepared);
std::cout << "done" << std::endl;
return env.Undefined();
}When run in a linux/amd64 docker container, the above outputs: So, it seems some combination of the way the native binaries are built in Node (using node-gyp and the node-addon-api) and the use of the Azure & Delta extensions triggers this problem. I thought for a while that perhaps I was mismanaging memory in Node Neo (i.e. double-freeing), but the fact that the above example reproduces the problem eliminates that possibility. Unfortunately, this means I don't know how to solve the problem. It seems most likely that there's a bug in the Azure or Delta extensions that is exposed in this particular build environment. |
|
Thanks for further checking, I cross-posted this within the delta extension issues for now 🤞 |
Hi all, I've been observing a weird crash (uncaught target signal 11 (Segmentation fault) - core dumped) of the entire process while reading data from a delta table from an azure storage container. The specific error does not occur when reading a local delta table OR running on an ARM64 setup (e.g. local macbook).
Since there is no specific error message it is hard to understand the actual issue.
Any tips are highly appreciated 🙌
Other things where I could confirm that things work as expected on the amd64 setup:
Setup for reproducing the issue
Simple package.json including the duckdb package
{ "name": "node-duck", "type": "module", "version": "1.0.0", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "description": "", "dependencies": { "@duckdb/node-api": "^1.2.0-alpha.14" } }Simple JS file demonstrating the behavior
Sample Dockerfile to reproduce
To run the example I copied this delta-table to my local env as well as to an azure storage container.
Building & running the image (AZURE_CONNECTION_STRING env var needed)
The text was updated successfully, but these errors were encountered: