Discussion on deploying (pre-)build static libraries

[wearpants]

While I understand statically linked duckdb .a libraries are not included in upstream packages by default & building them is slow, including binary blobs in the source tree smells bad at best and is a supply chain attack vector (by targeting whoever is building them) at worst. Also seems to be the source of a number of other problems, see: #336, #323, #54, #38, #120, #279, #330

It should be possible to build Go bindings entirely from source and/or publicly released artifacts from other projects.

[ayuhito]

If I understand correctly, go-duckdb uses an amalgamation file generated from the C++ source which introduces a mountain of complexity and made Windows blocked for a good year or two.

Looking at the new Node Neo client, I wonder if it makes sense to rely on the C API directly (which should also play nicer with CGO).

To use DuckDB’s C API, on the other hand, one only needs to depend on released binaries. This significantly simplifies the maintenance required, speeds up builds, and minimizes the cost of updates. It removes the uncertainty and risk of rebuilding DuckDB.

It's definitely a monumental undertaking to switch bindings, but I think it would significantly help in the long term. Using the same API as Neo would also mean we have a direct shared point of reference and the C API would get a lot of priority for bug fixes etc.

[taniabogatsch]

It's definitely a monumental undertaking to switch bindings, but I think it would significantly help in the long term.

I agree. Both with the expected effort and the necessity...

We are currently using the C API function calls, but we need to change how we use them in the (not too far) future. I'd love to have go-duckdb run on top of some bindings-only package. I've been experimenting with our recent C API changes here... I also saw this new discussion here: #342. Although I have no experience with purego, and I am not sure how active that project is.

[wearpants]

Is there a smaller than "rewrite the whole thing" solution that's possible here? Such as:

• get upstream to build & publish .a's
• building them ourselves in this project
• or an (unofficial) duckdb-static-libs project
• something else I haven't thought of

[taniabogatsch]

get upstream to build & publish .a's

If duckdb were to provide these, how would we then pull them into the project with go get / build / import ...go-duckdb? To keep the experience as seamless as possible.

[wearpants]

I'm not a go expert, but we seem to have hit the limit of what's possible with plain go get

• Including the .a in this repo's source tree is compromising functionality (b/c of GH's file size limits) as well as compromising security/reproducibility (by having the .a's hand built).
• that seems to necessitate some external step or tooling, whether manually or Makefile downloading the artifact from a published GH release (in this project or upstream).
• A compromise option might be using git-lfs with a published, reproducible artifact included, but that still requires users to set up git-lfs.

But given the various conflicting restrictions in play, I don't see how to do this just with go get.

A good first step would be getting the build recipe and artifacts for the .a's available somewhere, along with necessary instructions/tooling to use that in builds of this library, so that users can customize & reproducibly build the .a. Basically separate the "build .a" and "go get distribution" issues for the moment.

[taniabogatsch]

A good first step would be getting the build recipe and artifacts for the .a's available

I agree. I've opened a first PR for OSX arm + amd builds here: duckdb/duckdb#15682.
I'll try to get these into our deploy pipelines (also for other OS + arch combinations), as duckdb publishing some pre-build static libraries is generally a good idea, independent of this project.

Having both

1. some pre-build static libraries available through duckdb
2. and instructions on how to build your own .as

Sounds like a good way forward to me.

We should also discuss how people would best go get this project and pull in their necessary library if they aren't building it (you mentioned manually or Makefile). I'd love some more input from people using this project about this... :)