Please add binary for arm64 #903

creationix · 2017-11-04T04:50:16Z

I'm trying to use this module on some arm64 devices and our build times are really long because there is no prebuilt binary for us.

node-pre-gyp ERR! Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v3.1.13/node-v57-linux-arm64.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v57 ABI) (falling back to source compile with node-gyp)

I'm using the new LTS version of node.

You can reproduce this on a raspberry pi 3 using https://github.com/bamarni/pi64 which gives a 64-bit kernel. It does eventually compile from source, but every time uses about 15 minutes of time.

The text was updated successfully, but these errors were encountered:

creationix · 2017-11-06T22:34:03Z

Hmm, actually, I'm having trouble build from source in some cases as well.

/srv/walnut/core/node_modules/sqlite3 # uname -m
aarch64
/srv/walnut/core/node_modules/sqlite3 # npm rebuild sqlite3
/srv/walnut/core/node_modules/sqlite3 # node -r sqlite3
module.js:538
    throw err;
    ^

Error: Cannot find module '/srv/walnut/core/node_modules/sqlite3/lib/binding/node-v57-linux-arm64/node_sqlite3.node'
    at Function.Module._resolveFilename (module.js:536:15)
    at Function.Module._load (module.js:466:25)
    at Module.require (module.js:579:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/srv/walnut/core/node_modules/sqlite3/lib/sqlite3.js:4:15)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)

I don't think it even tried to build from source when I initially ran npm install and npm rebuild doesn't seem to do anything.

Any ideas how I can debug node-prep-gyp issues here?

creationix · 2017-11-06T22:34:56Z

There is no ./lib/binding folder at all FWIW, which leads me to believe more it didn't try building the binary.

springmeyer · 2017-11-07T19:19:36Z

Please add binary for arm64

Yes, this would really help users. To make providing an arm64 binary sustainable for each release, we'd need an automated way of doing this. Do you have suggestions on how to script building for arm64 on travis (or similar free CI infra)?

There is no ./lib/binding folder at all FWIW, which leads me to believe more it didn't try building the binary.

Odd, @creationix can you try passing --build-from-source to your npm rebuild command? Without that node-pre-gyp will be attempting to install from a remote binary each time, by default. Clearly something is wrong with the fallback though that I can't tell from your logs.

creationix · 2017-11-08T04:17:12Z

Assuming your CI server has support for qemu-aarch64-static and docker, something like the following should work:

FROM arm64v8/node:carbon
COPY qemu-aarch64-static /usr/bin/qemu-aarch64-static
RUN apt-get update
RUN apt-get install -y git build-essential
RUN git clone --depth=1 https://github.com/mapbox/node-sqlite3.git
WORKDIR /node-sqlite3
RUN make

See this article for more details on using qemu for cross-compiling with docker.

johnelliott · 2017-11-24T17:49:55Z

Another raspberry pi 3 node-sqlite user here. Pre-built would be wonderful 👍

SridharRamasami · 2018-03-01T06:56:56Z

One more raspberry pi 3 user here.

lucianodlf · 2018-03-17T23:24:43Z

another user is added to the order

rzr · 2018-08-08T08:52:21Z

I guess this could be built on qemu , check this related change:
#1028

k1r0s · 2018-09-16T14:08:04Z

yet another non amd64 user

rzr · 2018-09-17T14:37:30Z

Relate to: #1028

Slos · 2018-11-08T16:37:04Z

and another ARM user here. I've been trying all sorts of tricks to get this to build on my raspi without success. Please add pre built binary to the repo... thanks a lot!

dvilelaf · 2018-11-09T10:47:50Z

And another one user. Couldn't make it work on a raspberry pi zero w.

rzr · 2018-11-09T10:49:43Z

have you tried to build in docker ? may I also request this package to land in debian too

https://tracker.debian.org/pkg/node-sqlite3

songzhuang · 2018-11-09T12:45:42Z

Hi,
I can volunteer to create the build for the Pi. If someone knows the steps, please point me to it. I guess it would be:

where to get the original files
how to compile
where and how to push the compiled files.

Regards,

rzr · 2018-11-09T13:27:27Z

Well I used this PR to build offline:

#1028

@songzhuang you can try to see how this can be adapted for arm64 for @k1r0s

Good luck

LudekStoklasa · 2019-01-30T21:33:44Z

Hi, any idea where to download binary for arm64 ?
I have been trying to compile it myself on Synology NAS DS118 (ARMv8 64), but was unsuccessful :-/
Any tip where to download it or how to compile it?

rzr · 2019-04-17T10:44:47Z

I managed to build current version using docker, it tooks me less than 8min

I shared file at:
https://bintray.com/rzr/devel/sqlite3#files
0568e185fdeb1433d8d92694f2d35a25e48694c53a210eaf09ed522e1c0633c9 sqlite3/v4.0.6/node-v57-linux-arm64.tar.gz

Patch to come next

Relate-to: #1138

Bug: TryGhost#903 Change-Id: Ic91292879deed92fcffd90d7b2d79f551c5b06f4 Signed-off-by: Philippe Coval <[email protected]>

It took less than 10 min to build on Ubuntu 18.04 x86_64 If merged other arch will come next, and then automated by CI. Bug: TryGhost#903 Forwarded: TryGhost#1147 Change-Id: Ic91292879deed92fcffd90d7b2d79f551c5b06f4 Signed-off-by: Philippe Coval <[email protected]>

It took less than 10 min to build on Ubuntu 18.04 x86_64 If merged other arch will come next, and then automated by CI. Bug: #903 Forwarded: #1147 Change-Id: Ic91292879deed92fcffd90d7b2d79f551c5b06f4 Signed-off-by: Philippe Coval <[email protected]>

n1ru4l · 2020-07-21T08:48:53Z

With docker buildx it is quite easy to build binaries for multiple platforms.

We are currently using it for building docker images for all platforms. We could incorporate it into the CI for easily generating pre-built binaries for SQLite.

I think it would be as easy as cloning this repository, running npm install --ignore-scripts and npx node-pre-gyp install --build-from-source and then upload the build artifacts.

A matrix could then cover all the platforms e.g. linux/arm/v6, linux/arm/v7, linux/arm64

louisabraham · 2020-08-22T10:18:08Z

Any update?

n1ru4l · 2020-08-22T10:31:07Z

@louisabraham I created this PR that adds binaries, feel free to review and test the binaries. #1362

louisabraham · 2020-11-01T21:56:12Z

Thank you so much, they work!

gpetrov · 2020-12-13T12:09:07Z

Any progress in making the binaries officially available?
I'm getting:

#13 11.19 > [email protected] install /opt/node_app/node_modules/sqlite3
#13 11.19 > node-pre-gyp install --fallback-to-build
#13 11.19 
#13 11.35 node-pre-gyp WARN Using request for node-pre-gyp https download 
#13 11.81 node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.0/napi-v3-linux-arm64.tar.gz 
#13 11.81 node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v83 ABI, glibc) (falling back to source compile with node-gyp)

Note linux/arm64 binaries are very important for the upcoming Docker Desktop for Apple Silicon M1 as those will run natively there.

dark-matter-matters · 2021-03-19T17:53:56Z

@n1ru4l thanks for the PR. I could successfully generate the binaries using docker buildx, but I can't get npm and node-pre-gyp to use them.
When passing --source=napi-v3-linux-arm64.tar.gz, node-pre-gyp keeps trying to download it on s3 (and fails with error 403).

I'm almost there... How can I specify to node-pre-gyp that he should take my local file instead of fetching it over the distant repo?

n1ru4l · 2021-03-19T17:56:59Z

You have to place it in a very specific folder so that the cache is hit. I don't remember the exact path though.

dark-matter-matters · 2021-03-21T11:28:42Z

Thanks for the heads up. The location is shown in the error log file of npm just before the 403 response:

[...]
npm ERR! node-pre-gyp info check checked for "root/project/server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node" (not found)

So all you have to do is to extract the node_sqlite3.node file from the tar.gz produced by docker buildx and place it in the node_module/sqlite3/lib/binding/napi-v3-linux-[ARCH]/node_sqlite3.node folder.
In my case I also had to delete first the project's node_modules folder to make it work (my previous commands had probably generated some invalid cache).

I hope they pull your PR soon to avoid this!

tom-boothman · 2021-05-28T18:04:59Z

I wanted to workaround this while doing an npm ci which sadly meant I couldn't put the .node file in its final location.

This will let you host the binaries somewhere:

npm ci --node_sqlite3_binary_host_mirror=https://somehost

It'll try and download from http://somehost/sqlite3/v5.0.1/napi-v6-linux-arm64.tar.gz (the filename is the same name as the folder in lib/binding napi-v[NAPI_VERSION]-linux-[ARCH])
The tarball contains the folder from ../node_modules/sqlite3/lib/binding and in it the node_sqlite3.node file

dark-matter-matters · 2021-06-21T13:23:11Z

Thanks for the workaround @tom-boothman, for some reason placing the file locally would sometimes not work for me, probably because I must use an older version of node-pre-gyp.
Having to upload the file on Internet while it's available on the very same device is quite an overkill but it does the job...

daniellockyer · 2022-04-14T08:58:01Z

This has been shipped in master and will be released soon 🙂

astorije mentioned this issue Mar 2, 2018

Add sqlite logging and reloading messages thelounge/thelounge#1839

Merged

k1r0s mentioned this issue Sep 16, 2018

(non amd64 support) Running abstruse on aarch64 [odroid c2] bleenco/abstruse#413

Closed

rzr added a commit to CrossStream/node-sqlite3 that referenced this issue Apr 17, 2019

tools: Crossbuild for ARMv8-64

aecc5b6

Bug: TryGhost#903 Change-Id: Ic91292879deed92fcffd90d7b2d79f551c5b06f4 Signed-off-by: Philippe Coval <[email protected]>

rzr mentioned this issue Apr 17, 2019

tools: Crossbuild for ARMv8-64 #1147

Merged

oceanlvr mentioned this issue Sep 16, 2021

[Bug] Compatibility issues with the sqlite package on apple M1 emqx/MQTTX#690

Closed

daniellockyer closed this as completed Apr 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Please add binary for arm64 #903

Please add binary for arm64 #903

creationix commented Nov 4, 2017 •

edited

Loading

creationix commented Nov 6, 2017

creationix commented Nov 6, 2017

springmeyer commented Nov 7, 2017

creationix commented Nov 8, 2017 •

edited

Loading

johnelliott commented Nov 24, 2017

SridharRamasami commented Mar 1, 2018 •

edited

Loading

lucianodlf commented Mar 17, 2018

rzr commented Aug 8, 2018

k1r0s commented Sep 16, 2018

rzr commented Sep 17, 2018

Slos commented Nov 8, 2018

dvilelaf commented Nov 9, 2018

rzr commented Nov 9, 2018 •

edited

Loading

songzhuang commented Nov 9, 2018

rzr commented Nov 9, 2018

LudekStoklasa commented Jan 30, 2019

rzr commented Apr 17, 2019

n1ru4l commented Jul 21, 2020

louisabraham commented Aug 22, 2020

n1ru4l commented Aug 22, 2020

louisabraham commented Nov 1, 2020

gpetrov commented Dec 13, 2020

dark-matter-matters commented Mar 19, 2021

n1ru4l commented Mar 19, 2021

dark-matter-matters commented Mar 21, 2021 •

edited

Loading

tom-boothman commented May 28, 2021 •

edited

Loading

dark-matter-matters commented Jun 21, 2021 •

edited

Loading

daniellockyer commented Apr 14, 2022

Please add binary for arm64 #903

Please add binary for arm64 #903

Comments

creationix commented Nov 4, 2017 • edited Loading

creationix commented Nov 6, 2017

creationix commented Nov 6, 2017

springmeyer commented Nov 7, 2017

creationix commented Nov 8, 2017 • edited Loading

johnelliott commented Nov 24, 2017

SridharRamasami commented Mar 1, 2018 • edited Loading

lucianodlf commented Mar 17, 2018

rzr commented Aug 8, 2018

k1r0s commented Sep 16, 2018

rzr commented Sep 17, 2018

Slos commented Nov 8, 2018

dvilelaf commented Nov 9, 2018

rzr commented Nov 9, 2018 • edited Loading

songzhuang commented Nov 9, 2018

rzr commented Nov 9, 2018

LudekStoklasa commented Jan 30, 2019

rzr commented Apr 17, 2019

n1ru4l commented Jul 21, 2020

louisabraham commented Aug 22, 2020

n1ru4l commented Aug 22, 2020

louisabraham commented Nov 1, 2020

gpetrov commented Dec 13, 2020

dark-matter-matters commented Mar 19, 2021

n1ru4l commented Mar 19, 2021

dark-matter-matters commented Mar 21, 2021 • edited Loading

tom-boothman commented May 28, 2021 • edited Loading

dark-matter-matters commented Jun 21, 2021 • edited Loading

daniellockyer commented Apr 14, 2022

creationix commented Nov 4, 2017 •

edited

Loading

creationix commented Nov 8, 2017 •

edited

Loading

SridharRamasami commented Mar 1, 2018 •

edited

Loading

rzr commented Nov 9, 2018 •

edited

Loading

dark-matter-matters commented Mar 21, 2021 •

edited

Loading

tom-boothman commented May 28, 2021 •

edited

Loading

dark-matter-matters commented Jun 21, 2021 •

edited

Loading