Crash on SQLite insert query #12079

kozharskyad · 2024-06-23T08:36:10Z

How can we reproduce the crash?

prerequisites:

MacBook Pro 2023 14"
CPU: Apple Silicon M3 Pro
RAM: 18 Gb
OS: macOS Sonoma 14.5
Runtime: Docker Desktop 26.1.4
Image: oven/bun:1.1.16-alpine OR oven/bun:1.1.15-alpine OR oven/bun:1.1.14-alpine

OR

VMWare ESXi 6.5 virtual machine
CPU: Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz x2
RAM: 2 Gb
OS: Debian GNU/Linux 11 (bullseye)
Runtime: Docker Engine CE 24.0.5
Image: oven/bun:1.1.16-alpine OR oven/bun:1.1.15-alpine OR oven/bun:1.1.14-alpine

Difference of this machines probably means machine architecture is not that important.
On oven/bun:1.1.13-alpine crash not reproducible.

Clone repository:

git clone [email protected]:kozharskyad/harbor-scanner-events-processor.git

Change CWD to cloned repository:

cd harbor-scanner-events-processor

Run crash test:

make crash

You can also open Makefile and see "crash" task with key commands:

docker run -itdp8123:8123 --rm --name $(basename $PWD)-crash $(basename $PWD)
timeout 5s curl -svkXPOST -d'{"type":"SCANNING_COMPLETED","event_data":{"resources":[{"digest":"sha256:f302741af6fa1e2ebfead5a7828d953816e8e9eca773d812ca8a95637d414401","tag":"latest","resource_url":"some-image-ref:latest","scan_overview":{"application/vnd.security.vulnerability.report; version=1.1":{"report_id":"d96e643f-a886-4687-8803-e3a3a0ee8725","scan_status":"Success","severity":"Critical","duration":2,"summary":{"total":77,"fixable":77,"summary":{"Critical":6,"High":31,"Low":4,"Medium":36}},"start_time":"2024-06-19T08:59:32Z","end_time":"2024-06-19T08:59:34Z","scanner":{"name":"Trivy","vendor":"Aqua Security","version":"v0.42.0"},"complete_percent":100}}}],"repository":{"name":"dev/xxx/xxx-front","namespace":"library","repo_full_name":"library/dev/xxx/xxx-front","repo_type":"public"}}}' 127.0.0.1:8123/process

Relevant log output

============================================================
Bun v1.1.16 (bf7b327f) Linux x64 (baseline)
Args: "/usr/local/bin/bun" "index.js"
Features: jsc http_server 
Builtins: "bun:main" "bun:sqlite" "node:path" 
Elapsed: 1175ms | User: 497ms | Sys: 91ms
RSS: 1.07GB | Peak: 98.60MB | Commit: 1.07GB | Faults: 0

panic(main thread): Segmentation fault at address 0x80
oh no: Bun has crashed. This indicates a bug in Bun, not your code.

To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:

 https://bun.report/1.1.16/Ba1bf7b327AiggB216ymE+xxQ_o7n+8Dyom+8D_6t/98D_4rz28Dypt45EA2AgI

Stack Trace (bun.report)

Bun v1.1.16 (bf7b327) on linux x86_64_baseline [AutoCommand]

Segmentation fault at address 0x00000080

ld-temp.o:0: WTF::jscSignalHandler
??
1 unknown/js code
ld-temp.o:0: llint_op_call_varargs
ld-temp.o:0: llint_op_call_ignore_result
1 unknown/js code
ld-temp.o:0: llint_op_call
1 unknown/js code
ld-temp.o:0: llint_call_javascript
ld-temp.o:0: JSC::Interpreter::executeCall

The text was updated successfully, but these errors were encountered:

Jarred-Sumner · 2024-06-23T08:41:24Z

Are you able to reproduce this outside of alpine linux? Musl LibC is not supported right now in Bun (#918), and using glibc compat can cause strange issues like crashes when loading e.g. Prisma

Edit: I think it's unlikely that alpine is the cause - this stacktraace is entirely coming from JSC.

kozharskyad · 2024-06-23T08:47:24Z

Yes, I be able reproduce crash outside MUSL environment. Before Docker+Alpine+MUSL I run application in single-binary variant on Debian GNU/Linux 11 (bullseye). To make single binary, I run this command:

bun build --compile --target=bun-linux-x64-baseline --outfile=harbor-scanner-events-processor index.js

OR

bun build --compile --target=bun-linux-x64 --outfile=harbor-scanner-events-processor index.js

Run build on macOS Sonoma 14.5 or on Debian 11 - not important. Same result.

Jarred-Sumner · 2024-06-23T09:11:24Z

I haven't been able to reproduce it from the example on either linux x64 or macOS arm64 or on a debug build of macOS arm64, but one idea: if you move the resolve call to happen after the readableStreamToJSON() call, does it still occur? Or otherwise, if you await the readableStreamToJSON call. Another thought: if you switch from readableStreamToJSON to instead use await req.json()?

https://github.com/kozharskyad/harbor-scanner-events-processor/blob/3d1343fe451858d40fa44d17a21c07a4882c773f/index.js#L68-L70

kozharskyad · 2024-06-23T09:27:20Z

This two variants produces same error:

const processRequest = req => new Promise(resolve =>
  req.json()
    .then(json => (resolve(), processRawEvent(json)))
    .catch(() => resolve())
)

const processRequest = req =>
  req.json()
    .then(json => processRawEvent(json))

kozharskyad · 2024-06-23T09:30:42Z

I miss important thing in the description of the bug, but this thing in Makefile:

--platform=linux/amd64

Only intel/amd64 binary have this bug. On arm64 processors all OK.

kozharskyad · 2024-06-23T11:58:00Z

Make some refactoring, middle Promise now absent, but error not gone.
Also bad statistics:

RSS: 1.07GB | Peak: 119.92MB | Commit: 1.07GB | Faults: 295

1.07GB - it's not normal, if this metric shows us resident memory consumption, seems like memory leak.

UPD: If I change line 73 and replace .then(processRawEvent) to .then(console.log), error is gone.

UPD: If I comment out line 47 .forEach(resource => processResource(resource)), error is gone.

UPD: If I comment out line 22 createResource(resource), error is gone. Interesting. Seems the problem in internal SQLite engine.

UPD: If I comment return undefined from createResource function in db.js and completely comment out call of createResourceQuery.run({...}), error is gone. Problem is in INSERT query.

UPD: Tryed to turn on strict: true mode for database, run the code, got an error about absent url parameter. Then removed all dollar signs in input query variables and got fatal error about segmentation from description of the bug.

UPD: Tryed to check theory about conflicting concurrent SQLite queries. After isResourceExists and deleteResource insert 3 seconds timeout before doing createResource. But got same fatal crash after exactly 3 seconds. Also process.nextTick not helps. So changed name of the issue to "Crash on SQLite insert query".

Jarred-Sumner · 2024-06-24T09:10:47Z

@kozharskyad in about an hour (once CI finishes), can you try running

bunx bun-pr 12079

This script downloads the build artifact of Bun from a PR.

Can you try running in that version of Bun and see if the issue still happens?

I haven't been able to verify this, but here are two guesses I want to test:

Is this happening due to running a SQLite query that takes long enough to run where the GC attempts to free the SQLStatement before the function finishes executing? I don't think that would happen regardless, but I added some code to ensure that doesn't happen
Is there a missing exception check when rebinding values? I added an exception check in each call site to check.

kozharskyad · 2024-06-24T10:30:03Z

Probably no success with pipeline - https://github.com/oven-sh/bun/actions/runs/9642668629
Also I got an error:

Searching GitHub for artifact bun-linux-x64 from PR #12079...
65 |     requestCopy.url = requestCopy.url.replace(/\bclient_secret=\w+/g, "client_secret=[REDACTED]").replace(/\baccess_token=\w+/g, "access_token=[REDACTED]");
66 |     this.request = requestCopy;
67 |     Object.defineProperty(this, "code", {
68 |       get() {
69 |         logOnceCode(
70 |           new import_deprecation.Deprecation(
               ^
Deprecation: [@octokit/request-error] `error.code` is deprecated, use `error.status`.
      at get (/tmp/bunx-0-bun-pr@latest/node_modules/@octokit/request-error/dist-node/index.js:70:11)

120 |         request: requestOptions
121 |       });
122 |     }
123 |     if (status >= 400) {
124 |       const data = await getResponseData(response);
125 |       const error = new import_request_error.RequestError(toErrorMessage(data), status, {
                          ^
HttpError: Not Found - https://docs.github.com/rest/pulls/pulls#get-a-pull-request
 code: "404"

      at /tmp/bunx-0-bun-pr@latest/node_modules/@octokit/request/dist-node/index.js:125:21

Bun v1.1.16 (Linux x64 baseline)

kozharskyad · 2024-06-24T13:18:29Z

Download artifact from #12111 success, but:

root@77e65db42f8f:/opt/app# ./bun --version
Illegal instruction

It's on macOS, force linux/amd64 platform. Probably artifact from pipeline is not baseline version.

UPD: Ok, found --baseline :)

UPD: Got the same error:

Bun Canary v1.1.17-canary.1 (b0db40dc) Linux x64 (baseline)
Args: "./bun" "run" "index.js"
Features: jsc http_server 
Builtins: "bun:main" "bun:sqlite" "node:path" 
Elapsed: 1187ms | User: 484ms | Sys: 114ms
RSS: 1.07GB | Peak: 96.52MB | Commit: 1.07GB | Faults: 18

panic(main thread): Segmentation fault at address 0x80
oh no: Bun has crashed. This indicates a bug in Bun, not your code.

To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:

 https://bun.report/1.1.17/Br2b0db40dAiggB2q+ymE+1mO_o7m+8Dyol+8D_6t+98D_4ry28Dy9w45EA2AgI

Jarred-Sumner · 2024-06-30T04:10:46Z

@kozharskyad can you run bunx bun-pr https://github.com/oven-sh/bun/pull/12246 and try it on that branch?

I'm wondering if it's a JavaScriptCore bug. Specifically, I'm wondering if this PR WebKit/WebKit#30298 fixes it (which is included in that branch above)

kozharskyad · 2024-06-30T07:10:07Z

Sorry, but no success :(

mbp-m3p:harbor-scanner-events-processor a.kozharsky$ make crash
/Library/Developer/CommandLineTools/usr/bin/make FLAGS="-d --name $(basename $PWD)-crash" run; \
        while ! curl 127.0.0.1:8123 >/dev/null 2>&1; do sleep 1; done
docker build --platform=linux/amd64 -t $(basename $PWD) .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  860.2kB
Step 1/13 : FROM oven/bun:1.1.17-alpine AS build
 ---> 67df01aab33f
Step 2/13 : WORKDIR /usr/src
 ---> Using cache
 ---> 923a2361f6d3
Step 3/13 : COPY package.json bun.lockb ./
 ---> Using cache
 ---> 097009c40a52
Step 4/13 : RUN GITHUB_TOKEN='XXX' bunx bun-pr https://github.com/oven-sh/bun/pull/12246 --baseline &&     mv /tmp/bun-node-*/bun-*-pr12246 /usr/local/bin/bun
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 91b071126c60
Resolving dependencies
Resolved, downloaded and extracted [72]
Saved lockfile
Searching GitHub for artifact bun-linux-x64-baseline from PR #12246...
Choosing artifact from run that started Jun 30, 2024, 6:36:14 AM
Downloading bun-linux-x64-baseline from PR #12246 - https://github.com/oven-sh/bun/actions/runs/9730027652
Downloaded to:

/tmp/bun-node-bb66bba1b/bun-01e423fc1a96a62bdd64a1807965a2a5bcf1ba34-pr12246

To run the downloaded executable, use any of the following following commands:

bun-01e423fc1a96a62bdd64a1807965a2a5bcf1ba34-pr12246
bun-12246
bun-latest
 ---> Removed intermediate container 91b071126c60
 ---> 4e90ae476ebd
Step 5/13 : RUN bun install
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 0169e859f4bf
bun install v1.1.18-canary.1 (9f9009b8)

+ [email protected]
+ @types/[email protected]
+ [email protected]

11 packages installed [8.98s]
 ---> Removed intermediate container 0169e859f4bf
 ---> 3d2ac2aaf025
Step 6/13 : COPY . .
 ---> 9b8aa47d10bf
Step 7/13 : RUN bun build --target=bun --sourcemap=inline --outdir=build index.js
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in a1ff78040e50

  index.js                      1772.31 KB

  fonts-3271f15256abe4c8.css    447.79 KB

  style-76221e884f72d11f.css    0.98 KB

[107ms] bundle 22 modules
 ---> Removed intermediate container a1ff78040e50
 ---> 28c50e6d445a
Step 8/13 : FROM oven/bun:1.1.17-distroless
 ---> 938ab078cf4c
Step 9/13 : WORKDIR /opt/app
 ---> Using cache
 ---> c4e6f9f03332
Step 10/13 : COPY --from=build /usr/local/bin/bun /usr/local/bin/bun
 ---> 3f0f629e31e7
Step 11/13 : COPY --from=build /usr/src/build/ ./
 ---> 7a7b9dc0b434
Step 12/13 : ENTRYPOINT ["/usr/local/bin/bun", "run", "index.js"]
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 81d2da00e5f0
 ---> Removed intermediate container 81d2da00e5f0
 ---> ea9960ee66b8
Step 13/13 : CMD []
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
 ---> Running in 3282d6d44083
 ---> Removed intermediate container 3282d6d44083
 ---> e82ca956176e
Successfully built e82ca956176e
Successfully tagged harbor-scanner-events-processor:latest
docker run -itp8123:8123 --rm -d --name harbor-scanner-events-processor-crash $(basename $PWD)
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2dd175bac70f99b3bb1e0b69a5a54ebedfd7de14d6f2ea2c728d109f96862d51
timeout 5s curl -svkXPOST -d'{"type":"SCANNING_COMPLETED","event_data":{"resources":[{"digest":"sha256:f302741af6fa1e2ebfead5a7828d953816e8e9eca773d812ca8a95637d414401","tag":"latest","resource_url":"some-image-ref:latest","scan_overview":{"application/vnd.security.vulnerability.report; version=1.1":{"report_id":"d96e643f-a886-4687-8803-e3a3a0ee8725","scan_status":"Success","severity":"Critical","duration":2,"summary":{"total":77,"fixable":77,"summary":{"Critical":6,"High":31,"Low":4,"Medium":36}},"start_time":"2024-06-19T08:59:32Z","end_time":"2024-06-19T08:59:34Z","scanner":{"name":"Trivy","vendor":"Aqua Security","version":"v0.42.0"},"complete_percent":100}}}],"repository":{"name":"dev/xxx/xxx-front","namespace":"library","repo_full_name":"library/dev/xxx/xxx-front","repo_type":"public"}}}' 127.0.0.1:8123/process || true
*   Trying 127.0.0.1:8123...
* Connected to 127.0.0.1 (127.0.0.1) port 8123
> POST /process HTTP/1.1
> Host: 127.0.0.1:8123
> User-Agent: curl/8.8.0
> Accept: */*
> Content-Length: 774
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 774 bytes
docker logs $(basename $PWD)-crash
============================================================
Bun Canary v1.1.18-canary.1 (9f9009b8) Linux x64 (baseline)
Linux Kernel v6.6.31 | glibc v2.31
Args: "/usr/local/bin/bun" "run" "index.js"
Features: jsc http_server 
Builtins: "bun:main" "bun:sqlite" "node:path" 
Elapsed: 1224ms | User: 509ms | Sys: 93ms
RSS: 1.07GB | Peak: 126.59MB | Commit: 1.07GB | Faults: 0

panic(main thread): Segmentation fault at address 0x80
oh no: Bun has crashed. This indicates a bug in Bun, not your code.

To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:

 https://bun.report/1.1.18/Br29f9009bAiggB27k5mE+1mO_ozhl9Dyggl9D_6l5k9D_+7s98Dy5xx6EA2AgI

docker rm -f $(basename $PWD)-crash
harbor-scanner-events-processor-crash

Unbelievable, but crush address always the same: 0x80

kozharskyad added the crash An issue that could cause a crash label Jun 23, 2024

github-actions bot added linux An issue that only occurs on Linux runtime labels Jun 23, 2024

Jarred-Sumner removed the linux An issue that only occurs on Linux label Jun 23, 2024

Jarred-Sumner assigned cirospaciari Jun 23, 2024

kozharskyad changed the title ~~Crash on web server request~~ Crash on SQLite insert query Jun 23, 2024

Jarred-Sumner added a commit that referenced this issue Jun 24, 2024

Speculative fix for #12079

800ca86

Jarred-Sumner linked a pull request Jun 24, 2024 that will close this issue

Speculative fix for #12079 #12111

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on SQLite insert query #12079

Crash on SQLite insert query #12079

kozharskyad commented Jun 23, 2024

Jarred-Sumner commented Jun 23, 2024 •

edited

Loading

kozharskyad commented Jun 23, 2024 •

edited

Loading

Jarred-Sumner commented Jun 23, 2024 •

edited

Loading

kozharskyad commented Jun 23, 2024 •

edited

Loading

kozharskyad commented Jun 23, 2024

kozharskyad commented Jun 23, 2024 •

edited

Loading

Jarred-Sumner commented Jun 24, 2024 •

edited

Loading

kozharskyad commented Jun 24, 2024

kozharskyad commented Jun 24, 2024 •

edited

Loading

Jarred-Sumner commented Jun 30, 2024

kozharskyad commented Jun 30, 2024 •

edited

Loading

Crash on SQLite insert query #12079

Crash on SQLite insert query #12079

Comments

kozharskyad commented Jun 23, 2024

How can we reproduce the crash?

Relevant log output

Stack Trace (bun.report)

Jarred-Sumner commented Jun 23, 2024 • edited Loading

kozharskyad commented Jun 23, 2024 • edited Loading

Jarred-Sumner commented Jun 23, 2024 • edited Loading

kozharskyad commented Jun 23, 2024 • edited Loading

kozharskyad commented Jun 23, 2024

kozharskyad commented Jun 23, 2024 • edited Loading

Jarred-Sumner commented Jun 24, 2024 • edited Loading

kozharskyad commented Jun 24, 2024

kozharskyad commented Jun 24, 2024 • edited Loading

Jarred-Sumner commented Jun 30, 2024

kozharskyad commented Jun 30, 2024 • edited Loading

Jarred-Sumner commented Jun 23, 2024 •

edited

Loading

kozharskyad commented Jun 23, 2024 •

edited

Loading

Jarred-Sumner commented Jun 23, 2024 •

edited

Loading

kozharskyad commented Jun 23, 2024 •

edited

Loading

kozharskyad commented Jun 23, 2024 •

edited

Loading

Jarred-Sumner commented Jun 24, 2024 •

edited

Loading

kozharskyad commented Jun 24, 2024 •

edited

Loading

kozharskyad commented Jun 30, 2024 •

edited

Loading