v18.20.5 is missing all download files

[tillig]

Version

N/A

Platform

No response

Subsystem

No response

What steps will reproduce the bug?

• Look at https://nodejs.org/dist/index.tab - the version 18.20.5 is shown as available.
• Look at https://nodejs.org/dist/v18.20.5/ - only SHA hash files are there.
• Look at https://nodejs.org/dist/v18.20.4/ as a comparison - hashes and downloads are available.

How often does it reproduce? Is there a required condition?

It's consistent.

What is the expected behavior? Why is that the expected behavior?

I expect either the version to be removed from index.tab or the downloads to be available. It's messing up version managers like n and others that assume the index has the list of available releases.

What do you see instead?

Errors from version managers due to missing files (404).

Additional information

No response

[juanheyns]

We use n auto with a .nvmrc file set to lts/hydrogen - this results in our builds failing, would be nice if this is fixed quickly so we don't have to update all our builds.

[neo-rivan-mota]

Thank you for reporting.

[taniarascia]

GitHub actions running on 18 without a minor version declared will fail.

runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18.x]

[c-castillo]

Netlify builds with node 18 are failing as well

[jaymefSO]

breaking a lot of stuff in our infra as well due to heavy use of n version manager

[jay-motwani-trend]

Is there any ETA on a fix for this? This breaks deployments and wondering if we should look into updating all our builds

[anonrig]

cc @nodejs/releasers

[flakey5]

Binaries aren't present in the staging bucket https://dash.cloudflare.com/07be8d2fbc940503ca1be344714cb0d1/r2/default/buckets/dist-staging so nothing got copied over when the release was promoted cc @nodejs/build @nodejs/web-infra

[flakey5]

https://direct.nodejs.org/dist/v18.20.5/ has it

cc @nodejs/build or anyone with write access to Cloudflare can y'all delete the worker route that points /dist/ and /download/release/ to the worker? This should point to the right place https://dash.cloudflare.com/07be8d2fbc940503ca1be344714cb0d1/nodejs.org/workers

[aduh95]

I don't know why the web UI shows fewer files, but it seems to me all files are there now, e.g.

$ curl -I https://nodejs.org/dist/v18.20.5/node-v18.20.5.tar.gz
HTTP/2 200 
date: Wed, 13 Nov 2024 00:49:25 GMT
content-type: application/gzip
content-length: 86241753
cache-control: public, max-age=3600, s-maxage=14400
etag: "4d8368a8627558bb6beb363c36fb4a9f"
last-modified: Tue, 12 Nov 2024 00:43:46 GMT
accept-range: bytes
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
server: cloudflare
cf-ray: 8e1ac307af25be13-DUB

[flakey5]

I don't know why the web UI shows fewer files, but it seems to me all files are there now, e.g.

+1 files are in dist-prod now

Directory listing won't show the files in the colos that cached the response

[flakey5]

Also for future reference please ping the web-infra and build teams as well if this happens

[aduh95]

Binaries aren't present in the staging bucket https://dash.cloudflare.com/07be8d2fbc940503ca1be344714cb0d1/r2/default/buckets/dist-staging so nothing got copied over when the release was promoted

Maybe related: #51394 wasn't backported to v18.x

[flakey5]

Maybe related: #51394 wasn't backported to v18.x

That would make sense, also @targos just did a recent change that updated the upload part to use rclone #55617.

[ljharb]

A long term fix, perhaps, would be to only update the index.tab/index.json files after the files are available?

If someone can point me to where I might work on that, I'd be happy to.

[flakey5]

A long term fix, perhaps, would be to only update the index.tab/index.json files after the files are available?

If someone can point me to where I might work on that, I'd be happy to.

They're generated in the same file that the SHASUMS256.txt is which is needed for promoting the release, I don't see a reason that it needs to be there however https://github.com/nodejs/build/blob/924eacbaf8444248d9e93afacebc006b1dd0eec3/ansible/www-standalone/tools/promote/_resha.sh#L57 There might be something I'm forgetting though on the promotion process

[richardlau]

A long term fix, perhaps, would be to only update the index.tab/index.json files after the files are available?

If someone can point me to where I might work on that, I'd be happy to.

FWIW a while ago I documented how the release process worked (in mermaid, with links to the scripts). Like all documentation, it's now out of date (it was written when we were still self-hosting the website (now Vercel) and before any of the R2 work)🙂. However the broad outline still applies -- we were very careful not to break the existing flow for the R2 work (so even in this case, all files were correctly uploaded to the Digital Ocean droplet -- the issue here was that we missed backporting #51394 to v18.x-staging which meant the builds were staged on the DO droplet but not in R2).

[jay-motwani-trend]

It looks like the issue has now been fixed. I can see the files now in https://nodejs.org/dist/v18.20.5/

[flakey5]

I think we can close this now since the incident is resolved. We still need to make sure #55617 gets backported to the v18 branch however to avoid this from happening again w/ future v18 releases

Btw - thanks to all of you reporting the issue, and apologizes as well!

[trivikr]

still need to make sure #55617 gets backported to the v18 branch

Looks like it should be backported to 20.x and 22.x too?

• node/Makefile

  Line 11 in d5fa767

  CLOUDFLARE_ENDPOINT ?= https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com

• node/Makefile

  Line 11 in 985262a

  CLOUDFLARE_ENDPOINT ?= https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com

[targos]

#55617 has the lts-watch labels and will be backported. I agree we can close this issue.