Skip to content

Bootstrap TypeScript compilation - #2849

Merged
bert-e merged 5 commits into
development/9.6from
improvement/BB-295-bootstrap-typescript
Sep 15, 2026
Merged

bert-e merged 5 commits into
development/9.6from
improvement/BB-295-bootstrap-typescript

Conversation

@francoisferrand

Copy link
Copy Markdown
Contributor

New modules should be writable in TypeScript, but nothing in the toolchain understands it today. This wires up the compiler, the linter, the image build and the test suites, then converts one module to prove the whole path works.

The guiding constraint was that existing code must not notice. 379 JavaScript files and 155 test files are untouched: no test renamed, no spec migrated, no script repointed.

What each commit does

Type checkingtsconfig.json plus a typecheck script, run in the existing lint job rather than a job of its own. allowJs with checkJs off keeps the current sources in the program so module resolution works, while leaving them unchecked, so the strict settings only ever apply to new TypeScript. typescript-eslint is scoped to **/*.ts.

Compiling into the image — the image now ships the build output. The contents of dist/ are flattened into the working directory, so lib/, bin/ and extensions/ keep the paths they have today and every package script and entrypoint stays as it is. Compilation happens in its own stage so the devDependencies it needs never reach the runtime image.

conf/ is copied separately because the compiler only emits JSON it sees required, and conf/config.json is read with readFileSync. The IAM policies are not copied at all: they ship as their own backbeat-policies artifact, built from the checkout.

Loading TypeScript in tests — registering ts-node in .mocharc.json covers every suite at once, so the thirteen mocha invocations stay as they are and a spec keeps requiring its subject without an extension, whichever language it is written in.

First modulelib/util/buffer.js becomes lib/util/buffer.ts. It uses export =, not export default, so requiring it is unchanged; export default would force every caller to reach through .default.

Worth knowing for the next migration

export = is the rule, not a preference. It emits a plain module.exports = {...}, which is what every existing caller expects.

strict implies alwaysStrict, so the compiler would otherwise add "use strict" to all 201 emitted files, 148 of which run sloppy today. Since mocha runs the sources and never the build output, no test would have caught the difference. It is explicitly off.

When migrating, the deletion of the .js has to land in the same commit as the .ts. With both present the compiler emits the .ts into dist/ while node resolves the .js — silently, so the image would run different code from the tests.

declaration is off: backbeat ships as a service image, not a consumed library, so nothing reads its declarations. Turning it on forces type annotations into JavaScript sources to satisfy a .d.ts nobody loads.

Checks

The build is a no-op for existing code: all 201 emitted files parse to the same syntax tree as their source, differing only in whitespace and the sourcemap comment.

The unit suite gives 1950 passing / 1 pending / 13 failing, identical with and without the loader registered, and identical again after the migration. The 13 need Kafka, Mongo and Redis.

tests/unit/lib/util/buffer.spec.js is unchanged and loads the TypeScript module, as do both plain-JavaScript callers. nyc attributes coverage to buffer.ts by name with line numbers from the source map.

The image was built and compared against one from the previous Dockerfile: same behaviour, and services still read /usr/src/app/conf/.

Test renames, the jest migration and re-enabling skipped tests are deliberately out of scope; they are tracked separately.

Issue: BB-295

@bert-e

bert-e commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/bypass_source_branch_lineage Bypass the cross-branch contamination check
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.09%. Comparing base (6778296) to head (09d85b1).
⚠️ Report is 5 commits behind head on development/9.6.

Files with missing lines Patch % Lines
lib/util/buffer.ts 80.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
bin/ensureServiceUser.js 0.00% <ø> (ø)
lib/MetricsConsumer.js 32.65% <100.00%> (+4.69%) ⬆️
lib/util/buffer.ts 80.00% <80.00%> (ø)

... and 1 file with indirect coverage changes

Components Coverage Δ
Bucket Notification 80.25% <ø> (ø)
Core Library 82.84% <92.30%> (+0.01%) ⬆️
Ingestion 75.65% <ø> (ø)
Lifecycle 81.25% <ø> (ø)
Oplog Populator 85.80% <ø> (ø)
Replication 63.75% <ø> (ø)
Bucket Scanner 85.76% <ø> (ø)
@@                 Coverage Diff                 @@
##           development/9.6    #2849      +/-   ##
===================================================
- Coverage            76.79%   76.09%   -0.70%     
===================================================
  Files                  205      206       +1     
  Lines                14288    14426     +138     
===================================================
+ Hits                 10972    10977       +5     
- Misses                3306     3439     +133     
  Partials                10       10              
Flag Coverage Δ
api:retry 9.20% <7.69%> (-0.10%) ⬇️
api:routes 8.97% <7.69%> (-0.09%) ⬇️
bucket-scanner 85.76% <ø> (ø)
ft_test:queuepopulator 9.73% <23.07%> (-0.07%) ⬇️
ingestion 12.74% <7.69%> (-0.13%) ⬇️
lib 8.98% <30.76%> (-0.10%) ⬇️
lifecycle 19.15% <7.69%> (-0.21%) ⬇️
notification 0.98% <0.00%> (-0.01%) ⬇️
oplogPopulator 0.13% <0.00%> (-0.01%) ⬇️
replication 18.87% <7.69%> (-0.19%) ⬇️
unit 57.61% <92.30%> (-0.48%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread .nycrc
@francoisferrand
francoisferrand force-pushed the improvement/BB-295-bootstrap-typescript branch from 37f6969 to 53d5568 Compare September 14, 2026 10:11
Comment thread Dockerfile
@bert-e

bert-e commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue BB-295 contains:

  • 8.5.6

  • 9.6.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.6.0

Please check the Fix Version/s of BB-295, or the target
branch of this pull request.

Comment thread .nycrc Outdated
@francoisferrand
francoisferrand force-pushed the improvement/BB-295-bootstrap-typescript branch from 2d5d4fe to 076e82e Compare September 14, 2026 22:27
Comment thread .github/workflows/tests.yaml
Comment thread eslint.config.mjs Outdated
@francoisferrand
francoisferrand force-pushed the improvement/BB-295-bootstrap-typescript branch 2 times, most recently from e808f34 to b894c5b Compare September 15, 2026 05:17
Comment thread package.json
Comment thread eslint.config.mjs Outdated
Comment thread tsconfig.json Outdated
Comment thread tsconfig.json
Comment thread Dockerfile Outdated
Comment thread Dockerfile
Comment thread Dockerfile Outdated

@SylvainSenechal SylvainSenechal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have much to say, if it works it works and its cool 🤷

Typescript 7 is available since July, but when I tried using it in Zenko last month there was an issue with eslint still not ready to accept it, so I guess it can wait.

Maybe there are some more config that we will want to add or change but no need to block on this pr.

We may want to do a release now and use it in Zenko to make sure nothing is broken when building the image ?

@scality scality deleted a comment from bert-e Sep 15, 2026
@francoisferrand
francoisferrand force-pushed the improvement/BB-295-bootstrap-typescript branch from b894c5b to 1959d8e Compare September 15, 2026 15:29
MetricsConsumer built its RedisClient as a local and handed it straight to
the StatsModel, so nothing could reach it afterwards: close() only tore down
the kafka consumer, and the socket stayed open for the life of the process.
Keep the client as a member and disconnect it alongside the consumer. Both
populators already close their metrics consumer, so there is nothing to wire
up on their side.

close() also dereferenced the consumer unconditionally, while start() only
assigns it on the ready event: closing during startup threw a TypeError and
left the caller's series waiting on a callback that never came. Guard it, and
still call back.

Issue: BB-882
New modules should be written in TypeScript, but the toolchain has no
notion of it today. Set up the compiler and the linter so `.ts` files are
understood, without changing how any existing JavaScript is built or run:
this commit only adds checking, nothing is emitted yet.

`allowJs` with `checkJs` off keeps the existing sources in the program for
module resolution while leaving them unchecked, so the strict settings
apply to new TypeScript only. `yarn typecheck` runs in the existing lint
job rather than a job of its own.

Issue: BB-295
TypeScript modules have to be compiled before they can run, so the image
now ships the build output instead of the sources. The contents of dist/
are flattened into the working directory, which keeps lib/, bin/ and
extensions/ exactly where they are today: the package scripts and every
entrypoint stay untouched, and a dist/ prefix would break all of them.

Compilation happens in a stage of its own so the devDependencies it needs
never reach the runtime image.

conf/config.json is read with readFileSync rather than required, so the
compiler never emits it and it has to be copied over on its own.

Checked that this leaves existing code alone: for all 201 compiled files
the output parses to the same syntax tree as its source, the differences
being whitespace and the source map comment. alwaysStrict is off because
strict implies it, and adding the directive to the 148 modules that run
without it today would change their semantics in the image only — the
test suite runs against the sources and would not notice.

Issue: BB-295
A module written in TypeScript is invisible to the test suites, which
require their subjects directly from the sources. Registering the loader
in .mocharc.json applies to every suite at once, so the thirteen mocha
invocations stay as they are, and a spec keeps requiring its subject
without an extension whichever language it is written in.

Coverage is collected from the same sources, so lib TypeScript joins the
files nyc reports on.

Issue: BB-295
First module to use the new toolchain, chosen for being a leaf: two pure
functions, no imports, and callers that stay in JavaScript. readUInt64BE
in particular takes a Buffer and returns a Number, which the signature now
states rather than leaving to a comment.

It uses `export =` so that requiring it is unchanged: `export default`
would force every caller to reach through `.default`.

Its spec is untouched, and neither are the two callers, which is the point
— a TypeScript module has to be indistinguishable from the JavaScript one
it replaces.

Issue: BB-295
@francoisferrand
francoisferrand force-pushed the improvement/BB-295-bootstrap-typescript branch from 1959d8e to 09d85b1 Compare September 15, 2026 17:20
@francoisferrand

Copy link
Copy Markdown
Contributor Author

/approve

@scality scality deleted a comment from bert-e Sep 15, 2026
@scality scality deleted a comment from bert-e Sep 15, 2026
@scality scality deleted a comment from bert-e Sep 15, 2026
@scality scality deleted a comment from bert-e Sep 15, 2026
@scality scality deleted a comment from bert-e Sep 15, 2026
@scality scality deleted a comment from bert-e Sep 15, 2026
@bert-e

bert-e commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.6
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3
  • development/9.4
  • development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

  • hotfix/7.10.8
  • hotfix/9.0.7
  • hotfix/7.4.4
  • hotfix/7.4.5
  • hotfix/7.4.6
  • hotfix/7.8.0
  • hotfix/7.9.0
  • hotfix/7.4.2
  • hotfix/7.10.4
  • hotfix/7.4.7
  • hotfix/7.6.0
  • hotfix/7.4.8
  • hotfix/7.4.10
  • hotfix/7.4.9
  • hotfix/7.4.3
  • hotfix/7.2.0
  • hotfix/7.10.0
  • hotfix/7.10.12
  • hotfix/7.70.12
  • hotfix/7.7.0
  • hotfix/7.10.1
  • hotfix/9.0.4
  • hotfix/8.2.12
  • hotfix/7.70.15
  • hotfix/7.4.0
  • hotfix/7.4.1
  • hotfix/7.10.2
  • hotfix/7.10.3
  • hotfix/7.70.1
  • hotfix/7.10.17

Please check the status of the associated issue BB-295.

Goodbye francoisferrand.

The following options are set: approve

@bert-e
bert-e merged commit 09d85b1 into development/9.6 Sep 15, 2026
24 checks passed
@bert-e
bert-e deleted the improvement/BB-295-bootstrap-typescript branch September 15, 2026 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants