[build][quality] more efficient CI, builds, improved linter config

[marcdumais-work]

What it does

This is a significant refactoring of the repo and CI.

1) JenKins CI: more efficient, quicker builds

I propose that we adopt a "run often, run quickly" approach to our Jenkins CI.
ATM we mostly run Jenkins when it's time to release new Blueprint installers
(when a PR is merged on master). The other case is if "jenkins" is detected
in the PR branch or PR title, then almost a full Jenkins build is done, just
stopping short of publishing new Blueprint installers (full signing and
notarization is done, which can take a couple of hours).

I made it so we run a very light version of the Jenkins CI by default, that
builds the "dev" version of the app and pre-packages it using electron builder,
generating a directory that contains the app but no installers.

We really ought to run tests then, on the unpacked pre-packaged app, but
it will need to be done separately. Until then, we do not require the built-in
extensions (plugins), so I gave openvsx.org a rest and skip fetching them, by
default.

When it's detected that the Jenkins CI is processing the result of a merge on
master or if the new "release dry-run" mode is enabled [1], a production version
of the Blueprint apps will be produced, including built-in plugins, which will
then be packaged for "real", producing genuine OS-specific installers, signed
and notarized as required. This will take a lot longer to run, of course, but
may be required only a minority of the time.

Conclusion: with this PR, running Jenkins CI now takes about 12 minutes, and
since this is relatively quick, we can afford to run this when any PR branch
is created or updated.

Running the same CI in "release dry-run mode" takes about an hour. A release
will take longer, a bit over an hour, since additional steps are executed,
like copying the various installers to a release folder.

[1] About the "release dry-run" mode, it can be enabled in JenkinsFile,
"pipeline" definition near the top, in "environment":
BLUEPRINT_JENKINS_RELEASE_DRYRUN = 'true

I wish I found a better mechanism to enable this "release dry-run" mode. But
this will serve for now. The "release dry-run" mode should be disabled before
merging a PR to master, but can in the meantime be used to test or troubleshoot
the whole pipeline, in particular signing and notarizing.

2) tsconfig/eslint configs:

Added configs, to improve linter coverage. This made it possible for some
source files, not previously covered, to get ts/linter feedback, both while
editing and when running yarn lint. This will help keep code in-line with
our standards. The config is not perfect and I would welcome further
improvements. But for now I think it's a nice improvement.

3) Build "scripts" (package.json)

Refactored the build commands ("scripts" section in package.json)

• previously, merely running 'yarn" in the repo's root would rebuild
  every application from scratch. This prevents running a quick
  "yarn install", e.g. just to re-install build dependencies
• the new version permits a granular build, with simple defaults
• inspired from a similar change not so long ago in the main repo
• see updated README for some examples

Other misc items:

• renamed extensions / applications

• made the browser application a first class citizen, equal to the
  Electron application

• all applications now share a common 'plugins' folder rather than
  each having their own. Moved the plugin-related entries to root
  package.json

• to gain flexibility about which yarn workspaces are invoked for a
  given lerna command, using the --scope= CLI option.I renamed the
  repo's extensions and applications. This permits easily composing
  commands that target only the extensions or only the applications.
  e.g.:

    "build:extensions": "lerna run --scope=\"blueprint*ext\" build",
    "build:applications": "lerna run --scope=\"blueprint*app\" build --concurrency 1","

• renamed the extensions folders, made them more straightforward

• For systems with limited RAM, like on a Raspberry Pi 4B board with 4GB of RAM,
  it's now possible to successfully build Blueprint. e.g. use the following cmd:
  yarn && yarn build:dev, optionally followed by a packing command like:
  yarn electron package

  • (build:dev will build the Blueprint app in dev mode, which can be achieved
    in 4GB RAM)

• [windows][jenkins] stash only dist folder: Currently, and only for Windows,
  we stash the whole git repo, which is very big and takes long to stash and
  un-stash. For the other OS, we only stash the dist folder, that contains the
  platform-specific installer, that we built. Let's try that for Windows too,
  and see if it works.

• [jenkins] Decrease timeout from 5h to 3h" Looking at the build history, all
  recent builds that succeeded, did do under 2h30. OTOH, when a build hangs,
  it needs to wait for the timeout to expire, wasting time at the current value
  of 5h. Let's compromise at 3 hours and see how it goes?

• [jenkins][installer build] exclude browser app for now: We do not yet publish
  the browser app, so let's skip building it to save time/resources. We may
  revisit when we use the new browser app bundling, recently made available
  upstream in the Theia framework.

How to test

Run the tests locally (currently only covers Electron):

cd theia-blueprint
yarn && yarn build && yarn download:plugins && yarn package:preview
yarn test

A few other things to test. For each make sure that the built-ins are present

• the browser app starts

yarn browser start

• the electron app starts

yarn electron start

• packaging and starting the electron app works

yarn electron package
# test the generated package. e.g. on Linux: `applications/electron/dist/TheiaBlueprint.AppImage`

Review checklist

• as an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• as a reviewer, I agree to behave in accordance with the review guidelines

[marcdumais-work]

Progress: using this PR branch I can successfully build and run Blueprint browser on my Pi 4:

[marcdumais-work]

@vince-fugnitto could you have another look?

[jfaltermeier]

I did not yet have the time for a detailed look, but I did some initial testing.

First, I think the production mode is not fully working. Are we missing a rebuild here?

Reproducer:

git clean -xfd
yarn && yarn build
yarn electron start

leads to
Error: node-loader:
Error: Module did not self-register: '.../applications/electron/lib/backend/native/drivelist.node'.

Also the Browser Docker Image does not seem to be working yet:

Failed to start the backend application:
Error: Cannot find module 'blueprint-product-ext/lib/node/theia-blueprint-backend-module'
    at webpackMissingModule (/home/theia/applications/browser/lib/backend/main.js:328:117)
    at /home/theia/applications/browser/lib/backend/main.js:328:253
    at async Object.start (/home/theia/applications/browser/lib/backend/vendors-node_modules_drivelist_build_Release_drivelist_node-node_modules_stroncium_procfs_lib-63b120.js:4987:20) {
  code: 'MODULE_NOT_FOUND'
}
/home/theia/applications/browser/lib/backend/vendors-node_modules_drivelist_build_Release_drivelist_node-node_modules_stroncium_procfs_lib-63b120.js:4984
    throw reason;
    ^

Error: Cannot find module 'blueprint-product-ext/lib/node/theia-blueprint-backend-module'
    at webpackMissingModule (/home/theia/applications/browser/lib/backend/main.js:328:117)
    at /home/theia/applications/browser/lib/backend/main.js:328:253
    at async Object.start (/home/theia/applications/browser/lib/backend/vendors-node_modules_drivelist_build_Release_drivelist_node-node_modules_stroncium_procfs_lib-63b120.js:4987:20) {
  code: 'MODULE_NOT_FOUND'
}

[marcdumais-work]

@jfaltermeier thanks for the initial look! i have rebased the PR on the latest master and added the couple suggested yarn rebuild. I will try to fix the docker problem and update the PR.

[marcdumais-work]

Also the Browser Docker Image does not seem to be working yet:

I believe that I have fixed the issue with browser.Dockerfile

[jfaltermeier]

Thanks, looks good to me!

[marcdumais-work]

Thanks for the review @jfaltermeier!

@JonasHelming @thegecko FYI, this PR has some changes in it that could potentially be breaking e.g. if someone has a product based on Blueprint, but I do not think it should be anything too difficult to fix. I wonder how e.g. cdt.cloud consumes / extends / depends on Blueprint?

[marcdumais-work]

I wonder how e.g. cdt.cloud consumes / extends / depends on Blueprint?

CDT Cloud Blueprint looks like a fork of this repo, and I see signs it's kept up-to date. So it will be a bigger rebase than usual.

[JonasHelming]

@planger FYI

[marcdumais-work]

@planger if it's welcomed, I can follow-up with a cdt-cloud-blueprint PR, that would update it , so it "fits" with the new Blueprint, after the merge button is pressed. I may require a bit of feedback, since I do not know the ins-and-outs of that product and how it's used. WDYT?

[planger]

@marcdumais-work You are certainly more than welcomed to create a follow-up PR for cdt-cloud-blueprint! Thank you very much! :-) However, if you are very busy, feel free to just let us know and we'll take care of upgrading cdt-cloud-blueprint.

[marcdumais-work]

@planger unfortunately I do not think I will have time to do the follow-up, in the next little while. Please tell the person that eventually does the uplift that they can tag me on the PR if there are questions.

[planger]

@marcdumais-work Ok thanks for letting me know anyways! We'll get back to you if there are questions. Thanks!