Migration to use of upstream Linuxbrew/Homebrew

[drbenmorgan]

As noted in #50 and in SuperNEMO-DBD/brew#12, the fork of Linuxbrew is a bit out of date. To reduce the maintenance burden and better support the upstream, we'll be moving to full use of Linuxbrew/Homebrew. Other than this basic reason, Home/Linuxbrew have a medium term plan to merge development for better cross-platform experience, making our fork redundant.

This does not affect existing, working, installs of the software. These can be retained until the upgrade is complete (before the next Falaise release).

This issue will keep track of developments and fixes required to enable this, including:

• Update of bootstrapping script/procedure when we only have the tap
• Update of documentation on new workflow
• Update of SuperNEMO-DBD/cadfael-deployment Docker images, or move of Dockerfile etc into this repository.

We also document the minimum steps needed to install Supernemo software in the intermediate period:

1. General setup:

$ git clone https://github.com/Linuxbrew/brew.git
$ export PATH=$(pwd)/brew/bin:$PATH
$ export HOMEBREW_NO_AUTO_UPDATE=1
$ brew tap supernemo-dbd/cadfael
$ brew tap-pin supernemo-dbd/cadfael

2. macOS setup/install

$ brew doctor
... check for errors ...
$ brew install falaise
$ brew test falaise

3. Linux (assumed CentOS/Ubuntu for now)

$ export HOMEBREW_BUILD_FROM_SOURCE=1
... HOMEBREW_ARCH is optional, only use if you expect to use this install across a range of machines with differing CPUs ...
$ export HOMEBREW_ARCH=core2
$ brew doctor
... check for errors ...
$ brew install gcc@7 --without-glibc
$ brew install falaise --cc=gcc-7
$ brew test falaise

We use the build-from-source environment variable on Linux so that we can use the system glibc and thus make use of system graphics and drivers. GCC is also installed without linuxbrew's glibc for now.

[drbenmorgan]

I've assigned WG coordinators so you are aware of this WIP and can point your groups to this info as/when needed. No action is otherwise required, and you can continue to use existing installs of the software.

[edbird]

Update for Debian 9.8, as of 2019-02-28

brew snemo-bootstrap
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
/home/ecb/snemo-sdk/bin/brew tap homebrew/bundle
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
==> Tapping homebrew/bundle
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Cloning into '/home/ecb/snemo-sdk/Library/Taps/homebrew/homebrew-bundle'...
remote: Enumerating objects: 84, done.
remote: Counting objects: 100% (84/84), done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 84 (delta 6), reused 31 (delta 1), pack-reused 0
Unpacking objects: 100% (84/84), done.
Tapped 1 command (177 files, 638.5KB).
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
Installing gcc@7 formula. It is not currently installed.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
==> Installing dependencies for gcc@7: m4, gmp, isl, mpfr, libmpc and zlib
==> Installing gcc@7 dependency: m4
==> Downloading https://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.xz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/m4/1.4.18
==> make
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/m4/1.4.18: 13 files, 625.4KB, built in 1 minute
==> Installing gcc@7 dependency: gmp
==> Downloading https://gmplib.org/download/gmp/gmp-6.1.2.tar.xz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/gmp/6.1.2_2 --enable-cxx --with-pic --build=core2-linux-gnu
==> make
==> make check
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/gmp/6.1.2_2: 20 files, 3.7MB, built in 2 minutes 3 seconds
==> Installing gcc@7 dependency: isl
==> Downloading http://isl.gforge.inria.fr/isl-0.20.tar.xz
######################################################################## 100.0%
==> ./configure --disable-silent-rules --prefix=/home/ecb/snemo-sdk/Cellar/isl/0.20 --with-gmp=system --with-gmp-prefix
==> make check
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/isl/0.20: 72 files, 5.4MB, built in 1 minute
==> Installing gcc@7 dependency: mpfr
==> Downloading https://ftp.gnu.org/gnu/mpfr/mpfr-4.0.1.tar.xz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/mpfr/4.0.1 --disable-silent-rules
==> make
==> make check
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/mpfr/4.0.1: 29 files, 5.1MB, built in 1 minute 14 seconds
==> Installing gcc@7 dependency: libmpc
==> Downloading https://ftp.gnu.org/gnu/mpc/mpc-1.1.0.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/libmpc/1.1.0 --with-gmp=/home/ecb/snemo-sdk/opt/gmp --with-mpfr=/ho
==> make
==> make check
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/libmpc/1.1.0: 13 files, 496.6KB, built in 31 seconds
==> Installing gcc@7 dependency: zlib
==> Downloading https://zlib.net/zlib-1.2.11.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/zlib/1.2.11
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/zlib/1.2.11: 12 files, 450KB, built in 9 seconds
Warning: gcc@7: this formula has no --without-glibc option so it will be ignored!
==> Installing gcc@7
==> Downloading https://ftp.gnu.org/gnu/gcc/gcc-7.4.0/gcc-7.4.0.tar.xz
######################################################################## 100.0%
==> ../configure --with-bugurl=https://github.com/Linuxbrew/homebrew-core/issues --disable-multilib --prefix=/home/ecb/
==> make
==> make install-strip
==> Creating the GCC specs file: /home/ecb/snemo-sdk/Cellar/gcc@7/7.4.0/lib/gcc/7/gcc/x86_64-pc-linux-gnu/7.4.0/specs
🍺  /home/ecb/snemo-sdk/Cellar/gcc@7/7.4.0: 1,432 files, 193.8MB, built in 22 minutes 47 seconds
Installing gcc@7
Installing patchelf formula. It is not currently installed.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
Warning: You passed `--cc=gcc-7`.
You may encounter build failures with some formulae as
brew is being run in an unsupported configuration
In these cases, please raise an issue at https://github.com/SuperNEMO-DBD/homebrew-core/issues

==> Downloading https://nixos.org/releases/patchelf/patchelf-0.9/patchelf-0.9.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/patchelf/0.9_1 CXXFLAGS=-static-libgcc -static-libstdc++ --disable-
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/patchelf/0.9_1: 7 files, 529.3KB, built in 7 seconds
Installing patchelf
Installing python formula. It is not currently installed.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
Warning: You passed `--cc=gcc-7`.
You may encounter build failures with some formulae as
brew is being run in an unsupported configuration
In these cases, please raise an issue at https://github.com/SuperNEMO-DBD/homebrew-core/issues

==> Installing dependencies for python: pkg-config, gdbm, openssl, gpatch, ncurses, readline, sqlite, xz, bzip2 and libffi
==> Installing python dependency: pkg-config
==> Downloading https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/pkg-config/0.29.2_1 --disable-host-tool --with-internal-glib --with
==> make
==> make check
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/pkg-config/0.29.2_1: 11 files, 730.8KB, built in 1 minute 24 seconds
==> Installing python dependency: gdbm
==> Downloading https://ftp.gnu.org/gnu/gdbm/gdbm-1.18.1.tar.gz
######################################################################## 100.0%
==> ./configure --disable-silent-rules --without-readline --prefix=/home/ecb/snemo-sdk/Cellar/gdbm/1.18.1 --enable-libg
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/gdbm/1.18.1: 40 files, 1MB, built in 23 seconds
==> Installing python dependency: openssl
==> Downloading https://www.openssl.org/source/openssl-1.0.2q.tar.gz
######################################################################## 100.0%
==> perl ./Configure --prefix=/home/ecb/snemo-sdk/Cellar/openssl/1.0.2q_2 --openssldir=/home/ecb/snemo-sdk/etc/openssl 
==> make depend
==> make
==> make test
==> make install MANDIR=/home/ecb/snemo-sdk/Cellar/openssl/1.0.2q_2/share/man MANSUFFIX=ssl
==> Downloading https://curl.haxx.se/ca/cacert-2019-01-23.pem
######################################################################## 100.0%
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
  /home/ecb/snemo-sdk/etc/openssl/certs

and run
  /home/ecb/snemo-sdk/opt/openssl/bin/c_rehash
==> Summary
🍺  /home/ecb/snemo-sdk/Cellar/openssl/1.0.2q_2: 1,794 files, 14.2MB, built in 7 minutes 11 seconds
==> Installing python dependency: gpatch
==> Downloading https://ftp.gnu.org/gnu/patch/patch-2.7.6.tar.xz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/gpatch/2.7.6
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/gpatch/2.7.6: 10 files, 364.7KB, built in 1 minute 29 seconds
==> Installing python dependency: ncurses
==> Downloading https://ftp.gnu.org/gnu/ncurses/ncurses-6.1.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/ncurses/6.1 --enable-pc-files --with-pkg-config-libdir=/home/ecb/sn
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/ncurses/6.1: 3,856 files, 9.0MB, built in 1 minute 22 seconds
==> Installing python dependency: readline
==> Downloading https://ftp.gnu.org/gnu/readline/readline-8.0.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/readline/8.0.0_1 --with-curses
==> make install SHLIB_LIBS=-lcurses
🍺  /home/ecb/snemo-sdk/Cellar/readline/8.0.0_1: 48 files, 1.9MB, built in 31 seconds
==> Installing python dependency: sqlite
==> Downloading https://sqlite.org/2019/sqlite-autoconf-3270100.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/home/ecb/snemo-sdk/Cellar/sqlite/3.27.1 --enable-dynamic-extensions --enable-readline --disab
==> make install
🍺  /home/ecb/snemo-sdk/Cellar/sqlite/3.27.1: 12 files, 4.4MB, built in 1 minute 39 seconds
==> Installing python dependency: xz
==> Downloading https://downloads.sourceforge.net/project/lzmautils/xz-5.2.4.tar.gz
==> Downloading from https://netcologne.dl.sourceforge.net/project/lzmautils/xz-5.2.4.tar.gz
######################################################################## 100.0%
==> ./configure --disable-silent-rules --prefix=/home/ecb/snemo-sdk/Cellar/xz/5.2.4
==> make check
Last 15 lines from /home/ecb/.cache/Homebrew/Logs/xz/02.make:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
unexpected output from xzgrep
FAIL: test_scripts.sh
=========================================
1 of 9 tests failed
Please report to [email protected]
=========================================
Makefile:592: recipe for target 'check-TESTS' failed
make[2]: *** [check-TESTS] Error 1
make[2]: Leaving directory '/tmp/xz-20190227-9074-yrtebj/xz-5.2.4/tests'
Makefile:715: recipe for target 'check-am' failed
make[1]: *** [check-am] Error 2
make[1]: Leaving directory '/tmp/xz-20190227-9074-yrtebj/xz-5.2.4/tests'
Makefile:613: recipe for target 'check-recursive' failed
make: *** [check-recursive] Error 1

READ THIS: https://github.com/SuperNEMO-DBD/brew/issues
Please do not report this issue to Homebrew/brew or Homebrew/core,
which support macOS only."

Installing python has failed!
Installing cmake formula. It is not currently installed.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
Warning: You passed `--cc=gcc-7`.
You may encounter build failures with some formulae as
brew is being run in an unsupported configuration
In these cases, please raise an issue at https://github.com/SuperNEMO-DBD/homebrew-core/issues

==> Installing dependencies for cmake: xz, bzip2, libffi, python and sphinx-doc
==> Installing cmake dependency: xz
==> Downloading https://downloads.sourceforge.net/project/lzmautils/xz-5.2.4.tar.gz
Already downloaded: /home/ecb/.cache/Homebrew/downloads/60057e32fccba99ff98a423a0d5bbb3299325346de36997e4e59bb96a3c2da2e--xz-5.2.4.tar.gz
==> ./configure --disable-silent-rules --prefix=/home/ecb/snemo-sdk/Cellar/xz/5.2.4
==> make check
Last 15 lines from /home/ecb/.cache/Homebrew/Logs/xz/02.make:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
unexpected output from xzgrep
FAIL: test_scripts.sh
=========================================
1 of 9 tests failed
Please report to [email protected]
=========================================
Makefile:592: recipe for target 'check-TESTS' failed
make[2]: *** [check-TESTS] Error 1
make[2]: Leaving directory '/tmp/xz-20190227-22885-1t0h8fy/xz-5.2.4/tests'
Makefile:715: recipe for target 'check-am' failed
make[1]: *** [check-am] Error 2
make[1]: Leaving directory '/tmp/xz-20190227-22885-1t0h8fy/xz-5.2.4/tests'
Makefile:613: recipe for target 'check-recursive' failed
make: *** [check-recursive] Error 1

READ THIS: https://github.com/SuperNEMO-DBD/brew/issues
Please do not report this issue to Homebrew/brew or Homebrew/core,
which support macOS only."

Installing cmake has failed!
Installing ninja formula. It is not currently installed.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
Warning: You passed `--cc=gcc-7`.
You may encounter build failures with some formulae as
brew is being run in an unsupported configuration
In these cases, please raise an issue at https://github.com/SuperNEMO-DBD/homebrew-core/issues

==> Installing dependencies for ninja: xz, bzip2, libffi and python
==> Installing ninja dependency: xz
==> Downloading https://downloads.sourceforge.net/project/lzmautils/xz-5.2.4.tar.gz
Already downloaded: /home/ecb/.cache/Homebrew/downloads/60057e32fccba99ff98a423a0d5bbb3299325346de36997e4e59bb96a3c2da2e--xz-5.2.4.tar.gz
==> ./configure --disable-silent-rules --prefix=/home/ecb/snemo-sdk/Cellar/xz/5.2.4
==> make check
Last 15 lines from /home/ecb/.cache/Homebrew/Logs/xz/02.make:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
unexpected output from xzgrep
FAIL: test_scripts.sh
=========================================
1 of 9 tests failed
Please report to [email protected]
=========================================
Makefile:592: recipe for target 'check-TESTS' failed
make[2]: *** [check-TESTS] Error 1
make[2]: Leaving directory '/tmp/xz-20190227-4693-1wh50on/xz-5.2.4/tests'
Makefile:715: recipe for target 'check-am' failed
make[1]: *** [check-am] Error 2
make[1]: Leaving directory '/tmp/xz-20190227-4693-1wh50on/xz-5.2.4/tests'
Makefile:613: recipe for target 'check-recursive' failed
make: *** [check-recursive] Error 1

READ THIS: https://github.com/SuperNEMO-DBD/brew/issues
Please do not report this issue to Homebrew/brew or Homebrew/core,
which support macOS only."

Installing ninja has failed!
Installing falaise formula. It is not currently installed.
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/home/ecb/snemo-sdk/Library/Homebrew/brew.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
Warning: You passed `--cc=gcc-7`.
You may encounter build failures with some formulae as
brew is being run in an unsupported configuration
In these cases, please raise an issue at https://github.com/SuperNEMO-DBD/homebrew-core/issues

==> Tapping linuxbrew/extra
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Cloning into '/home/ecb/snemo-sdk/Library/Taps/linuxbrew/homebrew-extra'...
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 22 (delta 0), reused 9 (delta 0), pack-reused 0
Unpacking objects: 100% (22/22), done.
Tapped 8 formulae (61 files, 223.1KB).
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
==> Installing dependencies for falaise: xz, bzip2, libffi, python, sphinx-doc, cmake, bison, help2man, gettext, flex, doxygen, icu4c, boost, camp, clhep, libbsd, expat, curl, xerces-c, geant4, gsl, libpng, freetype, berkeley-db, perl, autoconf, automake, gperf, libtool, json-c, util-linux, fontconfig, qt5-base, libxml2, xrootd, xxhash, lz4, root6 and bayeux
==> Installing falaise dependency: xz
==> Downloading https://downloads.sourceforge.net/project/lzmautils/xz-5.2.4.tar.gz
Already downloaded: /home/ecb/.cache/Homebrew/downloads/60057e32fccba99ff98a423a0d5bbb3299325346de36997e4e59bb96a3c2da2e--xz-5.2.4.tar.gz
==> ./configure --disable-silent-rules --prefix=/home/ecb/snemo-sdk/Cellar/xz/5.2.4
==> make check
Last 15 lines from /home/ecb/.cache/Homebrew/Logs/xz/02.make:
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
unexpected output from xzgrep
FAIL: test_scripts.sh
=========================================
1 of 9 tests failed
Please report to [email protected]
=========================================
Makefile:592: recipe for target 'check-TESTS' failed
make[2]: *** [check-TESTS] Error 1
make[2]: Leaving directory '/tmp/xz-20190227-18636-124hbb7/xz-5.2.4/tests'
Makefile:715: recipe for target 'check-am' failed
make[1]: *** [check-am] Error 2
make[1]: Leaving directory '/tmp/xz-20190227-18636-124hbb7/xz-5.2.4/tests'
Makefile:613: recipe for target 'check-recursive' failed
make: *** [check-recursive] Error 1

READ THIS: https://github.com/SuperNEMO-DBD/brew/issues
Please do not report this issue to Homebrew/brew or Homebrew/core,
which support macOS only."

Installing falaise has failed!
Homebrew Bundle failed! 4 Brewfile dependencies failed to install.

Hope the is the right place

[drbenmorgan]

@edbird please create a new issue on SuperNEMO-dbd/brew with the details it asks for!

[drbenmorgan]

Closing in favour of new issues on SuperNEMO-DBD/brew.