Update build workflow for Linux

[baierjan]

The Windows workflow is mostly untouched.

I updated the Linux workflow to run inside a newer container to build against newer system libraries which should result in more portable builds, hopefully targeting issue #107.

A new job has been added. This build job creates binary inside steamrt soldier container (issue #183).

I have no idea how to properly test the steam binary, though. I took the result of the builds and tested it inside fairly recent snapshot of openSUSE Tumbleweed, it was working well (the win binary worked under wine).

[Ext3h]

Do we need to deploy Qt in the Steam Runtime build, same as we do on Windows?

[baierjan]

The Qt version in the Steam container is unfortunately to old to be used, so an external one is needed for sure. When I tried to run your former linux binary (made with the install-qt-action), my system complained about some missing QT symbols. I am not entirely sure what the install-qt-action does as it includes a lot of abstraction (I assume those python scripts try to download some pre-compiled packages), however I think compiling the QT on the build system against the other libraries yields much more portable results.

We can maybe improve the process a little as we use more or less the same container over and over and introduce a new workflow to pre-compile the QT and use it as an artifact during the steam build. What do you think?

[Ext3h]

Checkout action v2 doesn't function inside the container?

[baierjan]

Good catch. v2 is using node, which in turn requires a minimum version of GLIBC which is not present in the scout runtime https://github.com/baierjan/Ingnomia/runs/4519417849?check_suite_focus=true#step:3:17

I replaced the container with the soldier a while after and totally forgot to retest the compatibility with the v2 after removing scout. Apparently, if we no longer need scout, we can use v2 everywhere. I will update the request.

[Ext3h]

Same with all the steps replaced by naked wget / unzip further down I suppose? Even if it's more overhead, keeping it uniform for readability is preferred.

[baierjan]

I will retest that. Initially, I was thinking about downloading and extracting all the dependencies in one step on Linuxes and/or extracting it into a custom action inside the repository as I also do not like skipping every other step just because of the target architecture.

[Ext3h]

I expect pulling in backports is not going to result in a build compatible with the Steam Runtime.

[baierjan]

From the documentation at https://gitlab.steamos.cloud/steamrt/steamrt/-/tree/steamrt/soldier#preferred-versions-to-backport, mainly from

The most likely version to be able to backport is the version from the next stable release of Debian, for example backporting from Debian 11 into soldier - this matches official Debian backports. We can use a version from buster-backports as-is, if it exists.

I got the idea, it should not be a problem. After all we need that only for the cmake. On the other hand, it might be a good idea to also limit apt to only use backports repository when stated explicitly (weirdly enough, that is the default which was actually changed by Steam).

[Ext3h]

The Python scripts are downloading pre-built Qt from qt.io, reasonable choice if the correct version isn't available from regular package sources. Building Qt on your own is a hour-long job on these nodes.

When building Qt on your own, going for the static build would be the reasonable choice though. We may do so, as Qt is licensed under GPL 3.0 (and some components under LGPL v3), either of which we are compatible with.

We are not actually using most of Qt, so at least in terms of compile size that would be worth it.

Looks like we don't get around deploying all of SFML, Qt, Noesis and Steam API when targeting the Steam Runtime.
Only OpenAL itself is shipped with a known compatible version from Steam side.

On Windows the same, we can get Steam to install OpenAL system wide for us, so same stuff needs to be deployed.

[baierjan]

So I did some experiments. The new Steam runtime is compatible with the new actions, that is good. However it is not compatible with the install-qt-action as it tries to install some predefined set of packages which are not present in the runtime: https://github.com/baierjan/Ingnomia/runs/4548324452?check_suite_focus=true#step:8:34

I tweaked the Qt installation process to only checkout the qtbase submodule as no other module is needed. That took the time down to 25 minutes (without any cache). I would say that is a good start.

The windows build is still mostly untouched, the plain linux build is linked against reasonable new libraries and thus should be playable on recent systems without problems.

[Ext3h]

Those missing packages came to Debian in Bulseye, while Soldier runtime is based on Buster. And why it's even installing these, is to work around a (now fixed) Qt bug: https://bugreports.qt.io/browse/QTBUG-88688

Should we go straight for Heavy instead of Soldier runtime?

I'm still not happy with building Qt.

[baierjan]

I fixed my issue with caching and now it behaves almost the same as downloading pre-build binaries (at least timewise).

It seems to me, that there is no other way around for the steam builds. You can either

1. use the Qt version available in Buster (5.11.3) which would unfortunately need a lot of changes in the current codebase
2. make your own / fix the install-qt-action to count also with Debian and use correct dependencies (instead of some random ones) on https://github.com/jurplel/install-qt-action/blob/10cbdcd020ebd47297c2c40d37d610fed13f7d93/src/main.ts#L25
3. compile the needed Qt version yourself once and use it (my current solution)
4. wait for Steam Runtime 3 (Sniper) which is currently not yet available at all; in fact the Heavy will not help as it is even older that Soldier (Heavy is based on Debian 8), so Debian 10 is basically the most recent one

It is basically your choice as a maintainer, from my point of view, I would use option 3 as it is basically done and it is easy and then go for number 4, when that option become possible.

I am not sure if and how good we can test compatibility of the binary with the steam runtime, we might be able to tweak the runtime image more than we think.

[Ext3h]

Okay, if the caching works, then compilation isn't such a bad option after all.

Still, a couple of options please:

• -force-debug-info (Qt strips symbols per default - we have crash reporting in place which depends on those!)
• -no-gif -no-freetype -no-libjpeg -no-harfbuzz -no-openssl -no-libproxy -no-pulseaudio -no-alsa -no-nis -no-cups -no-evdev -no-tslib -no-icu -no-fontconfig -no-dbus -no-gstreamer -no-eglfs -no-linuxfb -no-kms -no-gtkstyle -opengl desktop (we don't need most of the functionality with external dependencies, not even from the base module)
• -no-avx -no-avx2 (we can't assume that the target hardware platform supports that, Qt build with native arch by default!)
• -qt-libpng -qt-sql-sqlite (we actually need those)

Please let's have a look at what's still left in the bin/ folder after reducing it such. And then we'll have to decide on -static or not, either way we have to deploy it.

[baierjan]

I applied requested options (I needed to experiment a little as some of them are were not valid, probably because I only do a checkout of the qtbase submodule).

This is the configure output inside the runtime

Build type: linux-g++ (x86_64, CPU features: mmx sse sse2)
Compiler: gcc 8.3.0
Configuration: sse2 aesni compile_examples enable_new_dtags force_debug_info largefile precompile_header rdrnd shani sse3 ssse3 sse4_1 sse4_2 x86SimdAlways shared shared rpath release c++11 c++14 c++1z concurrent reduce_exports reduce_relocations stl
Build options:
  Mode ................................... release (with debug info)
  Optimize release build for size ........ no
  Building shared libraries .............. yes
  Using C standard ....................... C11
  Using C++ standard ..................... C++17
  Using ccache ........................... no
  Using new DTAGS ........................ yes
  Generating GDB index ................... no
  Relocatable ............................ yes
  Using precompiled headers .............. yes
  Using LTCG ............................. no
  Target compiler supports:
    SSE .................................. SSE2 SSE3 SSSE3 SSE4.1 SSE4.2
    AVX .................................. <none>
    AVX512 ............................... <none>
    Other x86 ............................ AES RDRAND SHA
    Intrinsics without -mXXX option ...... yes
  Build parts ............................ libs tools
Qt modules and options:
  Qt Concurrent .......................... yes
  Qt D-Bus ............................... no
  Qt D-Bus directly linked to libdbus .... no
  Qt Gui ................................. yes
  Qt Network ............................. yes
  Qt Sql ................................. yes
  Qt Testlib ............................. yes
  Qt Widgets ............................. yes
  Qt Xml ................................. yes
Support enabled for:
  Using pkg-config ....................... yes
  udev ................................... yes
  Using system zlib ...................... yes
  Zstandard support ...................... no
Qt Core:
  DoubleConversion ....................... yes
    Using system DoubleConversion ........ yes
  GLib ................................... yes
  iconv .................................. yes
  ICU .................................... no
  Built-in copy of the MIME database ..... yes
  Tracing backend ........................ <none>
  Logging backends:
    journald ............................. no
    syslog ............................... no
    slog2 ................................ no
  PCRE2 .................................. yes
    Using system PCRE2 ................... yes
Qt Network:
  getifaddrs() ........................... yes
  IPv6 ifname ............................ yes
  libproxy ............................... no
  Linux AF_NETLINK ....................... yes
  OpenSSL ................................ no
    Qt directly linked to OpenSSL ........ no
  OpenSSL 1.1 ............................ no
  DTLS ................................... no
  OCSP-stapling .......................... no
  SCTP ................................... no
  Use system proxies ..................... yes
  GSSAPI ................................. no
Qt Gui:
  Accessibility .......................... yes
  FreeType ............................... yes
    Using system FreeType ................ no
  HarfBuzz ............................... no
    Using system HarfBuzz ................ no
  Fontconfig ............................. no
  Image formats:
    GIF .................................. no
    ICO .................................. yes
    JPEG ................................. no
      Using system libjpeg ............... no
    PNG .................................. yes
      Using system libpng ................ no
  Text formats:
    HtmlParser ........................... yes
    CssParser ............................ yes
    OdfWriter ............................ yes
    MarkdownReader ....................... yes
      Using system libmd4c ............... no
    MarkdownWriter ....................... yes
  EGL .................................... yes
  OpenVG ................................. no
  OpenGL:
    Desktop OpenGL ....................... yes
    OpenGL ES 2.0 ........................ no
    OpenGL ES 3.0 ........................ no
    OpenGL ES 3.1 ........................ no
    OpenGL ES 3.2 ........................ no
  Vulkan ................................. yes
  Session Management ..................... yes
Features used by QPA backends:
  evdev .................................. no
  libinput ............................... yes
  INTEGRITY HID .......................... no
  mtdev .................................. yes
  tslib .................................. no
  xkbcommon .............................. yes
  X11 specific:
    XLib ................................. yes
    XCB Xlib ............................. yes
    EGL on X11 ........................... yes
QPA backends:
  DirectFB ............................... no
  EGLFS .................................. no
  LinuxFB ................................ no
  VNC .................................... yes
  XCB:
    Using system-provided XCB libraries .. yes
    XCB XKB .............................. yes
    XCB XInput ........................... no
    Native painting (experimental) ....... no
    GL integrations:
      GLX Plugin ......................... yes
        XCB GLX .......................... yes
      EGL-X11 Plugin ..................... yes
Qt Sql:
  SQL item models ........................ yes
Qt Widgets:
  GTK+ ................................... yes
  Styles ................................. Fusion Windows
Qt PrintSupport:
  CUPS ................................... no
Qt Sql Drivers:
  DB2 (IBM) .............................. no
  InterBase .............................. no
  MySql .................................. no
  OCI (Oracle) ........................... no
  ODBC ................................... no
  PostgreSQL ............................. no
  SQLite2 ................................ no
  SQLite ................................. yes
    Using system provided SQLite ......... no
  TDS (Sybase) ........................... no
Qt Testlib:
  Tester for item models ................. yes

The /bin folder is quite empty

/usr/local/Qt-5.14.2/bin#  ls
fixqt4headers.pl  moc  qlalr  qmake  qvkgen  rcc  syncqt.pl  tracegen  uic

and we do have a few libraries

/usr/local/Qt-5.14.2/lib# ls *.so
libQt5Concurrent.so
libQt5Core.so
libQt5Gui.so
libQt5Network.so
libQt5OpenGL.so
libQt5PrintSupport.so
libQt5Sql.so
libQt5Test.so
libQt5Widgets.so
libQt5XcbQpa.so
libQt5Xml.so

[stefaang]

might be faster to lock these in the container image

[baierjan]

I am not sure what do you mean by that? That would require to use own container image instead of a generic ubuntu:21.04. Although it would probably be easy to do it, you only spare 1 minute from the total time.