Cross-compile kiwix-tools on MIPS

[kelson42]

This is to get kiwix-serve working on most OpenWRT devices

[cm8]

I invested some time into this, but its far from finished (read: compilation results untested)

• kiwixbuild bails out with error in kiwix-lib when targeting mips32r2_static, see
  • pthread static link errors and
  • possible workaround to include in kiwix-lib (?)
• mips32r2_dyn target tested to compile fully

---

There is a major issue preventing 'conventional' cross-compilation of icu4c - the way it's intended to work is build a native icu4c first. The resulting (native to the build host) tools/* ought to execute and process the data/* files in the target icu4c buildtree, in practice this fails:

• The endianess of the data does differ, i.e. host tools and target data/* need to agree on the endianess in order to mix them during build process. Or in other words: icu4c tools built for x86_64 little-endian host will not work with big-endian data thrown at them when -i include directory cli switches are passed during a cross-build.
• This difference, the mismatch, can be seen in filenames and directory names - big endian icu data is prefixed icudt58b, little endian data icudt58l (58 being the major version currently used by kiwix-tools).

The patch included below, written for the icu4c sources, uses qemu-mips-static in order to run the (mips-)native tools on the (x86_64) build host. It works if

• tools are built in the target tree, of course (i.e. --disable-tools must not be supplied to configure)
• QEMU_SET_ENV and QEMU_LD_PREFIX are set correctly in the platform spec

It should only be applied to icu4c sources if the target is mips32r2_*. dependencies/icu4c.py however applies patches in prepares sources stage which affects all possible targets, so careful: the patch below may break cross compilation of icu4c for platforms != mips32r2. There are two routes to solve this

• kiwixbuild script could apply this platform specific patch to icu4c /conditionally/, i.e. if and only if mips32r2 is actually chosen by --target-platform <target> switch. It seems that the Builder classes could benefit from the possibility to apply patches to source fragments copied to BUILD_* directories, before configure is run.
• alter the icu4c patch to be conditional itself, i.e. add logic for icu4c to decide during configure when/how icucross.mk is altered (an environment variable passed by the Builder classes may serve to detect positive cases). (realized now)

(This absolutely needs to be resolved in order for the patch below to be include-able in the kiwixbuild repo. We should not modify build behaviour of other target platforms just to have mips ones build.)

---

Current overall state of cross_compiling to mips using the patch below is of limited use for production routers:

• The toolchain used/tested is the 'official' one in debian/ubuntu main repos (mips-linux-gnu-*) which has NO uclibc target support and apparently cannot be used with soft-float emulation. It may be installed using sudo apt-get gcc-mips-linux-gnu g++-mips-linux-gnu libc6-mips-cross.
• Other toolchains require adjustments to kiwixbuild/platforms/mips32r2.py, should work in theory but have not been tested here.

(Alternatively use buildroot.org and compile kiwixbuild with target native_* within a mips chroot; the compiler is then a mips-native one and thus has nothing to do with the cross-compilation approach followed below. After all, this ticket is about getting cross-compilation to work.)

EDIT 2018-06-17:

• update patch to work with mips32r2_dyn target (QEMU_SET_ENV in icu4c patch adjusted)
• mips32r2_dyn tested to fully (cross-)compile kiwix-tools now
• kiwixbuild_mips_support_wip_2018-06-17.patch.txt

EDIT 2018-06-19:

• fixed mips32r2 icu4c patch affecting other cross-builds
• re-add --disable-tools if cross-building for iOS
• kiwixbuild_mips_support_wip_2018-06-19.patch.txt
• pushed to branch mips32r2 for further development, see c1f281e

[kelson42]

@cm8 Thank you very much. I hope we will be able to merge this first steps to allow further ones to finally have a fully support in our toolchain. Please join #kiwix on Freenode if you want to chat about that.
@mgautierfr Could you please help/guide @cm8 in his work to add mips support to kiwix-build.

[cm8]

I reworked my first comment on this issue with (hopefully) less ambiguous doc and an updated patch version (that is now also linked in case someone prefers that over copy+pasting the syntax highlighted version).

Personally I dislike nagging anyone for help. If this sparks some interest to further work and improve the mips target flavor, then that's what this issue is for. Share freely, not as a burden.

@kelson42 Have you tried the patch? Have you read the comment? Is there room for misinterpretation (if yes where and how could it be improved)? Where you able to reproduce the compilation with the sketched outline?

Answers to these questions help to advance on the issue, imho.

[cm8]

There is another, minor issue in that the libdir name is wrong, i.e. BUILD_mips32r2_dyn/INSTALL/lib/x86_64-linux-gnu/ instead of BUILD_mips32r2_dyn/INSTALL/lib/mips-linux-gnu/. The libraries however are indeed built for mips32r2 target. Since this affects all packages, I suppose this should be fixable within mips32r2.py

[mgautierfr]

Wow, this is a long comment and a lot of thing in it. It is nice to see it.
First of all, thank to work on it, it definitively sparks some interest :)

I haven't tested the patch but here some first comments :

possible workaround to include in kiwix-lib (?)

It seems that it could even be an issue for meson. The solution proposed in SO is general enough to be integrated in the build system itself instead of in each meson.build project. Especially when meson claims to handle everything about thread libraries (http://mesonbuild.com/Threads.html, http://mesonbuild.com/FAQ.html#what-is-the-correct-way-to-use-threads-such-as-pthreads)

There is a major issue preventing 'conventional' cross-compilation of icu4c - the way it's intended to work is build a native icu4c first. The resulting (native to the build host) tools/* ought to execute and process the data/* files in the target icu4c buildtree, in practice this fails:

The endianess of the data does differ, i.e. host tools and target data/* need to agree on the endianess in order to mix them during build process. Or in other words: icu4c tools built for x86_64 little-endian host will not work with big-endian data thrown at them when -i include directory cli switches are passed during a cross-build.
This difference, the mismatch, can be seen in filenames and directory names - big endian icu data is prefixed icudt58b, little endian data icudt58l (58 being the major version currently used by kiwix-tools).

Yes, since now we always have little endian arch so the problem doesn't occur and I avoid it :)
There are another thinks to fix also, kiwix-lib/kiwix-build have hard coded icudt58l filename, this should be changed.

tools are built in the target tree, of course (i.e. --disable-tools must not be supplied to configure)

--disable-tools were added to be allow to cross-compile to iOS from macOS. (6d507651a), so it should not be simply removed.

It should only be applied to icu4c sources if the target is mips32r2_*. dependencies/icu4c.py however applies patches in prepares sources stage which affects all possible targets, so careful: the patch below may break cross compilation of icu4c for platforms != mips32r2. There are two routes to solve this

kiwixbuild script could apply this platform specific patch to icu4c /conditionally/, i.e. if and only if mips32r2 is actually chosen by --target-platform switch. It seems that the Builder classes could benefit from the possibility to apply patches to source fragments copied to BUILD_* directories, before configure is run.
alter the icu4c patch to be conditional itself, i.e. add logic for icu4c to decide during configure when/how icucross.mk is altered (an environment variable passed by the Builder classes may serve to detect positive cases).

Yes, the SOURCE directory must be arch/platform independent. The first solution is possible but I would prefer the second solution. If it is too complicated somehow, it may be not the right approach. cross-compiling to a different endianess should be handled by the icu4c (build system) directly, either we miss something or icu4c should be fixed. (but I would accept the first solution if it seems to "good" one after all)

There is another, minor issue in that the libdir name is wrong, i.e. BUILD_mips32r2_dyn/INSTALL/lib/x86_64-linux-gnu/ instead of BUILD_mips32r2_dyn/INSTALL/lib/mips-linux-gnu/. The libraries however are indeed built for mips32r2 target. Since this affects all packages, I suppose this should be fixable within mips32r2.py

Nice catch. You are right, the detected lib dir is the one of the build platform, not the host one. As you said, it is not related to mips. However it would not be so easy to fix. Even for native compilation (and so same arch), the libdir may change depending of the distribution (lib64 for fedora, lib/x86_64-linux-gnu for debian/ubuntu) (It is somehow related to mesonbuild/meson#1972)
If you have a solution, it would be nice to have it in a separated PR.

Hope it helps.
Do not hesitate to create a PR (not an issue), it will be easier to comment on the patch itself.

[cm8]

The icu4c patch bits were altered.

• when icucross.mk is created for the native build, makefile code to detect an endian mismatch is written to it. qemu-$(host_cpu)-static execution will only be used, if there is a mismatch. Otherwise the previous code runs unaltered. (host_cpu being the target host we are cross-compiling for)
• host_cpu is read from Makefile.inc, so it may work when cross-compiling for other big endian targets != mips on little endian build machines.
• in icu4c.py --disable-tools is added to options specifically when cross-compiling for iOS now, however if there are (!?) big endian machines running iOS, then cross-compiling icu4c for iOS on a little-endian build machine will fail - so it may be better to re-implement the test that is now done in icucross.mk within the scope of icu4c.py (check it the python way, i.e. make the option conditional to difference in endianess between build machine and target host)

It should be in good enough shape to be added to the repo, but some of the bespoken problems remain:

• hardcoded values of icudt58l in some dependent packages (unnoticable during compile, but will affect runtimes)
• the libdir name problem
• pthread linking problem when compiling statically (i.e. mips32r2_static instead of mips32r2_dyn)

It would be nice if someone actually tested/checked that other platforms still compile ident, before commiting. While I'd think it should be ok, I've not tested this assumption.

[cm8]

I'd like to push this to master in a couple of weeks if noone has objections or finds side effects affecting other platforms, unless you absolutely want this in an extra branch first, that is. Pushing it to master may lead to other people creating issues about its unfinished state, but that way we will find out for sure, whats still needed to support mips, I guess. Again, peer tests, improvements and comments here are welcome.

[kelson42]

@cm8 It's of course up to you to decide how you want to proceed, but in general I push for the step-by-step approach - so it is not a problem to merge piece by piece what is necessary to get MIPS working properly - to the contrary. The continuous integration should catch any problem with other architectures anyway before merging.

[kelson42]

@cm8 thx for your commit/branch, feel free to make a PR and assign it to @mgautierfr whenever you think this is the right moment.

[kelson42]

@cm8 @mgautierfr is currently extremely busy with other stuff. Thank your patience.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.