Missing 8-byte hashes for SSL certificates

[mariospr]

As @erikos found out when we upgraded to flatpak 0.8.2 in Endless, the freedesktop runtime is not installing 8-byte hashes under /etc/ssl/certs, which are used by OpenSSL to look up the certificates, as per http://www.gagravarr.org/writing/openssl-certs/others.shtml. This is a problem because it makes it impossible for flatpak-builder to download sources for modules using the https:// protocol, which forced us to put this workaround in place for now: endlessm/gnome-weather@5f01b2a

We initially were a bit confused because this used to work on previous versions of flatpak (up to 0.8.1), but it seems the difference is this commit, as before that point flatpak would bind mount the host's /etc/ssl/certs contents too, and then the 8-byte hash files would be available inside the sandbox.

Investigating a bit how this files are generated in debian, it seems like those hashes are created by a c_rehash tool that comes with the openssl package, which gets called from update-ca-certicates (from the ca-certificates package)... which is then executed on the debian/postinst script of the ca-certificates package:

#! /bin/sh
# postinst script for ca-certificates
#
# see: dh_installdeb(1)

[...]

case "$1" in
    configure)
        if [ ! -e /usr/local/share/ca-certificates ]
        then
            if mkdir /usr/local/share/ca-certificates 2>/dev/null
            then
                chown root:staff /usr/local/share/ca-certificates
                chmod 2775 /usr/local/share/ca-certificates
            fi
        fi

        . /usr/share/debconf/confmodule
        db_version 2.0
        db_capb multiselect
        db_metaget ca-certificates/enable_crts choices
        CERTS_AVAILABLE="$RET"
        db_get ca-certificates/enable_crts
        CERTS_ENABLED="$RET"
        # XXX unmark seen for next configuration
        db_fset ca-certificates/new_crts seen false
        db_stop || true
        if test -f /etc/ca-certificates.conf; then

           [...]

        else
          # new file
          cat > /etc/ca-certificates.conf <<EOF
# This file lists certificates that you wish to use or to ignore to be
# installed in /etc/ssl/certs.
# update-ca-certificates(8) will update /etc/ssl/certs by reading this file.
#
# This is autogenerated by dpkg-reconfigure ca-certificates.
# Certificates should be installed under /usr/share/ca-certificates
# and files with extension '.crt' is recognized as available certs.
#
# line begins with # is comment.
# line begins with ! is certificate filename to be deselected.
#
EOF
          (echo $CERTS_ENABLED | tr ',' '\n'; \
           echo $CERTS_AVAILABLE | tr ',' '\n') | \
            sed -e 's/^[[:space:]]*//' | \
            sort | uniq -c | \
            sed -e 's/^[[:space:]]*2[[:space:]]*//' \
                -e 's/^[[:space:]]*1[[:space:]]*/!/' \
            >> /etc/ca-certificates.conf
        fi
        # update /etc/ssl/certs without running the hooks
        # fix bogus symlink to ca-certificates.crt on upgrades; see
        # Debian #643667; drop after wheezy
        if dpkg --compare-versions "$2" lt-nl 20111025; then
            update-ca-certificates --hooksdir "" --fresh
        else
            update-ca-certificates --hooksdir ""
        fi
        # deferred update of /etc/ssl/certs including running the hooks
        dpkg-trigger --no-await update-ca-certificates
    ;;

    triggered)
        for trigger in $2; do
            case "$trigger" in
                update-ca-certificates)
                    update-ca-certificates
                    ;;
                update-ca-certificates-fresh)
                    update-ca-certificates --fresh
                    ;;
                *)
                    echo "postinst called with unknown trigger \`$2'">&2
                    exit 1
                    ;;
            esac;
        done;
        ;;

    abort-upgrade|abort-remove|abort-deconfigure)

    ;;

[...]

To test this theory, I manually removed all the hashes from my /etc/ssl/certs directory, and then re-run sudo update-ca-certificates --fresh -v manually... and all the hashes were generated again.

So, at this point it seems like the right solution to this problem would be to somehow do something similar to what Debian does at the time of creating the runtime, and generate those 8-byte hashes files inside /etc/ssl/certs, so that they are available inside the sandbox now that flatpak does not mount the hosts's certs directory.

What I'm not that certain of is how exactly that should be achieved so, while I'm happy to work on this if needed, I could use some guidance to at least know where to touch the right things.

[alexlarsson]

For whatever reason it seems yocto is disabling the c_rehash calls:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-support/ca-certificates/ca-certificates/0001-update-ca-certificates-remove-c-rehash.patch?h=jethro

We need to figure out why, and if we should drop that patch in our builds.

[alexlarsson]

Also, i don't seem to have the hash files on my fedora machine...

[alexlarsson]

This seems to be related:
https://lists.yoctoproject.org/pipermail/yocto/2015-July/025774.html
I think we have to add openssl-misc to the base sdk, and then run c_hash in the freedesktop-sdk-images cleanup-commands or something like that.

[alexlarsson]

Hmm, seems like c_rehash is not included in our builds because we don't have perl in PACKAGECONFIG. We should add that and see what else changes that causes.

[erikos]

Right, on Fedora the situation is completely different, no hashes there. When I run my test script it will look at /etc/crypto-policies/back-ends/openssl.config and then at /etc/pki/tls/cert.pem which is part of the ca-certificates (The Mozilla CA root certificate bundle) package.

[mariospr]

Hmm, seems like c_rehash is not included in our builds because we don't have perl in PACKAGECONFIG. We should add that and see what else changes that causes.

Where would such an addition go? I looked around the repository and that is not clear at all to me, probably because my understanding of the Yocto project is fairly limited / non-existent, but still...

[alexlarsson]

I think it goes in:

https://github.com/flatpak/freedesktop-sdk-base/blob/master/meta-freedesktop/conf/distro/freedesktop.conf

Probably as PACKAGECONFIG_append = " perl" or something.

[ebassi]

@alexlarsson Shouldn't perl be added to the freedesktop platform rdepends like Python is? I'm actually surprised stuff based on the fd.o runtime builds without Perl.

[ebassi]

So, fun stuff: this commit in oe-core may help fix the issue.

[alexlarsson]

Oh. Perl is in the sdk, just not set in the packageconfig

[mariospr]

JFTR, I'm devoting some time to this now, still in the phase of building the runtime locally, though, so that I can actually test whatever fix I'll be proposing.

Btw, I had some trouble so far because it was a bit difficult to figure out what the right way to build the runtime, and after some unsuccessful attempts (with different chroots, compilers and such) it seems that the best way to do it according to @alexlarsson is to build the runtime from inside the SDK itself, but again the steps to do that were not clear, so I'm sharing what I'm doing here in case I'm not yet in the right patch (still building as we speak):

1. Download the git repository for the freedesktop base SDK:

     git clone [email protected]:flatpak/freedesktop-sdk-base.git

2. Build the "builder app environment", that will leave the "app" in the app/ directory:

     make sandboxed  # this command will fail due to missing sh. That's ok

3. Enter the sandboxed environment, to build the runtime from inside of it:

     flatpak-builder --run app org.freedesktop.Builder.json bash

4. Once inside the environment, navigate to the sources directory and build:

     make # This will take quite a while...

@alexlarsson I think it would be nice if this steps (plus anything else that I might have missed) were documented somewhere, and I guess a README or a HACKING file checked in this repo would be a good place for it. It would certainly have saved me some time and headaches, what do you say?

[alexlarsson]

"make sandboxed" is supposed to be enough, and it works for me. How did it fail for you?

[mariospr]

flatpak-builder --force-clean  app org.freedesktop.Builder.json
Emptying app dir 'app'
Downloading sources
Starting build of org.freedesktop.Builder
Cache hit for libfuse, skipping build
Cache hit for ostree, skipping build
Cache hit for flatpak, skipping build
Cache hit for diffstat, skipping build
Cache hit for wget, skipping build
Cache hit for cleanup, skipping
Cache miss, checking out last cache hit
Finishing app
Error: Command '/usr/bin/sh' not found
Makefile:78: recipe for target 'sandboxed' failed

I don't have /usr/bin/sh on my host system (only /bin/sh), but I would not expect that to be an issue as the sandbox does have /usr/bin/sh. However, I did a quick test and it seems that just setting /bin/sh as the command in the JSON file fixes the problem:

$ make sandboxed
flatpak-builder --force-clean  app org.freedesktop.Builder.json
Emptying app dir 'app'
Downloading sources
Starting build of org.freedesktop.Builder
Cache hit for libfuse, skipping build
Cache hit for ostree, skipping build
Cache hit for flatpak, skipping build
Cache hit for diffstat, skipping build
Cache hit for wget, skipping build
Cache hit for cleanup, skipping
Cache hit for finish, skipping
Everything cached, checking out from cache
Pruning cache
flatpak build app make all
make: Nothing to be done for 'all'.

I'm a bit confused, maybe is a misunderstanding from my side, but I'd appreciate some clarification.

[alexlarsson]

Weird! I'll look into it.

[mariospr]

Thanks. I'm running flatpak and flatpak-builder 0.8.3 right from the Debian Testing repos in that machine, by the way. Let me know if there's any specific test / experiment you want me to help with.

[alexlarsson]

So, build-finish tries to look for the app, and files/sh in the runtime is a symlink to /bin/bash, which is resolved on the host when build-finish is looking for the result. I think the easiest fix is just to use /usr/bin/bash