Add support for devuan, ccache, eatmydata, git format

[nextime]

As we in devuan build packages intensively, performances are needed, so, with this pull request we add:

• support for building with CCACHE ( false by default, usable concurrently with eatmydata )
• support for building with EATMYDATA (false by default, usable concurrently with ccache)

Also, we add a jenkins-debian-glue-buildenv-devuan optional package that install some files specific for devuan packages

Last but not least, we introduce a way to limit the size of the git clone for git format packages, this is very usefull for packages with a huge commit history.

Enjoy!

[mika]

Thanks for this PR! It includes great features which I'd absolutely like to have in j-d-g available out of the box!

There are some minor things I'd like to change though:

• simplify code (for example there's just "export USE_EATMYDATA" needed instead of the variable check, the LD_PRELOAD handling could be simplified, some trailing white space cleanups,...)
• I'm not sure whether we should really create derivatives specific Debian packages like jenkins-debian-glue-buildenv-devuan in the upstream repos, because 1) once j-d-g makes it into official Debian this will be harder to maintain and 2) if you'd need changes to your configs you then depend on upstream instead of just changing your own package internally, but I've absolutely no objections to include such example configurations in upstream git (if you'd want to use the same config as present in our upstream git you could still create this trivial Debian package by just copying the files from the examples directory to the according place and be done) - that's actually a lovely idea that should be extended further!

I just need some more time to think about it and merge it with configs from some of my customer setups, so please give me some more time until I can finalize + merge this PR.

PS: Is there any chance that someone of the devuan team is at debconf15? I'll give a talk about j-d-g (https://summit.debconf.org/debconf15/meeting/286/continuous-delivery-of-debian-packages/) and I'm waiting for acceptance of a j-d-g BoF session as well where contributors of j-d-g will meet and I'd love to have someone of devuan attend the session as well!
PPS: Do you have a URL for your devuan j-d-g setup I could/should add to http://jenkins-debian-glue.org/contribute/?

Thanks! 👍

[nextime]

Hello Mika,
well, for the minor changes, feel free to do them. For the LD_PRELOAD take care that as an eventual pbuilderrc can also contain LD_PRELOAD settings, it's anyway mandatory to add it without removing eventual other addition to the same env var.

For the "devuan" specific package, it isn't our specific config, we use a more complex one, it's a standard config for who want to build packages for devuan, assuming even a debian or ubuntu use can do that without installing devuan. For debian and ubuntu there are some hardcoded checks if i remember right, for devuan, as it set some things like the needs of a different keyring, i preferred to let it as an optional external package, i think it can be accepted even in case of addition on debian.

Anyway, feel free to remove the changes for that ( basically 2 lines in debian/rules, the "devuan" directory, the .install file and the package declaration in the control file ) and, in this case, i will split it to a separate package distributed by our repository.

for debconf15 i don't know if anyone from devuan will go, i will be at more than 10k km from germany, but in case i will let you know

For our devuan jdg setup, we maintain our package here: https://git.devuan.org/devuan-infrastructure/jenkins-debian-glue and our build cluster use it at https://ci.devuan.org

Thanks!
ciao.

[zeha]

can use LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}libeatmydata.so" instead of the if thing

[mika]

JFTR, I didn't forget about that but need further time to work on this since e.g. support on older distributions needs to be tested as well. I'll try to get towards it ASAP.

[mika]

The eatmydata + ccache support is implemented as of 220f87e

Looking at the rest now.

[mika]

I just took care of the git source format thingy in: 6658d9a

Some notes about it:

• I think we shouldn't modify the behavior of "--source-option=--git-depth=" by default, so I'd suggest to use what's set int debian/source/options. If GIT_DEPTH is set then use that to overwrite the behaviour of debian/source/options.
• The "$GIT_REF" isn't initialized in our code, so I rewrote that to support a similar behavior as for the depth use case. I think it makes more sense to accept whatever is in $GIT_REF and use that instead of defaulting to $BRANCH.
• I added support for SKIP_GIT_SOURCE_OPTIONS to suppport skipping the function if that's needed for whatever reason
• The logic of the opt_ref (see https://github.com/devuan/jenkins-debian-glue/commit/fd6d687f4f1344095643aafda13a1d87bfc8c496#diff-48c97c5d0f3f1abb799891dab1c18139R369) is the other way around AFAICT, I changed that to match what I think is the right thing to do. Can you please clarify that?

What do you think of my branch, does that implement what you'd need?

[mika]

Hi @nextime, did you notice my recent work WRT to your PR? Any chance you could take a look at it?

[nextime]

hello @mika sorry for the late response, i was in a long working trip around the world and really busy, i'll look at it tonight (italian timezone) but seems good at first look, thanks!

[mika]

Hej @nextime, no worries at all, take your time, just wanted to make sure it doesn't get lost/forgotten. :) Looking forward to your feedback and greetings towards Italian from Austria. :)

[nextime]

@mika seems good for me:

• GIT_DEPTH: make sense. We was setting it with a default limited depth cause i experienced issues in qemu-user when building for alien architectures like arm* and sparc*, some segfaults related on threading issues in qemu not so easy to solve make it segfault when it has a long git history apparently. Anyway, this can be managed by setting GIT_DEPTH by default in the build job in jenkins if needed, and make sense to do things the way you do assuming the issues in qemu will be solved soon or later
• defaulting to $BRANCH is related on how we manage packages in devuan, as we use branches with an internal standard structure ( suites/, like suites/jessie for jessie ). We can manage that in the jenkins build job too, so, it's ok this way
• SKIP_GIT_SOURCE_OPTIONS make sense
• opt_ref: related on how we use to structure our git repos, it's ok the other way around as you implemented it, we can manage that

Thanks for your time and j-d-g!

[guillemj]

I think that the vendor specific config/behavior could be integrated upstream by using the dpkg-vendor logic. So by default it would use the default vendor, which could be overridden via DEB_VENDOR as usual. This would allow for a somewhat vendor-neutral upstream, and something easy to maintain even in Debian or on downstreams.

Just putting this out there. :)

[sylvestre]

@mika maybe close this issue ? :)