Migrate CI to bionic #148

mikaelarguedas · 2018-04-16T18:06:34Z

start migrating Linux CI to bionic

connects to ros2/ros2#481

dirk-thomas · 2018-04-16T18:09:48Z

I would suggest to split the bullets into two groups:

what needs to happen to get this merged and use Bionic in the CI jobs
what should happen afterwards

mikaelarguedas · 2018-04-16T18:24:37Z

I would suggest to split the bullets into two groups:

what needs to happen to get this merged and use Bionic in the CI jobs
what should happen afterwards

Agreed, though it depends what we consider "enough" to merge this on CI.
If we want all our nightly to build on bionic without warnings we will need almost all of them to be addressed.

I see several stages:

Being able to run our regular ci_linux
Being able to run all the nightlies (including tb2 demo jobs)
Being able to run all ci jobs including the ones using opensplice.
Audit dependencies and previous hacks

I think we need at least the first 2 to be done before merging this on CI. Do you agree?

dirk-thomas · 2018-04-16T18:41:14Z

I think we need at least the first 2 to be done before merging this on CI. Do you agree?

Yes, except that I don't see the tb2 demo as a blocker - these jobs could continue to use Xenial until kobuki has been released.

nuclearsandwich · 2018-04-16T20:26:32Z


Edit: moving this question to https://github.com/ros2/ros2/issues/481

mikaelarguedas · 2018-04-19T00:30:17Z

Status update:
This branch with connected PRs gives green CI for both opensplice and connext including cross_vendor tests between the 2.

Fast-RTPS still segfaults on any operation involving setting parameters or security

mikaelarguedas · 2018-05-04T18:24:21Z

The todos have been moved to ros2/ros2#481

mikaelarguedas · 2018-05-05T01:21:17Z

All bionic fixup PR's have been merged

Linux job
Packaging amd64
Coverage:
Linux Debug
Linux Release

aarch jobs have xenial hardcoded, set of job using upstream arm64 bionic docker images:

ci aarch64:
packaging aarch64:

Newly unveiled issue: there's a new snprintf warning that shows up only when building with RelWithDebInfo (aka packaging job).
All the other jobs are yellow because of false positive CMake warnings.

mikaelarguedas · 2018-05-08T16:04:28Z

New round of packaging job including the warning fixups from ros2/system_tests#269 and ros2/rosidl#275

Packaging amd64
Packaging Windows:
Packaging MacOS:

mikaelarguedas · 2018-05-08T16:30:09Z

Squashed and rebased all commits.

Ready for review

Last round of CI to confirm all is well:

Linux
Linux-aarch64 (custom job)
macOS
Windows
Linux Debug
Linux Release
Coverage: (custom job)
Packaging aarch64: (custom job)
Packaging amd64: (custom job)

dirk-thomas

The patch looks good to me 👍

mikaelarguedas · 2018-05-09T00:32:03Z

Looking a bit more into it we were not using custom arm images but the images from that project: https://hub.docker.com/r/aarch64/ubuntu/

Now that the official docker library/ubuntu hosts armhf and arm64 images I kept my change as is to use the upstream docker images.
If the need to provide CI that can run with qemu comes up we move to use our custom images. In the meantime we'll keep upstream to avoid having custom "magic"

nuclearsandwich · 2018-05-09T12:42:37Z

Now that the official docker library/ubuntu hosts armhf and arm64 images I kept my change as is to use the upstream docker images.
If the need to provide CI that can run with qemu comes up we move to use our custom images. In the meantime we'll keep upstream to avoid having custom "magic"

Makes complete sense to me. If there's no qemu support to preserve then there's no reason not to go straight to upstream images.

nuclearsandwich

One non-blocking question inline. Everything else looks good. Thanks for all the time invested here! 🙇‍♂️ 🙇‍♂️ 🙇‍♂️

nuclearsandwich · 2018-05-09T12:44:06Z

linux_docker_resources/Dockerfile

-# We are building poco from source for now as we need at least 1.4.1p1 and xenial ships with 1.3.6p1 (https://github.com/ros2/poco_vendor/pull/10)
-# RUN apt-get update && apt-get install --no-install-recommends -y libpoco-dev libpocofoundation9v5 libpocofoundation9v5-dbg
+# Install build dependencies for class_loader.
+# TODO(mikaelarguedas) Uncomment this when we decide what to do about Poco debug builds


Are we cool to merge with this TODO still pending?

TL;DR: IMO yes, this doesnt change the current behavior ATM.

Poco doesn't ship debs with debug libraries anymore (Bionic or Brew) and the release libraries will be found even in Debug builds.
We should be able to just use system [poco on Ubuntu] (https://ci.ros2.org/job/ci_packaging_linux/91/) and MacOS and build in Debug on top without facing ros2/poco_vendor#11.
@sloretz is currently modifying his poco_vendor PR to build only in Release and not overriding the Poco CMake module, my hope is that this should work on windows and allow us to build poco, only on Windows and only in Release that should save use some CI time.

This reverts commit d4d82a7.

…melodic testing repo

mikaelarguedas · 2018-06-07T16:35:16Z

New round of CI using the last state of this branch and the newly generated jobs with the ROS1DISTRo selector:

Linux None:
Linux Debug:
Linux Release (using account for null-terminator character rosidl#275):
Linux None (with opensplice): ( test_publisher_subscriber__Builtins__rclcpp__rmw_opensplice_cpp__rmw_fastrtps_cpp flaky/failing on xenial as well)
Linux aarch64 None:
Linux aarch64 Debug:
Linux aarch64 Release (using account for null-terminator character rosidl#275):
Linux aarch64 None (with opensplice): (multiple tests failure, they seem consistent with the ones we see on the nightly on xenial)
Coverage:
Packaging amd64:
Packaging aarch64:

dhood

lgtm, not sure if you were looking for "fresh" eyes on it though

mikaelarguedas · 2018-06-07T18:22:02Z

not sure if you were looking for "fresh" eyes on it though

Not necessarily :) as we worked on different parts of the PR I think that's fine 👍

It feels a bit weird to have the CI_UBUNTU_DISTRO option now displayed on all jobs including the Macos/Windows ones but I think that's the price to pay for having a single template for all of them.

I'm planning on merging this as is in ~1hr.

If you happen to want to review and this is already merged, please feel free to comment here and I'll address comments post merge in a follow-up.

mikaelarguedas · 2018-06-07T22:08:33Z

And some jobs for the turtlebot demo:

tb2 amd64:
tb2 aarch64:

mikaelarguedas · 2018-06-08T12:16:33Z

Nightly turtlebot jobs failed due to an outstanding PR that wasn't merged. This is now fixed

tb2 amd64:
tb2 aarch64:

As the ros2 ci build farm has switched the ubuntu from Xenial to Bionic(ros2/ci#148), we have to change the url of the package url in order to get the right one. Fix NONE

mikaelarguedas added the in progress Actively being worked on (Kanban column) label Apr 16, 2018

mikaelarguedas mentioned this pull request Apr 16, 2018

Target bionic (meta) ros2/ros2#481

Closed

31 tasks

mikaelarguedas force-pushed the bionic branch 4 times, most recently from b1f4fae to 4f9d1a5 Compare April 18, 2018 01:48

mikaelarguedas mentioned this pull request Apr 18, 2018

Place RTI OpenSSL on the (LD_LIBRARY_)PATH on Linux ros2/system_tests#263

Merged

mikaelarguedas force-pushed the bionic branch from 39abfd0 to 50b8a4a Compare April 23, 2018 21:11

mikaelarguedas force-pushed the bionic branch from 50b8a4a to ba24bf5 Compare May 2, 2018 23:00

mikaelarguedas changed the title ~~[WIP] migrate to bionic~~ Migrate CI to bionic May 4, 2018

mikaelarguedas force-pushed the bionic branch from 8645c77 to c1ccd60 Compare May 8, 2018 01:57

mikaelarguedas force-pushed the bionic branch from c1ccd60 to ddf7e57 Compare May 8, 2018 16:21

mikaelarguedas added in review Waiting for review (Kanban column) and removed in progress Actively being worked on (Kanban column) labels May 8, 2018

dirk-thomas approved these changes May 8, 2018

View reviewed changes

mikaelarguedas force-pushed the bionic branch from fe11c58 to 07be532 Compare May 9, 2018 00:29

nuclearsandwich approved these changes May 9, 2018

View reviewed changes

mikaelarguedas mentioned this pull request May 9, 2018

Up poco version to 1.8.0.1 ros2/poco_vendor#14

Merged

mikaelarguedas force-pushed the bionic branch from d5b674b to c1a5b53 Compare May 16, 2018 02:55

dhood and others added 10 commits June 7, 2018 12:37

Define ubuntu_distro in more templates

41e224b

Use double quotes for sed command

f4ed614

Create job for ARM too

17437e4

Revert "Add back workarounds for ARM on xenial"

9419da3

This reverts commit d4d82a7.

Connext not supported on ARM

db0febd

Factor out ubuntu distro

358fb27

Highlight tb demo not supported on xenial

d8a11ca

Use elif

9c149f8

Remove duplicated deps (rebase mixup)

a415177

Label ARM jobs correctly

071d881

mikaelarguedas force-pushed the bionic branch from 167c586 to 071d881 Compare June 7, 2018 10:38

mikaelarguedas added 2 commits June 7, 2018 14:41

fix unexpected operator sh error

4126e8c

this should not be needed now that the new version of rosmsg hit the …

6079a37

…melodic testing repo

'on bionic' -> 'not on xenial' for poco debs

2592704

dhood approved these changes Jun 7, 2018

View reviewed changes

remove unnecessary blank line

e31eee5

mikaelarguedas merged commit b0bf985 into master Jun 7, 2018

wjwwood removed the in review Waiting for review (Kanban column) label Jun 7, 2018

mikaelarguedas deleted the bionic branch June 7, 2018 22:36

mikaelarguedas mentioned this pull request Jun 11, 2018

Build C++ application failed with last successful package on Linux #187

Closed

minggangw mentioned this pull request Jun 14, 2018

Change the ros2 package url RobotWebTools/rclnodejs#359

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate CI to bionic #148

Migrate CI to bionic #148

mikaelarguedas commented Apr 16, 2018 •

edited

Loading

dirk-thomas commented Apr 16, 2018

mikaelarguedas commented Apr 16, 2018

dirk-thomas commented Apr 16, 2018

nuclearsandwich commented Apr 16, 2018 •

edited

Loading

mikaelarguedas commented Apr 19, 2018

mikaelarguedas commented May 4, 2018

mikaelarguedas commented May 5, 2018

mikaelarguedas commented May 8, 2018

mikaelarguedas commented May 8, 2018 •

edited

Loading

dirk-thomas left a comment

mikaelarguedas commented May 9, 2018

nuclearsandwich commented May 9, 2018

nuclearsandwich left a comment

nuclearsandwich May 9, 2018

mikaelarguedas May 9, 2018

mikaelarguedas commented Jun 7, 2018 •

edited

Loading

dhood left a comment

mikaelarguedas commented Jun 7, 2018

mikaelarguedas commented Jun 7, 2018

mikaelarguedas commented Jun 8, 2018 •

edited

Loading

Migrate CI to bionic #148

Migrate CI to bionic #148

Conversation

mikaelarguedas commented Apr 16, 2018 • edited Loading

dirk-thomas commented Apr 16, 2018

mikaelarguedas commented Apr 16, 2018

dirk-thomas commented Apr 16, 2018

nuclearsandwich commented Apr 16, 2018 • edited Loading

mikaelarguedas commented Apr 19, 2018

mikaelarguedas commented May 4, 2018

mikaelarguedas commented May 5, 2018

mikaelarguedas commented May 8, 2018

mikaelarguedas commented May 8, 2018 • edited Loading

dirk-thomas left a comment

Choose a reason for hiding this comment

mikaelarguedas commented May 9, 2018

nuclearsandwich commented May 9, 2018

nuclearsandwich left a comment

Choose a reason for hiding this comment

nuclearsandwich May 9, 2018

Choose a reason for hiding this comment

mikaelarguedas May 9, 2018

Choose a reason for hiding this comment

mikaelarguedas commented Jun 7, 2018 • edited Loading

dhood left a comment

Choose a reason for hiding this comment

mikaelarguedas commented Jun 7, 2018

mikaelarguedas commented Jun 7, 2018

mikaelarguedas commented Jun 8, 2018 • edited Loading

mikaelarguedas commented Apr 16, 2018 •

edited

Loading

nuclearsandwich commented Apr 16, 2018 •

edited

Loading

mikaelarguedas commented May 8, 2018 •

edited

Loading

mikaelarguedas commented Jun 7, 2018 •

edited

Loading

mikaelarguedas commented Jun 8, 2018 •

edited

Loading