calico

Building Calicoctl

The instructions provided below specify the steps to build Calico for Docker version v1.1.1 on Linux on the IBM z Systems for following distributions:

RHEL (7.1, 7.2, 7.3)
SLES (12, 12 SP1, 12 SP2)
Ubuntu (16.04, 16.10)

General Notes:

When following the steps below please use a standard permission user unless otherwise specified.
A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Prerequisites

Go -- Instructions for building Go can be found here.
Docker -- Instructions for install Docker can be found here
etcd -- Instructions for building etcd can be found here

Building Calicoctl

1. Install the system dependencies

For RHEL (7.1, 7.2, 7.3)

   sudo yum install curl git wget tar gcc glibc-static.s390x make

For SLES (12, 12 SP1, 12 SP2)

   sudo zypper install curl git wget tar gcc glibc-static.s390x make

For Ubuntu (16.04, 16.10)

   sudo apt-get update
   sudo apt-get install git curl tar gcc glibc-static.s390x make

2. Build dependent 3rd part packages

2.1 Build `runit`

   cd /<source_root>/
   wget http://smarden.org/runit/runit-2.1.2.tar.gz
   tar xvf runit-2.1.2.tar.gz
   cd admin/runit-2.1.2/src
   make load
   cd ../

Update src/load to add -static option to make link statically

   #!/bin/sh
   # WARNING: This file was auto-generated. Do not edit!
   main="$1"; shift
   exec gcc -s -o "$main" "$main".o -static ${1+"$@"}

Then build and pack executable files

   package/compile
   tar cvf runit.tar command/
   gzip runit.tar

This runit.tar.gz is used in building calico/node docker images

2.2 Build `glibc`

   cd /<source_root>/
   git clone https://github.com/sgerrand/docker-glibc-builder
   cd docker-glibc-builder/

Update Dockerfile to use s390x/ubuntu base image:

@@ -1,4 +1,4 @@
-FROM ubuntu-debootstrap:14.04
+FROM s390x/ubuntu
 MAINTAINER Sasha Gerrand <[email protected]>

Build an docker image and then use it to build glibc

   docker build -t glibc-builder .
   docker run --rm -e STDOUT=1 glibc-builder 2.24 /usr/glibc-compat > glibc-bin-2.24.tar.gz

glibc package is used in building docker images.

3. Build `calicoctl` and other components of `calico`

3.1 Build `go-build`

This builds a docker image calico/go-build that is used to build other components by containerized build

   cd /<source_root>/
   mkdir -p calico/src/github.com/projectcalico
   cd calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/go-build
   cd go-build
   git checkout v0.5

Modify Dockerfile for s390x

@@ -1,4 +1,4 @@
-FROM golang:1.8.1-alpine
+FROM tmh1999/alpine-s390x
 MAINTAINER Tom Denham <[email protected]>

 # Install su-exec for use in the entrypoint.sh (so processes run as the right user)
@@ -9,22 +9,25 @@ MAINTAINER Tom Denham <[email protected]>
 # Install wget for fetching glibc
 # Install make for building things
 # Install util-linux for column command (used for output formatting).
+RUN apk update
 RUN apk add --no-cache su-exec curl bash git mercurial make wget util-linux

+RUN apk add go
 # Install glibc
 RUN wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub
-RUN wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk
-RUN apk add glibc-2.23-r3.apk
+COPY pkg /pkg/
+RUN  cd / && \
+     tar xvf pkg/glibc-bin-2.24.tar.gz && \
+     mv /pkg/ld.so.conf /usr/glibc-compat/etc/   &&  \
+     /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc/usr/lib && \
+     rm -rf /pkg

 # Disable cgo so that binaries we build will be fully static.
 ENV CGO_ENABLED=0
+ENV GOROOT=/usr/lib/go
+ENV GOPATH=/go
+ENV PATH=$PATH:$GOROOT/bin:$GOPATH/bin

-# Apply patches to Go runtime and recompile.
-# See https://github.com/golang/go/issues/5838 for defails of vfork patch.
-COPY patches/use-clone-vfork-b7edfba429d982e3e065d637334bcc63ad49f8f9.patch \
-     /tmp/use-clone-vfork-b7edfba429d982e3e065d637334bcc63ad49f8f9.patch
-RUN cd /usr/local/go && \
-    patch -p 1 < /tmp/use-clone-vfork-b7edfba429d982e3e065d637334bcc63ad49f8f9.patch
 RUN go install -v -a syscall

   mkdir pkg
   cp /<source_root>/docker-glibc-builder/glibc-bin-2.24.tar.gz pkg/

Create a file pkg/ld.so.conf with follow content:

     /usr/local/lib
     /usr/glibc-compat/lib
     /usr/lib
     /lib

Then start to build calico/go-build image

   make

3.2 Build `calicoctl`

This component builds interactive command interface calicoctl and docker image calico/node

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/calicoctl
   cd calicoctl
   git checkout v1.1.1

Modify Makefile.calicoctl for s390x:

@@ -6,7 +6,7 @@
 ###############################################################################
 # Determine which OS / ARCH.
 OS := $(shell uname -s | tr A-Z a-z)
-ARCH := amd64
+ARCH := s390x
 ###############################################################################
 # Subcomponent versions:
 GO_BUILD_VER:=latest
@@ -75,11 +75,11 @@ binary: $(CALICOCTL_FILES) vendor
        GOOS=$(OS) GOARCH=$(ARCH) CGO_ENABLED=0 go build -v $$INSTALL_FLAG -o dist/calicoctl-$(OS)-$(ARCH) $(LDFLAGS) "./calicoctl/calicoctl.go"

 dist/calicoctl: $(CALICOCTL_FILES) vendor
-       $(MAKE) dist/calicoctl-linux-amd64
-       mv dist/calicoctl-linux-amd64 dist/calicoctl
+       $(MAKE) dist/calicoctl-linux-s390x
+       mv dist/calicoctl-linux-s390x dist/calicoctl

-dist/calicoctl-linux-amd64: $(CALICOCTL_FILES) vendor
-       $(MAKE) OS=linux ARCH=amd64 binary-containerized
+dist/calicoctl-linux-s390x: $(CALICOCTL_FILES) vendor
+       $(MAKE) OS=linux ARCH=s390x binary-containerized

Then make dist/calicoctl to build, the executable file is in dist/calicoctl Because calico/node needs the executable files built from other components, let's build other components first, then come back.

   make  dist/calicoctl
   mkdir -p /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/    ## store executable files built from other components

3.3 Build `bird`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/bird
   cd bird
   git checkout v0.3.0

Modify Dockerfile for s390x

@@ -1,4 +1,4 @@
-FROM alpine:latest
+FROM tmh1999/alpine-s390x
 MAINTAINER Tom Denham <[email protected]>

Then run build.sh to build 3 executable files (dist/) that are required by building calico/node image.

   ./build.sh
   cp dist/* /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

3.4 Build `confd`

   cd /<source_root>
   git clone https://github.com/kelseyhightower/confd.git calico/src/github.com/kelseyhightower/confd
   cd calico/src/github.com/kelseyhightower/confd
   export GOROOT=<go_instll_dir>
   export PATH=$GOROOT/bin:$PATH
   go build -ldflags "-v -linkmode=external '-extldflags=-v -static'"
   cp confd /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

This built executable file confd required by calico/node.

3.4 Build `libnetwork-plugin`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/libnetwork-plugin
   cd libnetwork-plugin
   git checkout out v1.1.0

Modify Dockerfile to change FROM alpine to FROM tmh1999/alpine-s390x

   make calico/libnetwork-plugin
   cp dist/libnetwork-plugin /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

This builds executable file dist/libnetwork-plugin required by calico/node and docker image calico/libnetwork-plugin

3.5 Build `felix`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/felix
   cd felix
   git checkout 2.1.1

Modify Makefile to use calico/go-build:latest

@@ -80,7 +80,7 @@ help:
 all: deb rpm calico/felix
 test: ut

-GO_BUILD_CONTAINER?=calico/go-build:v0.4
+GO_BUILD_CONTAINER?=calico/go-build

To build felix it needs felixbackend.pb.go that is generated by a docker image calico/protoc. Let's first built this images.

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/tigera/docker-protobuf
   cd  docker-protobuf

Modify Dockerfile as follows:

@@ -1,4 +1,4 @@
-FROM golang:1.6
+FROM s390x/ubuntu

 MAINTAINER Shaun Crampton <[email protected]>

@@ -8,6 +8,10 @@ WORKDIR /src
 # We want v3.0.0 but the tag for v3.0.0 has broken download links.  Pin to
 # master for now :-(
 ENV PROTOBUF_TAG master
+RUN apt-get -y  update \
+     && apt-get -y  install golang git curl
+ENV GOROOT=/usr/lib/go
+ENV GOPATH=/usr/local

Then build docker images calico/protoc

   docker build -t calico/protoc .

Let's come back to felix and build

   cd /<source_root>/calico/src/github.com/projectcalico/felix
   make bin/calico-felix
   cp bin/calico-felix /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

Built calico-felix is in bin/

3.6 Build `calico-bgp-daemon`

   cd /<source_root>/calico/src/github.com/projectcalico
   git clone https://github.com/projectcalico/calico-bgp-daemon
   cd calico-bgp-daemon
   git checkout v0.2.0

Modify Dockerfile to change to FROM tmh1999/alpine-s390x

@@ -1,4 +1,4 @@
-FROM alpine:3.4
+FROM tmh1999/alpine-s390x

   make build-containerized
   cp dist/* /<source_root>/calico/src/github.com/projectcalico/calicoctl/calico_node/filesystem/bin/

Two executable files calico-bgp-daemon and gobgp are in dist/

3.7 Build image `calico/node`

Now let's come beck to calicoctl and build image calico/node

   cd  /<source_root>/calico/src/github.com/projectcalico/calicoctl
   cp -r /<source_root>/calico/src/github.com/projectcalico/go-build/pkg  calico_node/
   cp /<source_root>/admin/runit-2.1.2/runit.tar.gz calico_node/pkg/

Modify Makefile.calico-node:

@@ -123,7 +123,7 @@ $(NODE_CONTAINER_BIN_DIR)/allocate-ipip-addr: dist/allocate-ipip-addr

 ## Build startup.go
 startup:
-       GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -v -i -o dist/startup $(LDFLAGS) "./calico_node/startup/startup.go"
+       GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -v -i -o dist/startup $(LDFLAGS) "./calico_node/startup/startup.go"

 dist/startup: $(STARTUP_FILES) vendor
        mkdir -p dist
@@ -139,7 +139,7 @@ dist/startup: $(STARTUP_FILES) vendor

 ## Build allocate_ipip_addr.go
 allocate-ipip-addr:
-       GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -v -i -o dist/allocate-ipip-addr $(LDFLAGS) "./calico_node/allocateipip/allocate_ipip_addr.go"
+       GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -v -i -o dist/allocate-ipip-addr $(LDFLAGS) "./calico_node/allocateipip/allocate_ipip_addr.go"

 dist/allocate-ipip-addr: $(ALLOCATE_IPIP_FILES) vendor
        mkdir -p dist
@@ -198,8 +198,8 @@ certs/.certificates.created:
        touch certs/.certificates.created

 busybox.tar:
-       docker pull busybox:latest
-       docker save --output busybox.tar busybox:latest
+       docker pull s390x/busybox:latest
+       docker save --output busybox.tar s390x/busybox:latest

and modify docker file calico_node/Dockerfile

@@ -11,25 +11,24 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-FROM alpine:3.4
+FROM tmh1999/alpine-s390x
 MAINTAINER Tom Denham <[email protected]>

 # Set the minimum Docker API version required for libnetwork.
 ENV DOCKER_API_VERSION 1.21

 # Download and install glibc for use by non-static binaries that require it.
+COPY pkg /pkg/
+RUN apk update
 RUN apk --no-cache add wget ca-certificates libgcc && \
     wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub && \
-    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && \
-    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-bin-2.23-r3.apk && \
-    apk add glibc-2.23-r3.apk glibc-bin-2.23-r3.apk && \
-    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc/usr/lib && \
-    apk del wget && \
-    rm -f glibc-2.23-r3.apk glibc-bin-2.23-r3.apk
+    tar xvf pkg/glibc-bin-2.24.tar.gz && \
+    mv pkg/ld.so.conf /usr/glibc-compat/etc/   &&  \
+    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc/usr/lib

 # Install runit from the community repository, as its not yet available in global
-RUN apk add --no-cache --repository "http://alpine.gliderlabs.com/alpine/edge/community" runit
-
+RUN  tar xvf pkg/runit.tar.gz  && \
+     cp command/* /sbin
 # Install remaining runtime deps required for felix from the global repository
 RUN apk add --no-cache ip6tables ipset iputils iproute2 conntrack-tools

Modify calico_node/filesystem/etc/service/available/confd/run

@@ -3,6 +3,6 @@ exec 2>&1

-exec confd -confdir=/etc/calico/confd -interval=5 -watch -no-discover --log-level=debug \
+exec confd -confdir=/etc/calico/confd -interval=5 -watch  --log-level=debug \

Start to build calico/node:

make calico/node
docker tag calico/node quay.io/calico/node

4. Build Kubernetes Support

4.1. Install Calico CNI plugins

mkdir -p /opt/cni/bin

Build CNI plugins calico and calico-ipam

cd /<source_root>/calico/src/github.com/projectcalico/
git clone https://github.com/projectcalico/cni-plugin.git
cd cni-plugin
git checkout v1.6.2
make dist/calico dist/calico-ipam
cp dist/* /opt/cni/bin

Build the standard CNI loopback plugin

cd /<source_root>/calico/src/github.com/projectcalico/
git clone https://github.com/containernetworking/cni.git
cd cni
git checkout v0.5.2
./build.sh
cp bin/loopback /opt/cni/bin

4.2. Install the Calico network policy controller

Build calico/kube-policy-controller image

cd /<source_root>/calico/src/github.com/projectcalico/
git clone https://github.com/projectcalico/k8s-policy.git
cd k8s-policy
git checkout v0.5.4

Make changes to Dockerfile

@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

-FROM alpine:3.4
+FROM tmh1999/alpine-s390x

ADD *.py /code/
ADD handlers /code/handlers
@@ -22,8 +22,8 @@ RUN /build.sh

# Symlinks needed to workaround Alpine/Pyinstaller incompatibilties
# https://github.com/gliderlabs/docker-alpine/issues/48
-RUN ln -s /lib/libc.musl-x86_64.so.1 ldd
+RUN ln -s /lib/libc.musl-s390x.so.1 ldd
RUN ln -s /lib /lib64
-RUN ln -s /lib/ld-musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
+RUN ln -s /lib/ld-musl-s390x.so.1 /lib64/ld-linux-s390x.so.2

-ENTRYPOINT ["/dist/controller"]
+CMD python /code/controller.py

Make changes to build.sh

@@ -3,18 +3,10 @@ set -e
set -x

# Install the system packages needed for building the PyInstaller based binary
-apk -U add --virtual temp python-dev py-pip alpine-sdk python py-setuptools openssl-dev libffi-dev
+apk -U add --virtual temp python python-dev py-pip alpine-sdk python py-setuptools openssl-dev libffi-dev

# Install python dependencies
pip install --upgrade pip
pip install -r https://raw.githubusercontent.com/projectcalico/libcalico/master/build-requirements-frozen.txt
pip install git+https://github.com/projectcalico/libcalico.git
pip install simplejson
-
-# Produce a binary - outputs to /dist/controller
-pyinstaller /code/controller.py -ayF
-
-# Cleanup everything that was installed now that we have a self contained binary
-apk del temp && rm -rf /var/cache/apk/*
-rm -rf /usr/lib/python2.7
-rm -rf /build

Then

make docker-image
docker tag calico/kube-policy-controller quay.io/calico/kube-policy-controller:v0.5.4

4.3 Installing the Calico network policy controller

The calico/kube-policy-controller implements the Kubernetes NetworkPolicy API by watching the Kubernetes API for Pod, Namespace, and NetworkPolicy events and configuring Calico in response. It runs as a single pod managed by a Deployment. To install the policy controller:

Download the policy controller manifest.
Modify <ETCD_ENDPOINTS> to point to your etcd cluster.
Install it using kubectl.

$ kubectl create -f policy-controller.yaml

After a few moments, you should see the policy controller enter Running state:

$ kubectl get pods --namespace=kube-system
NAME                              READY                    STATUS              RESTARTS             AGE
calico-policy-controller          1/1                      Running             0                    1m

5. Configure Kubernetes

5.1 The Kubelet needs to be configured to use the Calico network plugin when starting pods.

The kubelet can be configured to use Calico by starting it with the following options

    --network-plugin=cni
    --cni-conf-dir=/etc/cni/net.d
    --cni-bin-dir=/opt/cni/bin

For Kubernetes versions prior to v1.4.0, the cni-conf-dir and cni-bin-dir options are not supported. Use --network-plugin-dir=/etc/cni/net.d instead.

5.2. Configuring the Kube-Proxy

In order to use Calico policy with Kubernetes, the kube-proxy component must be configured to leave the source address of service bound traffic intact. This feature is first officially supported in Kubernetes v1.1.0 and is the default mode starting in Kubernetes v1.2.0.

We highly recommend using the latest stable Kubernetes release, but if you’re using an older release there are two ways to enable this behavior.

Option 1: Start the kube-proxy with the --proxy-mode=iptables option.
Option 2: Annotate the Kubernetes Node API object with net.experimental.kubernetes.io/proxy-mode set to iptables.

6. Testing

6.1. Build image `nginx`

   cd  /<source_root>/calico/src/github.com/projectcalico/
   git clone https://github.com/nginxinc/docker-nginx
   cd docker-nginx
   git clone 1.11.10
   cd stable/alpine

Modify Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.4
+FROM tmh1999/alpine-s390x

Then build image

   docker build -t nginx .

6.2. Build some images:

docker pull s390x/busybox

Also build gcr.io/google_containers/pause-s390x:3.0 following https://github.com/linux-on-ibm-z/docs/wiki/Building-Pause

Start with a local Kubernetes instance,

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
git checkout v1.6.0

Configure kubelet by modifying hack/local-up-cluster.sh,

@@ -31,9 +31,9 @@ KUBELET_AUTHENTICATION_WEBHOOK=${KUBELET_AUTHENTICATION_WEBHOOK:-""}
 POD_MANIFEST_PATH=${POD_MANIFEST_PATH:-"/var/run/kubernetes/static-pods"}
 KUBELET_FLAGS=${KUBELET_FLAGS:-""}
 # Name of the network plugin, eg: "kubenet"
-NET_PLUGIN=${NET_PLUGIN:-""}
+NET_PLUGIN=${NET_PLUGIN:-"cni"}
 # Place the binaries required by NET_PLUGIN in this directory, eg: "/home/kubernetes/bin".
-NET_PLUGIN_DIR=${NET_PLUGIN_DIR:-""}
+NET_PLUGIN_DIR=${NET_PLUGIN_DIR:-"/etc/cni/net.d"}
 SERVICE_CLUSTER_IP_RANGE=${SERVICE_CLUSTER_IP_RANGE:-10.0.0.0/24}
 FIRST_SERVICE_CLUSTER_IP=${FIRST_SERVICE_CLUSTER_IP:-10.0.0.1}
 # if enabled, must set CGROUP_ROOT

ETCD_ENDPOINTS=http://127.0.0.1:2379 calicoctl node run
kubectl create -f policy-controller.yaml
kubectl get pods --namespace=kube-system
kubectl create ns policy-demo
kubectl run --namespace=policy-demo nginx --replicas=2 --image=nginx
kubectl expose --namespace=policy-demo deployment nginx --port=80
kubectl run --namespace=policy-demo access --rm -ti --image s390x/busybox /bin/sh

References:

https://github.com/projectcalico

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

calico

Building Calicoctl

Prerequisites

Building Calicoctl

1. Install the system dependencies

2. Build dependent 3rd part packages

2.1 Build `runit`

2.2 Build `glibc`

3. Build `calicoctl` and other components of `calico`

3.1 Build `go-build`

3.2 Build `calicoctl`

3.3 Build `bird`

3.4 Build `confd`

3.4 Build `libnetwork-plugin`

3.5 Build `felix`

3.6 Build `calico-bgp-daemon`

3.7 Build image `calico/node`

4. Build Kubernetes Support

4.1. Install Calico CNI plugins

4.2. Install the Calico network policy controller

4.3 Installing the Calico network policy controller

5. Configure Kubernetes

5.1 The Kubelet needs to be configured to use the Calico network plugin when starting pods.

5.2. Configuring the Kube-Proxy

6. Testing

6.1. Build image `nginx`

6.2. Build some images:

References:

Clone this wiki locally

calico

Building Calicoctl

Prerequisites

Building Calicoctl

1. Install the system dependencies

2. Build dependent 3rd part packages

2.1 Build runit

2.2 Build glibc

3. Build calicoctl and other components of calico

3.1 Build go-build

3.2 Build calicoctl

3.3 Build bird

3.4 Build confd

3.4 Build libnetwork-plugin

3.5 Build felix

3.6 Build calico-bgp-daemon

3.7 Build image calico/node

4. Build Kubernetes Support

4.1. Install Calico CNI plugins

4.2. Install the Calico network policy controller

4.3 Installing the Calico network policy controller

5. Configure Kubernetes

5.1 The Kubelet needs to be configured to use the Calico network plugin when starting pods.

5.2. Configuring the Kube-Proxy

6. Testing

6.1. Build image nginx

6.2. Build some images:

References:

Clone this wiki locally

2.1 Build `runit`

2.2 Build `glibc`

3. Build `calicoctl` and other components of `calico`

3.1 Build `go-build`

3.2 Build `calicoctl`

3.3 Build `bird`

3.4 Build `confd`

3.4 Build `libnetwork-plugin`

3.5 Build `felix`

3.6 Build `calico-bgp-daemon`

3.7 Build image `calico/node`

6.1. Build image `nginx`