ONS Tutorial updates

README.md

@@ -21,7 +21,7 @@ Playbooks are located in the `playbooks/` directory.
 | `virt-host-setup.yml`            | `./inventory/virthost/`               | Provision a virtual machine host                                   |
 | `kube-install.yml`               | `./inventory/vms.local.generated`     | Install and configure a k8s cluster                                |
 | `kube-teardown.yml`              | `./inventory/vms.local.generated`     | Runs `kubeadm reset` on all nodes to tear down k8s                 |
-| `vm-teardown.yml`                | `./inventory/virthost/`               | Destroys VMs on the virtual machine host                           |
+| `vm-teardown.yml`                | `./inventory/virthost/`               | Destroys & removes VMs on the virtual machine host                           |
 | `multus-cni.yml`                 | `./inventory/vms.local.generated`     | Compiles [multus-cni](https://github.com/Intel-Corp/multus-cni)    |
 | `gluster-install.yml`            | `./inventory/vms.local.generated`     | Install a GlusterFS cluster across VMs (requires vm-attach-disk)   |
 | `fedora-python-bootstrapper.yml` | `./inventory/vms.local.generated`     | Bootstrapping Python dependencies on cloud images                  |
@@ -195,6 +195,24 @@ kube-node-3   Ready     <none>    9m        v1.8.3
 
 Everything should be marked as ready. If so, you're good to go!
 
+## Creating a bootstrapped image
+
+Should you need to spin up multiple clusters or otherwise spin up a bunch of VMs for a cluster, it may behoove you to "bootstrap" your VM images so that you don't have to download the dependencies many times. You can create a sort of golden image to use by using the `./playbooks/create-bootstrapped-image.yml` playbook.
+
+You can run it for example like so:
+
+```
+$ ansible-playbook -i inventory/virthost.inventory \
+  -e "@./inventory/examples/image-bootstrap/extravars.yml" \
+  playbooks/create-bootstrapped-image.yml
+```
+
+This will result in an image being created @ `/home/images/bootstrapped.qcow2` (by default, this can be altered otherwise). You can then specify this image to use when creating a cluster.
+
+For example...
+
+
+
 # About
 
 Initially inspired by:

contrib/multi-cluster/README.md

@@ -0,0 +1,207 @@
+# Multi-cluster creator!
+
+A series of scripts designed to spin up multiple clusters at once. Originally designed for a tutorial / classroom setup where you're spinning up a cluster for each attendee to use.
+
+These scripts are designed to be run from the root directory of this clone.
+
+## Prerequisites
+
+* A physical machine with CentOS 7
+    - We call this machine "the virthost", it hosts your virtual machines
+* On your client machine...
+    - A clone of this repo
+    - SSH keys to that physical machine that allow you to login as root (without a passowrd is convenient.)
+    - Ansible. Tested with version 2.5.7
+
+## General process
+
+In overview, what we're going to do is:
+
+* Setup the virtualization host ("virthost")
+* Create a "bootstrap image" (a golden image from which VMs are created)
+* Run the multi-cluster spin-up scripts.
+
+## Downloading Ansible Galaxy roles
+
+If this is your first time cloning this repository, go ahead and initialize the requirements for Ansible Galaxy with:
+
+```
+ansible-galaxy install -r requirements.yml
+```
+
+## Creating an inventory for your virthost
+
+We call the box we run the virtual machines on "the virthost" generally. Let's create an inventory for it.
+
+**NOTE**: You'll need to update the IP address to the proper one for your virthost. You can also change the name from `droctagon2` to any name you wish.
+
+```
+export VIRTHOST_IP=192.168.1.55
+cat << EOF > ./inventory/virthost.inventory
+droctagon2 ansible_host=$VIRTHOST_IP ansible_ssh_user=root
+
+[virthost]
+droctagon2
+EOF
+```
+
+## Setting up the virt-host
+
+You'll first need to run a playbook to setup the virt host. This has the  side-effect of also spinning up some VMs -- which we don't need yet. So you'll do this first, and then we'll use those VMs to test we can access them and then we'll remove those VMs.
+
+```
+ansible-playbook -i inventory/virthost.inventory -e 'ssh_proxy_enabled=true' playbooks/virthost-setup.yml
+```
+
+This will result in a locally generated inventory with the VMs that were spun up:
+
+```
+cat inventory/vms.local.generated
+```
+
+Now we can use information from that in order to access those machines -- a key has been created for us too in `/home/{your user name}/.ssh/{virthost name}/id_vm_rsa`
+
+So for example I can SSH to a VM using:
+
+```
+ssh -i /home/doug/.ssh/droctagon2/id_vm_rsa -o ProxyCommand="ssh -W %h:%p [email protected]" [email protected]
+```
+
+Where:
+
+* `/home/doug/.ssh/droctagon2/id_vm_rsa` is the name of the key at the bottom of the `./inventory/vms.local.generated`
+* `192.168.1.55` is the IP address of my virtualization host
+* `192.168.122.68` is the IP address of the VM from the top section of the `./inventory/vms.local.generated`
+
+Now you can remove those VMs (and I recommend you do) with:
+
+```
+ansible-playbook -i inventory/virthost.inventory playbooks/vm-teardown.yml
+```
+
+## OPTION: Download the bootstrap image
+
+Go ahead and place this image on your virtualization host, that is, SSH to the virt host
+
+```
+curl http://speedmodeling.org/kube/bootstrapped.qcow2 -o /home/images/bootstrapped.qcow2
+```
+
+## Creating the bootstrap image.
+
+You can skip this if you downloaded an existing one.
+
+You can run it for example like so:
+
+```
+$ ansible-playbook -i inventory/virthost.inventory \
+  -e "@./inventory/examples/image-bootstrap/extravars.yml" \
+  playbooks/create-bootstrapped-image.yml
+```
+
+
+## Run the multi-cluster spin up all at once...
+
+These scripts expect your virthost inventory to live @ `./inventory/virthost.inventory`.
+
+It might be convenient to set the number of clusters like so:
+
+```
+export CLUSTERS=3
+```
+
+"Run it all" with the all.sh script which runs all the individual plays.
+
+```
+./contrib/multi-cluster/all.sh $CLUSTERS
+```
+
+After you've set it up, you'll find the information to log into the clusters in your inventory directory...
+
+```
+cat inventory/multi-cluster/cluster-1.inventory
+```
+
+Replace `1` with whatever cluster number. So if you had `CLUSTERS=3` you should have `cluster-1.inventory` through `cluster-3.inventory`
+
+You can then use the IP addresses as listed in these inventories to SSH to each of the hosts. The same SSH key as used earlier is still the key you'll use, and should be listed in each of the inventories.
+
+When this completes, you should now have a number of clusters. Let's take a look at the first cluster. 
+
+```
+ssh -i /home/doug/.ssh/droctagon2/id_vm_rsa -o ProxyCommand="ssh -W %h:%p [email protected]" centos@$(cat inventory/multi-cluster/cluster-1.inventory | grep kube-master-1 | head -n1 | cut -d= -f2)
+```
+
+Replace the SSH key with your own, as well as the `[email protected]` with the IP address of your virthost.
+
+Now, after SSHing to that machine -- you should be able to see:
+
+```
+[centos@kube-master-1 ~]$ kubectl get nodes
+NAME            STATUS     ROLES     AGE       VERSION
+kube-master-1   NotReady   master    1h        v1.11.2
+kube-node-2     NotReady   <none>    1h        v1.11.2
+kube-node-3     NotReady   <none>    1h        v1.11.2
+```
+
+Note that the `NotReady` state is expected, as this cluster is up, however, it is intentionally not ready because the attendees are expected to install the CNI plugins.
+
+You can then tear down those VMs if you please:
+
+```
+./contrib/multi-cluster/multi-teardown.sh $CLUSTERS
+```
+
+
+## Giving access via SSH to people
+
+Firstly, you must set the `CLUSTERS` environment variable for this to work. Requires a Perl install on the machine you're running it from.
+
+```
+export CLUSTERS=3
+./contrib/multi-cluster/tmate.pl
+```
+
+This will create 2 tmate sessions for each master machine. (One for a backup in case the user types 'exit', which will ruin that session)
+
+The output will give you a JSON structure, you're looking for the line that looks like:
+
+```
+     "link": "https://markdownshare.com/view/ea8571af-8c97-469a-935b-470f33476214",
+```
+
+This will be a link to the posted markdown showing the tmate SSH urls.
+
+## Adding additional interfaces
+
+In case you have to do it manually...
+
+```
+virsh list --all | grep node | awk '{print $2}' | xargs -L1 -i virsh attach-interface --domain {} --type bridge --model virtio --source virbr0 --config --live
+```
+
+## Multi-cluster a la carte -- step-by-step if you please.
+
+Run it with the number of clusters you're going to create.
+
+```
+./contrib/multi-cluster/extravars-creator.sh $CLUSTERS
+```
+
+Then you can run the multi spinup...
+
+```
+./contrib/multi-cluster/multi-spinup.sh $CLUSTERS
+```
+
+Bring up the kube clusters with a multi init...
+
+```
+./contrib/multi-cluster/multi-init.sh $CLUSTERS
+```
+
+And tear 'em down with the multi-teardown...
+
+```
+./contrib/multi-cluster/multi-teardown.sh $CLUSTERS
+```

contrib/multi-cluster/all.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+./contrib/multi-cluster/extravars-creator.sh $1
+./contrib/multi-cluster/multi-spinup.sh $1
+sleep 15
+./contrib/multi-cluster/multi-init.sh $1

contrib/multi-cluster/extravars-creator.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+
+# Usage: ./contrib/multi-cluster/extravars-creator.sh $number_of_clusters
+
+# Alright what do we need...
+# 1. We need to generate inventories..
+
+echo "Warning: You're about to delete the existing extravars files!"
+# sleep 2
+
+rm -Rf ./inventory/multi-cluster
+mkdir -p ./inventory/multi-cluster
+
+masternumber=-2
+ip_master=47
+
+for (( c=1; c<=$1; c++ ))
+do
+  filename="./inventory/multi-cluster/cluster-$c.yml"
+  echo "Creating extravars file $filename"
+  # Increment the node numbers.
+  masternumber=$(($masternumber+3))
+  firstnodenumber=$(($masternumber+1))
+  secondnodenumber=$(($masternumber+2))
+  ip_master=$(($ip_master+3))
+  ip_first=$(($ip_master+1))
+  ip_second=$(($ip_master+2))
+  # Create the extra vars we need.
+  cat <<EOF > $filename
+kubeadm_version: v1.11.2
+hugepages_enabled: true
+image_destination_name: bootstrapped.qcow2
+spare_disk_attach: false
+pod_network_type: "none"
+enable_compute_device: true
+customize_kube_config: true
+network_type: "extra_interface"
+system_network: 192.168.122.0
+system_netmask: 255.255.255.0
+system_broadcast: 192.168.122.255
+system_gateway: 192.168.122.1
+system_nameservers: 192.168.122.1
+system_dns_search: example.com
+# ignore_preflight_version: true
+# bridge_networking: true
+# bridge_name: br0
+# bridge_physical_nic: "enp1s0f1"
+# bridge_network_name: "br0"
+# bridge_network_cidr: 192.168.1.0/24
+virtual_machines:
+  - name: kube-master-$masternumber
+    node_type: master
+    system_ram_mb: 4096
+    system_cpus: 1
+    static_ip: 192.168.122.$ip_master
+  - name: kube-node-$firstnodenumber
+    node_type: nodes
+    system_ram_mb: 4096
+    system_cpus: 1
+    static_ip: 192.168.122.$ip_first
+#  - name: kube-node-$secondnodenumber
+#    node_type: nodes
+#    system_ram_mb: 4096
+#    system_cpus: 1
+#    static_ip: 192.168.122.$ip_second
+enable_userspace_cni: true
+EOF
+done

contrib/multi-cluster/multi-init.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# First argument is number of clusters. See README.md for more details.
+
+for (( c=1; c<=$1; c++ ))
+do
+  extravars="./inventory/multi-cluster/cluster-$c.yml"
+  inventory="./inventory/multi-cluster/cluster-$c.inventory"
+  cmd="ansible-playbook -i \"$inventory\" -e \"@$extravars\" playbooks/kube-init.yml"
+  echo Running: $cmd
+  eval $cmd
+done

contrib/multi-cluster/multi-spinup.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# First argument is number of clusters. See README.md for more details.
+
+for (( c=1; c<=$1; c++ ))
+do
+  filename="./inventory/multi-cluster/cluster-$c.yml"
+  cmd="ansible-playbook -i inventory/virthost.inventory -e 'ssh_proxy_enabled=true' -e 'attach_additional_virtio_device=true' -e \"@$filename\" playbooks/virthost-setup.yml"
+  echo Running: $cmd
+  eval $cmd
+  mv inventory/vms.local.generated ./inventory/multi-cluster/cluster-$c.inventory
+  echo "New inventory @ ./inventory/multi-cluster/cluster-$c.inventory"
+done

contrib/multi-cluster/multi-teardown.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# First argument is number of clusters. See README.md for more details.
+
+for (( c=1; c<=$1; c++ ))
+do
+  extravars="./inventory/multi-cluster/cluster-$c.yml"
+  cmd="ansible-playbook -i inventory/virthost.inventory -e \"@$extravars\" playbooks/vm-teardown.yml"
+  echo Running: $cmd
+  eval $cmd
+done

contrib/multi-cluster/rebuild_inventory.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# ----------------------------------------
+# -- WORK IN PROGRESS
+# attempt at rebuilding inventory
+# after rebooted virthost.
+# ----------------------------------------
+
+virthost_ip=$(cat inventory/virthost.inventory | grep ansible_host | awk '{ print $2 }' | cut -d= -f2)
+
+VM=kube-master-1
+
+cat <<'EOF' > /tmp/shell.txt
+arp -an | grep "`virsh dumpxml THE_VIRTUAL_MACHINE | grep "mac address" | sed "s/.*'\(.*\)'.*/\1/g"`" | awk '{ gsub(/[\(\)]/,"",$2); print $2 }'
+EOF
+
+sed -i -e "s/THE_VIRTUAL_MACHINE/$VM/" /tmp/shell.txt
+
+MYCOMMAND=$(base64 -w0 /tmp/shell.txt)
+echo $MYCOMMAND | base64 -d
+
+# ssh user@remotehost "echo $MYCOMMAND | base64 -d | bash"
+
+# ssh root@$virthost_ip "arp -an | grep \"`virsh dumpxml $VM | grep \"mac address\" | sed \"s/.*'\(.*\)'.*/\1/g\"`\" | awk '{ gsub(/[\(\)]/,\"\",$2); print $2 }'"
+
+# #!/bin/bash
+# # Returns the IP address of a running KVM guest VM
+# # Assumes a working KVM/libvirt environment
+# #
+# # Install:
+# #   Add this bash function to your ~/.bashrc and `source ~/.bashrc`.
+# # Usage: 
+# #   $ virt-addr vm-name
+# #   192.0.2.16
+# #
+# virt-addr() {
+#     VM="$1"
+#     arp -an | grep "`virsh dumpxml $VM | grep "mac address" | sed "s/.*'\(.*\)'.*/\1/g"`" | awk '{ gsub(/[\(\)]/,"",$2); print $2 }'
+# }