This repository has been archived by the owner on Jul 16, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Documentation: Create ciao development environment setup documentation
Document the developer setup directly in the code vs the wiki. Update the document to refect the new ease of use features introduced by ciao-down. Explicitly document support for two modes of development - Using ciao-down, where the development is done on host and the testing within the ciao-down VM - Using bare metal, where the development and testing is done on the host
- Loading branch information
1 parent
dc1bc76
commit 266206c
Showing
1 changed file
with
260 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,260 @@ | ||
| # Ciao Single Machine Development and Test Environment | ||
|
|
||
| Developing cluster software is complicated if you must actually run a whole cluster on a set of physical machines. This begs for a development environment that is self contained and can be run without any setup. | ||
|
|
||
| The goal for the ciao development environment that | ||
| - Required very minimal setup by the user | ||
| - Does not affect the user's development system in any manner (i.e. the user can keep the firewall rules, selinux setup,... intact) | ||
| - Support a modes that allow running on powerful workstations to less powerful laptops | ||
| - Provide the ability to validate all code changes the user made against the Ciao release criterion | ||
|
|
||
| This page documents a way to set up an entire Ciao cluster inside a single machine. This cluster-in-a-machine mode is ideal for developers that desire the ability to build Ciao from sources, make changes and perform quick end to end functional integration testing without requiring multiple machines/VM's or creating a custom networking environment or maintaining a bevy of physical machines and a physical network. | ||
|
|
||
| We support two modes of operation | ||
| - ciao-down mode: Where a virtual machine is automatically created and launched, and the virtual cluster is setup and tested within the virtual machine | ||
| - bare metal mode: Where the virtual cluster is setup on the host machine itself | ||
|
|
||
| The ciao-down mode is the preferred mode of development on systems that have the resources and CPU capabilities needed, as it fully isolates the ciao virtual cluster. The ciao-down mode also does not require any change to the user's network firewall setup. Ciao down mode required VT-x nesting to be supported by the host. | ||
|
|
||
| The bare metal mode is the highest performance mode, but may require some network firewall modification. It also uses less resources and can run on machines whose CPU's do not support VT-x nesting. | ||
|
|
||
| In both modes Ciao is configured in a special all in one development mode where cluster nodes have dual roles (i.e launcher can be a Network Node and a Compute Node at the same time) | ||
|
|
||
| In the text below **machine** refers to the ciao-down VM in the case of the ciao-down mode, it refers to the host system in the case of the bare metal mode. | ||
|
|
||
| ## Components running on the Machine | ||
| 1. Controller | ||
| 2. Scheduler | ||
| 3. Compute+Network Node Agent (i.e. CN + NN Launcher) | ||
| 4. Workloads (Containers and VMs) | ||
| 5. WebUI | ||
| 6. Mock Openstack Services | ||
| 7. Machine Local DHCP Server | ||
| ... | ||
|
|
||
| The machine acts as the ciao compute node, network node, ciao-controller, ciao-scheduler and also hosts the ciao-webui as well as other openstack and dhcp services. | ||
|
|
||
| ## Graphical Overview | ||
|
|
||
| When the system is functioning the overall setup manifests as follows: | ||
|
|
||
| As you can see below the Cluster runs on a isolated virtual network resident inside the machine. | ||
| Hence the cluster is invisible outside the machine and completely self contained. | ||
|
|
||
| ``` | ||
| _____________________________________________________________________________________ | ||
| | | | ||
| | | | ||
| | | | ||
| | [Tenant VMs] [CNCI VMs] | | ||
| | | | | || | | ||
| | Tenant Bridges ---------- || | | ||
| | | || | | ||
| | | || | | ||
| | [scheduler] [controller] [keystone] [CN+NN Launcher] | || | | ||
| | || || || || | || | | ||
| | || || || || | || | | ||
| | || || || || | || | | ||
| | || || || || | || | | ||
| | || || || || | || | | ||
| | || || || || [DHCP/DNS | || | | ||
| | || || || || Server] | || | | ||
| | || || || || || | || | | ||
| | -------------------------------------------------------------------------- | | ||
| | Host Local Network Bridge + macvlan (ciao_eth, ciaovlan) | | ||
| | | | ||
| | | | ||
| | ____________________________________________________________________________________| | ||
| Development Machine | ||
| ``` | ||
|
|
||
| ---- | ||
|
|
||
| # Cluster External Network Access | ||
|
|
||
| If you desire to provide external network connectivity to the workloads then the host needs to act as gateway to the internet. The host needs to enable ipv4 forwarding and ensure all traffic exiting the cluster via the host is NATed. | ||
|
|
||
| This assumes the host has a single network interface. For multi homed systems, the setup is more complicated and needs appropriate routing setup which is outside the scope of this document. If you have a custom firewall configuration, you will need set things up appropriately. | ||
|
|
||
| Very simplistically this can be done by | ||
| ``` | ||
| iptables -t nat -A POSTROUTING -o $device -j MASQUERADE #$device is the network interface on the host | ||
| echo 1 > /proc/sys/net/ipv4/ip_forward | ||
| ``` | ||
|
|
||
| # Install Go | ||
| On the host install the latest release of go for your distribution | ||
| [Installing Go](https://golang.org/doc/install). | ||
|
|
||
| > NOTE: Go version 1.7 or later is required for Ciao. Ciao will not work with older version of Go. Hence it is best you download and install the latest version of Go if you distro is not on Go 1.7. | ||
|
|
||
| # Getting Started with ciao-down | ||
| ciao-down is a small utility for setting up a VM that contains | ||
| everything you need to run ciao's Single VM. All you need to have | ||
| installed on your machine is: | ||
|
|
||
| - Go 1.7 or greater | ||
|
|
||
| Then simply type | ||
|
|
||
| ``` | ||
| go get github.com/01org/ciao/testutil/ciao-down | ||
| $GOPATH/bin/ciao-down prepare | ||
| ``` | ||
|
|
||
| ciao-down will install some needed dependencies on your local PC such | ||
| as qemu and xorriso. It will then download an Ubuntu Cloud Image and | ||
| create a VM based on this image. It will boot the VM and install in that | ||
| VM everything you need to run ciao Single VM, including docker, ceph, | ||
| go, gcc, etc. When ciao-down prepare has finished you can connect to the | ||
| newly created VM with | ||
|
|
||
| ``` | ||
| $GOPATH/bin/ciao-down connect | ||
| ``` | ||
|
|
||
| Your host's GOPATH is mounted inside the VM. Thus you can edit your | ||
| the ciao code on your host machine and test in Single VM. | ||
|
|
||
| For more details and full set of capabilities of ciao-down see the full [ciao-down documentation ](https://github.com/01org/ciao/blob/master/testutil/ciao-down/README.md) | ||
|
|
||
|
|
||
| # Getting Started with Bare Metal | ||
|
|
||
| ## Install Docker | ||
| Install latest docker for your distribution based on the instructions from Docker | ||
| [Installing Docker](https://docs.docker.com/engine/installation/). | ||
|
|
||
| ## Install ciao dependencies | ||
|
|
||
| Install the following packages which are required: | ||
| 1. qemu-system-x86_64 and qemu-img, to launch the VMs and create qcow images | ||
| 2. gcc, required to build some of the ciao dependencies | ||
| 3. dnsmasq, required to setup a test DHCP server | ||
|
|
||
| On clearlinux all of these dependencies can be satisfied by installing the following bundles: | ||
| ``` | ||
| swupd bundle-add cloud-control go-basic os-core-dev kvm-host os-installer | ||
| ``` | ||
| ## Setup password less sudo | ||
|
|
||
| Setup passwordless sudo for the user who will be running the script below. | ||
|
|
||
| ## Download and build the sources | ||
|
|
||
| Download and build the ciao sources: | ||
| ``` | ||
| cd $GOPATH/src | ||
| go get -v -u -tags debug github.com/01org/ciao/... | ||
| ``` | ||
|
|
||
| You should see no errors. | ||
|
|
||
| # Verify that Ciao is fully functional using the **machine** | ||
|
|
||
| Now that you have the machine setup (either a bare metal setup or a ciao-down VM setup). | ||
|
|
||
| You can now quickly verify that all aspects of Ciao including VM launch, container launch, and networking. | ||
| These steps are performed inside the machine. | ||
|
|
||
| To do this simply run the following: | ||
| ``` | ||
| cd $GOPATH/src/github.com/01org/ciao/testutil/singlevm | ||
| #Cleanup any previous setup | ||
| ./cleanup.sh | ||
| #Set up the test environment | ||
| ./setup.sh | ||
| #Perform a full cluster test | ||
| ./verify.sh | ||
| ``` | ||
|
|
||
| The ```verify.sh``` script will: | ||
| - Create multiple Instances of Tenant VMs and Containers | ||
| - Test network connectivity between containers | ||
| - Test for ssh reach ability into VMs with private and external IPs | ||
| - Delete all the VM's and Container that were created | ||
|
|
||
| If the script reports success, it indicates to the developer that any changes made have not broken any functionality across all the Ciao components. | ||
|
|
||
| Meeting the goal originally outlined at the top of the page, build/setup/running your cluster all-in-one all transpires quickly and easily from the single script. The time needed for ./setup.sh and ./verify.sh to build ciao from source, configure it components into a virtual cluster, then launch and teardown containers and VMs is on the order of one minute total elapsed time. | ||
|
|
||
| ## Ongoing Usage | ||
|
|
||
| Once it's finished, the ```setup.sh``` script leaves behind a virtual cluster which can be used to perform manual tests. These tests are performed using the [ciao-cli](https://github.com/01org/ciao/blob/master/ciao-cli/README.md) tool. The ciao-cli tool requires that some environment variables be set up before it will work properly. These variables contain the URLs of the various ciao services and the credentials needed to access these services. The setup.sh script creates a shell source that contains valid values for the newly set up cluster. To initialise these variables you just need to source that file, e.g, | ||
|
|
||
| ``` | ||
| . ~/local/demo.sh | ||
| ``` | ||
|
|
||
| To check everything is working try the following command | ||
|
|
||
| ``` | ||
| ciao-cli workload list | ||
| ``` | ||
|
|
||
| # Running the BAT tests | ||
|
|
||
| The ciao project includes a set of acceptance tests that must pass before each release is made. The tests perform various tasks such as listing workloads, creating and deleting instances, etc. These tests can be run inside the machine | ||
|
|
||
| ``` | ||
| # Source the demo.sh file if you have not already done so | ||
| . ~/local/demo.sh | ||
| cd $GOPATH/src/github.com/01org/ciao/_release/bat | ||
| go test -v ./... | ||
| ``` | ||
|
|
||
| For more information on the BAT tests please see the [README](https://github.com/01org/ciao/blob/master/_release/bat/README.md). | ||
|
|
||
| # Cleanup / Teardown | ||
|
|
||
| To cleanup and tear down the cluster: | ||
| ``` | ||
| cd $GOPATH/src/github.com/01org/ciao/testutil/singlevm | ||
| #Cleanup any previous setup | ||
| ./cleanup.sh | ||
| ``` | ||
|
|
||
| # Known Issues with Bare Metal | ||
|
|
||
| - Does not work on Fedora due to default firewall rules. | ||
| https://github.com/01org/ciao/issues/526 | ||
|
|
||
| In order to allow the traffic required by the test cases you can add temporary rules like the ones show below | ||
| ``` | ||
| #!/bin/bash | ||
| iptables -I INPUT 1 -p tcp -m tcp --dport 8888 -j ACCEPT | ||
| iptables -I INPUT 1 -p 47 -j ACCEPT | ||
| iptables -I OUTPUT 1 -p 47 -j ACCEPT | ||
| iptables -I INPUT 1 -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT | ||
| iptables -I OUTPUT 1 -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT | ||
| iptables -I FORWARD 1 -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT | ||
| iptables -I FORWARD 1 -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT | ||
| iptables -I FORWARD 1 -p udp -m udp --dport 67:68 -j ACCEPT | ||
| iptables -I FORWARD 1 -p udp -m udp --dport 123 -j ACCEPT | ||
| iptables -I FORWARD 1 -p udp -m udp --dport 53 -j ACCEPT | ||
| iptables -I FORWARD 1 -p udp -m udp --dport 5355 -j ACCEPT | ||
| iptables -I FORWARD 1 -p icmp -j ACCEPT | ||
| ``` | ||
|
|
||
| And delete them after the tests using | ||
| ``` | ||
| #!/bin/bash | ||
| iptables -D INPUT -p tcp -m tcp --dport 8888 -j ACCEPT | ||
| iptables -D INPUT -p 47 -j ACCEPT | ||
| iptables -D OUTPUT -p 47 -j ACCEPT | ||
| iptables -D INPUT -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT | ||
| iptables -D OUTPUT -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT | ||
| iptables -D FORWARD -p tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT | ||
| iptables -D FORWARD -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT | ||
| iptables -D FORWARD -p udp -m udp --dport 67:68 -j ACCEPT | ||
| iptables -D FORWARD -p udp -m udp --dport 123 -j ACCEPT | ||
| iptables -D FORWARD -p udp -m udp --dport 53 -j ACCEPT | ||
| iptables -D FORWARD -p udp -m udp --dport 5355 -j ACCEPT | ||
| iptables -D FORWARD -p icmp -j ACCEPT | ||
| ``` | ||
|
|