diff --git a/docs/setup-guides/dev-in-a-box.md b/docs/setup-guides/dev-in-a-box.md
index a71bbfa7..b0a4783e 100644
--- a/docs/setup-guides/dev-in-a-box.md
+++ b/docs/setup-guides/dev-in-a-box.md
@@ -8,14 +8,27 @@ description: Minimal evaluation site with no compute services
To get familiar with the setup procedures for chi-in-a-box, or to develop services, you can follow this (very) minimal setup guide.
-Note: This doesn't include support for HAProxy, TLS, or any compute services (Nova, Ironic, or Zun). Please don't use this guide for a production site, as it compromises security and reliability for simplicity.
+Note: This doesn't include support for TLS, or for communication outside of the single controller node. Services will be accessed via SSH tunnel or similar method.
## Prerequisites:
-Baremetal or Virtual Machine with 4 cores, 8 gb ram, 40gb disk.
+Baremetal or Virtual Machine with 8 cores, 16gb ram, 80gb disk.
+This is roughly double the minimum for just control-plane services, as we will also be using
+these resources to host "virtual" baremetal nodes.
You must have a user account on the machine with passwordless sudo.
+If launching a VM on Chameleon to host this, follow the following steps.
+
+1. Launch an instance on KVM
+ flavor: m1.xlarge
+ image: CC-Ubuntu20.04
+ network: sharednet1
+2. Attach a floating IP
+3. Allow SSH via the floating IP (by applying the appropriate security group)
+4. SSH into the node as cc@floating-ip
+
+
## Installation
1. Prepare a working directory. We use `/opt` by convention. It must be writable by your current user.
@@ -33,33 +46,65 @@ You must have a user account on the machine with passwordless sudo.
```
3. export an env var so you don't need to type "--site" for the remaining commands
- export CC_ANSIBLE_SITE=/opt/site-config/
-
-4. Create a dummy loopback interface to bind services to. You can choose whatever interface name and IP you want. The name will be later used for the `network_interface` and the IP for `kolla_internal_vip_address` in the next step.
+ export CC_ANSIBLE_SITE=/opt/site-config/
+
+
+4. Create some veth-pairs to act as "dummy" network interfaces.
+
+ External API interface
+ ```bash
+ sudo ip link add name ext_api_veth type veth peer ext_api_vethb
+ sudo ip addr add 192.168.100.2/24 dev ext_api_veth
+ sudo ip link set ext_api_veth up
+ sudo ip link set ext_api_vethb up
+ ```
+
+ Internal API interface
+ ```bash
+ sudo ip link add name int_api_veth type veth peer int_api_vethb
+ sudo ip addr add 10.10.10.2/24 dev int_api_veth
+ sudo ip link set int_api_veth up
+ sudo ip link set int_api_vethb up
+ ```
+ Neutron Provider Network interfaces
```bash
- ip link add name diab type dummy
- ip addr add 10.100.100.1/32 dev diab
+ sudo ip link add name physnet1_veth type veth peer physnet1_vethb
+ sudo ip link set physnet1_veth up
+ sudo ip link set physnet1_vethb up
+
+ sudo ip link add name physnet2_veth type veth peer physnet2_vethb
+ sudo ip link set physnet2_veth up
+ sudo ip link set physnet2_vethb up
+
+ sudo ip link add name physnet3_veth type veth peer physnet3_vethb
+ sudo ip link set physnet3_veth up
+ sudo ip link set physnet3_vethb up
+ ```
+
+ We also want to make sure these intefaces aren't filtered by any firewalls.
+ If you're using firewalld, then the following commands will do it:
+
+ Here, `ens3` is our external facing interface, so we don't want to keep it
+ firewalled for safety, but open up everything else.
+
+ ```
+ sudo firewall-cmd --zone=public --add-interface=ens3
+ sudo firewall-cmd --set-default-zone trusted
```
-5. Edit `/opt/site-config/defaults.yml` to contain ONLY the following lines.
- ```yaml
- ---
- kolla_base_distro: ubuntu
- network_interface: diab
- kolla_internal_vip_address: 10.100.100.1
- enable_haproxy: no
+5. In your site-config, replace defaults.yml with the one for dev-in-a-box
- # Disable central logging to reduce resource usage (no elasticsearch or kibana)
- enable_central_logging: no
- # Disable prometheus to speed up deployment
- enable_prometheus: no
```
+ cp /opt/site-config/dev-in-a-box.yml /opt/site-config/defaults.yml
+ ```
+
6. Bootstrap the controller node, this will install apt packages, configure Docker, and modify /etc/hosts
```
./cc-ansible bootstrap-servers
```
+
7. Run prechecks to ensure common issues are avoided.
```
@@ -71,37 +116,119 @@ You must have a user account on the machine with passwordless sudo.
```
sudo systemctl disable --now nscd.service
```
-8. Next, we'll pull container images for all configured services. This is done by running:
-
+8. At this point, we should be ready to "deploy".
```
- ./cc-ansible pull
+ ./cc-ansible deploy
```
-9. We now need to generate the configuration that all of these services will use. This will combine the upstream defaults, contents of `chi-in-a-box`, and your `site-config`, and template config files into `/etc/kolla/`
+ Deploy is a composite action, executing the following sub-actions:
+ * `pull`: download relevant docker container images.
- ```
- ./cc-ansible genconfig
- ```
+ Deploy will not pull newer images if one is in the cache already. You can execute `pull` directly if you need to explicitly pull a newer version.
- * If you've added additional configuration, this step can warn you about invalid or missing config values, before actually modifying any running containers.
- * Even if the step passes, you may want to inspect files under `/etc/kolla/` to make sure they match your expectations.
-10. Finally, we want to deploy the containers for each service. This step will start each necessary container, including running one-off bootstrap steps. If you've updated any of the service configurations, this step will restart the relevant containers and apply that config.\
- Technically, this step includes the `genconfig` step above, but it's mentioned separately for clarity.
+ * `genconfig`: template and copy config files into /etc/kolla, but don't restart services to apply yet
+ This will combine the upstream defaults, contents of `chi-in-a-box`, and your `site-config`, and template config files into `/etc/kolla/`.
+ If you've added additional configuration, this step can warn you about invalid or missing config values, before actually modifying any running containers.
+ Even if the step passes, you may want to inspect files under `/etc/kolla/` to make sure they match your expectations.
+
+
+ * `deploy-containers`: check and if necessary update running containers. This can be run separately if you want to make sure containers are running, but explicitly don't want to touch the templated configuration, for example if you've made edits to your site config, or temporary changes directly in /etc/kolla.
+
+9. If all the steps so far have passed, all the core services should now be running! However, this isn't everything needed for a useful cloud. `post-deploy` consists of all the steps that require a functioning control plane. These include:
+ * Creating default networks
+ * Creating compute "flavors"
+ * Uploading default disk images for users to use
+ * Installing various hammers and other utility services/cron-jobs.
+ To run this step, execute:
```
- ./cc-ansible deploy
+ ./cc-ansible post-deploy
```
-11. If all the steps so far have passed, all the core services should now be running! However, this isn't everything needed for a useful cloud. `post-deploy` consists of all the steps that require a functioning control plane. These include:
- 1. Creating default networks
- 2. Creating compute "flavors"
- 3. Uploading default disk images for users to use
- 4. Installing various hammers and other utility services/cron-jobs.
- 5. To run this step, execute:
- ```
- ./cc-ansible post-deploy
- ```
-12. Use your site! You can access it by the following methods:
- 1. Horizon is listening on 127.0.0.1:80, you can access it by forwarding your browser over SSH, for example via sshuttle. The username is `admin`, and the password can be viewed by running the following command:
+10. Now, we'll create some "virtual" baremetal nodes, used to test out the site. These are just VMs run with libvirt, but are
+configured via IPMI and pxe network booted like a baremetal node, so we can exercise Ironic. The following playbook will install and configure the `tenks` utility to accomplish this. At the end of the invocation, it will print out some commands for you to run yourself to finish the setup.
+
+```
+./cc-ansible --playbook playbooks/fake_baremetal.yml
+```
+
+11. Run the commands to bring up tenks. They will look something like the following, but may vary if you've changed your site-config from what's included here.
+
+```
+cd /opt/tenks
+source .venv/bin/activate # activate tenks virtualenv
+source /opt/site-config/admin-openrc.sh # source admin credentials to enroll nodes
+
+ansible-playbook \
+ --inventory ansible/inventory/ \
+ ansible/deploy.yml \
+ --extra-vars="@override.yml"
+```
+
+Once it's finished, we'll need to add an IP address to the bridge it attached
+for the ironic-provisioning network.
+
+```
+sudo ip addr add 10.205.10.1/24 dev brtenks0
+```
+
+12. At this point, it should all be up and running! Try out one of the "baremetal nodes"
+
+ * `openstack baremetal node list`
+ ```
+ +--------------------------------------+------+---------------+-------------+--------------------+-------------+
+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+ +--------------------------------------+------+---------------+-------------+--------------------+-------------+
+ | 44d0ac06-77d6-4444-93ce-7c8db5b54fff | tk0 | None | power off | available | False |
+ +--------------------------------------+------+---------------+-------------+--------------------+-------------+
+ ```
+ * let's trigger an inspection.
+ ```
+ openstack baremetal node manage tk0
+ openstack baremetal node inspect tk0
+ ```
+ * to see what's happening on the controller:
+ ```
+ sudo tail -f /var/log/kolla/ironic/ironic-conductor.log
+ ```
+ * once you see a line like
+ > Successfully set node 44d0ac06-77d6-4444-93ce-7c8db5b54fff power state to power on by power on.
+
+ this means that the VM has been successfully "powered on".
+ * to watch the machine's "serial console", execute:
+ ```
+ sudo virsh console tk0
+ ```
+ Note: to exit this shell, press `ctrl+]`
+
+13. Download some real images to use! To avoid downloading "all" the images, we'll manually invoke the chameleon image download tool.
+ ```
+ sudo docker run --rm --net=host \
+ -v "/etc/chameleon_image_tools/site.yaml:/etc/chameleon_image_tools/site.yaml" \
+ ghcr.io/chameleoncloud/chameleon_image_tools:latest \
+ deploy \
+ --site-yaml /etc/chameleon_image_tools/site.yaml \
+ --latest ubuntu jammy base;
+ ```
+ After a little while, you'll start seeing images like `CC-Ubuntu22.04` in the output of `openstack image list`
+
+14. Lets launch a real instance. NOTE!: At this point, we've configured the node in ironic, but NOT in blazar, so we can launch instances without a reservation.
+ * Our node should have finished inspection, so we'll need to move it from `manageable` back to `available`
+ ```
+ openstack baremetal node provide tk0
+ ```
+ * Now that it's "available", we can launch a server on it.
+ ```
+ openstack server create --flavor baremetal --image CC-Ubuntu22.04 --network sharednet1 test-instance
+ ```
+ If you access the console again via `sudo virsh console tk0`, you should see the image get written to disk, then the VM reboot into the final image.
+
+15. Access your site! You can access it by the following methods:
+
+ 1. All services are listening on the haproxy VIP on "ext_api_veth" interface, so you need a way to get to 192.168.100.0/24.
+ We recommend using `sshuttle` on your local machine, invoked as:
+ `sshuttle -r cc@ 192.168.100.0/24`
+
+ 2. HAProxy and thus Horizon are listening on 192.168.100.254:80, which you can access after starting sshuttle. The username is `admin`, and the password can be viewed by running the following command:
```
./cc-ansible view_passwords | grep "^keystone_admin_password"
diff --git a/playbooks/fake_baremetal.yml b/playbooks/fake_baremetal.yml
new file mode 100644
index 00000000..5fdbc37d
--- /dev/null
+++ b/playbooks/fake_baremetal.yml
@@ -0,0 +1,4 @@
+- hosts: fake-baremetal
+ roles:
+ - role: fake-baremetal
+ when: enable_ironic
diff --git a/roles/fake-baremetal/defaults/main.yml b/roles/fake-baremetal/defaults/main.yml
new file mode 100644
index 00000000..4b741b7d
--- /dev/null
+++ b/roles/fake-baremetal/defaults/main.yml
@@ -0,0 +1,25 @@
+tenks_install_dir: ~/tenks
+ansible_roles_dir: "{{tenks_install_dir}}/ansible/roles"
+
+libvirt_pool_path: "{{ tenks_libvirt_pool_path | default('/var/lib/libvirt/tenks_pool/')}}"
+
+tenks_git_repo: https://github.com/ChameleonCloud/tenks
+
+tenks_ansible_galaxy_roles:
+ - role_name: stackhpc.libvirt-host
+ role_path: "{{tenks_install_dir}}/ansible/roles/stackhpc.libvirt-host"
+ - role_name: stackhpc.libvirt-vm
+ role_path: "{{tenks_install_dir}}/ansible/roles/stackhpc.libvirt-vm"
+
+ironic_deploy_image_names:
+ - pxe_deploy_kernel
+ - pxe_deploy_ramdisk
+
+# credentials to allow neutron ngs to ssh to local ovs
+generic_switch_user: ngs_ovs_manager
+generic_switch_pubkey: "{{ kolla_ssh_key.public_key }}"
+# what neutron physnet is the ironic provisioning network attached to
+fake_baremetal_node_ram_mb: 4096
+fake_baremetal_node_disk_gb: 10
+fake_baremetal_node_vcpu: 2
+fake_baremetal_node_count: 3
diff --git a/roles/fake-baremetal/tasks/main.yml b/roles/fake-baremetal/tasks/main.yml
new file mode 100644
index 00000000..3a9bad7d
--- /dev/null
+++ b/roles/fake-baremetal/tasks/main.yml
@@ -0,0 +1,121 @@
+---
+- name: clone tenks git repo
+ ansible.builtin.git:
+ repo: "{{ tenks_git_repo }}"
+ dest: "{{ tenks_install_dir }}"
+
+- name: create venv and install build deps
+ ansible.builtin.pip:
+ virtualenv: "{{ tenks_install_dir }}/.venv/"
+ state: latest
+ name:
+ - pip
+ - wheel
+
+- name: pip install tenks from the git checkout
+ ansible.builtin.pip:
+ name: "{{ tenks_install_dir }}/"
+ virtualenv: "{{ tenks_install_dir }}/.venv/"
+
+- name: Install tenks dependency roles from Ansible Galaxy
+ command: "ansible-galaxy install {{ item.role_name }} --roles-path={{ ansible_roles_dir }}"
+ args:
+ creates: "{{ item.role_path }}"
+ loop: "{{ tenks_ansible_galaxy_roles }}"
+
+- name: get facts for ironic images
+ kolla_toolbox:
+ module_name: openstack.cloud.image_info
+ module_args:
+ auth: "{{ openstack_auth }}"
+ image: "{{ item }}"
+ run_once: True
+ register: "image_info"
+ become: True
+ loop: "{{ ironic_deploy_image_names }}"
+
+- name: set facts for ironic images
+ vars:
+ deploy_kernel_img: "{{ image_info.results | selectattr('item', 'eq', 'pxe_deploy_kernel') | first }}"
+ deploy_ramdisk_img: "{{ image_info.results | selectattr('item', 'eq', 'pxe_deploy_ramdisk') | first }}"
+ ansible.builtin.set_fact:
+ deploy_kernel_id: "{{ deploy_kernel_img.image.id }}"
+ deploy_ramdisk_id: "{{ deploy_ramdisk_img.image.id }}"
+
+- name: get facts for ironic-provisioning network
+ kolla_toolbox:
+ module_name: openstack.cloud.networks_info
+ module_args:
+ auth: "{{ openstack_auth }}"
+ filters:
+ name: "{{ ironic_provisioning_network }}"
+ run_once: True
+ become: True
+ register: "provisioning_network_return"
+
+- name: set fact for provisoning network name
+ ansible.builtin.set_fact:
+ provisioning_network_physnet_name: "{{ provisioning_network_return.openstack_networks[0]['provider:physical_network'] }}"
+- name: set fact for provisoning network
+ ansible.builtin.set_fact:
+ provisioning_network_physnet: "{{ neutron_networks | selectattr('name', 'equalto', provisioning_network_physnet_name ) | first }}"
+
+- name: template tenks overide file from site-config
+ vars:
+ provisioning_physnet_name: "{{ provisioning_network_physnet.name }}"
+ provisioning_physnet_bridge: "{{ provisioning_network_physnet.bridge_name }}"
+ ansible.builtin.template:
+ src: tenks-override.yml.j2
+ dest: "{{ tenks_install_dir }}/override.yml"
+
+- name: create linux_group for neutron ssh to ovs
+ become: true
+ ansible.builtin.group:
+ name: "{{ generic_switch_user }}"
+
+- name: create linux_user for neutron ssh to ovs
+ become: true
+ ansible.builtin.user:
+ name: "{{ generic_switch_user }}"
+ create_home: yes
+ groups:
+ - "{{ generic_switch_user }}"
+
+- name: Allow 'generic_switch_user' group to have passwordless sudo
+ become: true
+ copy:
+ dest: /etc/sudoers.d/97_kolla_ngs_ovs
+ content: "%{{generic_switch_user}} ALL=(ALL) NOPASSWD: ALL"
+ validate: visudo -cf %s
+
+- name: set ssh publickey for neutron_ovs_ssh
+ become: true
+ ansible.posix.authorized_key:
+ user: "{{ generic_switch_user }}"
+ state: present
+ key: "{{ generic_switch_pubkey }}"
+
+- name: create wrapper for kolla-ovs
+ become: true
+ template:
+ src: ovs-vsctl.j2
+ dest: /usr/local/sbin/ovs-vsctl
+ mode: 'u+rwx'
+
+- name: create interface for ironic provisioning gw
+ become: true
+ vars:
+ ovs_bridge_name: "{{ provisioning_network_physnet.bridge_name }}"
+ vlan_tag: "{{ provisioning_network_return.openstack_networks[0]['provider:segmentation_id'] }}"
+ gateway_ip: "{{ ironic_provisioning_network_gateway }}"
+ gateway_prefix: "{{ironic_provisioning_network_cidr | ipaddr('prefix')}}"
+ block:
+ - name: create OVS patch port for ironic-gw
+ command: "ovs-vsctl add-port {{ovs_bridge_name}} vlan{{vlan_tag}} tag={{vlan_tag}} -- set Interface vlan{{vlan_tag}} type=internal"
+ failed_when: false
+ - name: set ip address on interface
+ command: "ip addr add {{gateway_ip}}/{{gateway_prefix}} dev vlan{{vlan_tag}}"
+ failed_when: false
+ - name: set link up
+ command: "ip link set vlan{{vlan_tag}} up"
+ failed_when: false
diff --git a/roles/fake-baremetal/templates/ovs-vsctl.j2 b/roles/fake-baremetal/templates/ovs-vsctl.j2
new file mode 100644
index 00000000..eded6d48
--- /dev/null
+++ b/roles/fake-baremetal/templates/ovs-vsctl.j2
@@ -0,0 +1,7 @@
+#!/bin/bash
+# execute ovs-vsctl inside the kolla docker container
+# installed by fake-baremetal playbook
+
+docker exec openvswitch_vswitchd \
+ ovs-vsctl \
+ "${@}"
\ No newline at end of file
diff --git a/roles/fake-baremetal/templates/tenks-override.yml.j2 b/roles/fake-baremetal/templates/tenks-override.yml.j2
new file mode 100644
index 00000000..d4a3d39c
--- /dev/null
+++ b/roles/fake-baremetal/templates/tenks-override.yml.j2
@@ -0,0 +1,44 @@
+# override where tenks vm disks will be stored
+libvirt_pool_path: {{libvirt_pool_path}}
+
+physnet_mappings:
+ {{provisioning_physnet_name}}: {{provisioning_physnet_bridge}}
+
+bridge_type: "openvswitch"
+
+# The Glance name or UUID of the image to use for the deployment kernel.
+deploy_kernel: {{ deploy_kernel_id }}
+# The Glance name or UUID of the image to use for the deployment ramdisk.
+deploy_ramdisk: {{ deploy_ramdisk_id }}
+
+node_types:
+ # The type name.
+ type0:
+ # The amount of RAM, in mebibytes.
+ memory_mb: {{fake_baremetal_node_ram_mb}}
+ # The number of virtual CPUs.
+ vcpus: {{fake_baremetal_node_vcpu}}
+ # A list of volumes, each with a capacity.
+ volumes:
+ - capacity: {{fake_baremetal_node_disk_gb}}GB
+ # A list of physical network names to connect to. These physical network
+ # names should be keyed in `physnet_mappings` in each hypervisor's host
+ # vars.
+
+ # note! only the first one listed will be used for PXE boot, and therefore must be ironic-provisioning!
+ physical_networks:
+ - {{provisioning_physnet_name}}
+
+specs:
+ # The type in `node_types` that this spec refers to. Required.
+ - type: type0
+ # The number of nodes to create of this spec. Required.
+ count: {{fake_baremetal_node_count}}
+ ironic_config:
+ # The resource class that nodes of this spec should use in Ironic.
+ # Required if `ironic_config` has been specified.
+ resource_class: baremetal
+
+nova_flavors:
+ - resource_class: baremetal
+ node_type: type0
diff --git a/site-config.example/dev-in-a-box.yml b/site-config.example/dev-in-a-box.yml
new file mode 100644
index 00000000..215fe13c
--- /dev/null
+++ b/site-config.example/dev-in-a-box.yml
@@ -0,0 +1,80 @@
+---
+# this configuration is meant to set up a CI/CD enviroment: "dev-in-a-box"
+# It assumes that the named network interfaces and IP addresses have been
+# created and assigned to veth interfaces, namely "ext_api_veth", "int_api_veth" and "neutron_veth"
+# no traffic will be sent outside of the VM.
+
+
+kolla_base_distro: ubuntu
+
+network_interface: int_api_veth
+kolla_external_vip_interface: "ext_api_veth"
+# api_interface: "{{ network_interface }}"
+
+kolla_internal_vip_address: "10.10.10.254"
+#kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"
+kolla_external_vip_address: "192.168.100.254"
+#kolla_external_fqdn: "{{ kolla_external_vip_address }}"
+
+
+# Note! This configuration does NOT invoke networking-generic-switch.
+# It assumes only "flat" networks are available, with a separate physnet
+# per neutron network. This is due to the networking limitation for "tenks"
+# virtual baremetal nodes. A regular chi-in-a-box site would be configured
+# with vlan networks, allowing users to create per-tenant isolated networks.
+
+neutron_networks:
+- name: public
+ bridge_name: br-ex
+ external_interface: public_veth
+ cidr: 192.168.100.0/24
+ gateway_ip: 192.168.100.1
+ allocation_pools:
+ - start: 192.168.100.20
+ end: 192.168.100.200
+- name: physnet1
+ bridge_name: br-physnet1
+ external_interface: physnet1_veth
+ on_demand_vlan_ranges:
+ - 200:250
+ reservable_vlan_ranges:
+ - 251:300
+ sharednet:
+ cidr: 10.200.10.0/22
+ allocation_pools:
+ - start: 10.200.10.20
+ end: 10.200.11.21
+
+ironic_provisioning_network_vlan: 199
+
+switch_configs:
+ - name: brtenks0
+ device_type: netmiko_ovs_linux
+ address: 127.0.0.1
+ auth:
+ username: ngs_ovs_manager
+ private_key: "{{ ovs_ssh_key.private_key }}"
+ ngs_config: {}
+
+# path on the system where glance images will be stored
+glance_file_datadir_volume: /var/lib/glance
+
+# disable to speed up deployment
+enable_central_logging: false
+enable_prometheus: false
+
+# enable image subscription. Make sure you have enough disk space before turning it on!
+# enable_image_deployer: true
+
+# for local OVS debugging
+generic_switch_pubkey: "{{ ovs_ssh_key.public_key }}"
+
+# fake baremetal node installation overrides
+tenks_install_dir: /opt/tenks
+tenks_libvirt_pool_path: /var/lib/libvirt/tenks_pool/
+
+# fake baremetal node vm config
+fake_baremetal_node_ram_mb: 4096
+fake_baremetal_node_disk_gb: 10
+fake_baremetal_node_vcpu: 2
+fake_baremetal_node_count: 1
diff --git a/site-config.example/inventory/hosts b/site-config.example/inventory/hosts
index 4844fb7d..909b8071 100644
--- a/site-config.example/inventory/hosts
+++ b/site-config.example/inventory/hosts
@@ -854,3 +854,7 @@ control
# Image
[chameleon-image-tools:children]
control
+
+# Development, fake baremetal hypervisor host using tenks
+[fake-baremetal:children]
+ironic