Skip to content
This repository has been archived by the owner on Jan 4, 2022. It is now read-only.

Home

Jump to bottom
Jayfrown edited this page Dec 3, 2016 · 9 revisions

Container/Overlay

Container/Overlay builds a base container and leverages overlayfs to re-use its rootfs (/).

Containers made with cor create use this base filesystem to eliminate storage redundancies, and write their respective changes in separate, configurable directories.

The ideal approach is to install tools and middleware in base, with basic hardening configuration, barring any network config like "Listen" directives. These can be configured in the containers that will actually use these services. base should have all but core services disabled, so that individual containers may enable services when needed.

Updating base then instantly updates the building blocks for every other container. Put some creative thought in SAN setup and you might combine this behavior with block-level copying mechanisms to allow for near-instant global base upgrades across hypervisors.

lxd cannot distinguish between normal and overlaid containers. They can exist alongside one another.

usage: cor action|help [containerName|topic]

  1. Gains
  2. Goals
  3. Base container
  4. OverlayFS
  5. Empty

Gains

Container/Overlay effectively layers an OS - with its binaries, libraries and interpreters - from individual applications and/or their configuration. This does not mean you should swap an OS out from under an app and expect it to work, but -- there are myriad gains. Some highlights:

  • near-instant, one-shot system updates across thousands of containers
  • near-instant horizontal scaling
  • shared rootfs saves ~5GB per container
  • overlayfs' efficient memory usage saves ~512MB per container by keeping a single rootfs in-memory

Goals

A small overview of features I'd wish to implement, or other goals set for Container/Overlay.

  • compatible with vanilla lxd
  • survive hypervisor reboots
  • rebuild base container
  • remount overlay per container
  • delete feature -- umount overlay && delete container
  • remount all feature (simplifies reboots)
  • clone feature -- new container with /upper copied from ..
  • clusterclone feature -- new container with /upper from ..
  • rewrite to use lxd API rather than the binary client
  • rewrite to client/daemon model (python?)

Base container

The base container built by Container/Overlay is conveniently called base. Can't miss it.
It's a normal lxd container. You can interact with it as you would any other.

For example -- update it:

/usr/bin/lxc start base
/usr/bin/lxc exec base /bin/bash -c "pacman -Syu"
/usr/bin/lxc stop base
/usr/bin/systemctl restart lxd #ensures no stale cached files are used

The base containers's rootfs is used as the /lower directory for overlayfs, so changes made to base will be reflected in every overlay-enabled container -- as long as it has not made a local copy in the /upper directory.

OverlayFS

OverlayFS merges a container's /upper directory -- which contains its persistent storage -- with base's rootfs (/lower). The result is mounted at /merged -- in this case, the container's rootfs. Changes made under /merged will be persistently stored under /upper.

OverlayFS is very efficient with in-memory file caches. Every read() on a /merged where the end-result is a file from /lower, the file is loaded into memory once. Multiple containers doing a read() on the same file from /lower will result in a single file in memory, which will be referenced by all containers - unless explicitly overwritten in /upper.

Empty

That leaves one more default container, called empty. As you'd guess, it's empty.
It serves a single purpose - to be copied around without having to copy an entire rootfs.

Clone this wiki locally