[streaming-system] Automated build and configuration of full streaming system

[mithro]

Brief explanation

System to build and configure all of the machines needed to record and stream a multi track conference.

Full automated deployment of both the collecting infrastructure (parts inside a conference venue / user group venue) and the off site infrastructure (parts in the cloud).

Expected results

Convert existing scripts and manual processes to a consistent and modular deployment system templates.
Demonstrate it working in VMs.

Detailed Explanation

Problem

A 25 track conference needs about 60 physical and 30 virtual machines. Each needs an os installed, host name assigned, networking that plays well with the conference infrastructure, various applications installed and configured.

Setup

For production there will be a physical box with Debian stable, access to a standard Debian repository, sudo user, a master config file, file of secrets and a script. The script will install and configure the needed services to provide automated pxe install for the remaining machines and provision and configure the vm's.

For testing, a single physical box running lots of vms that replicates the production setup. This adds a layer of complexity and such, but is the only sensible way to test and maintain and enhance such a system.

Configuration

Everything possible will be contained in the master config file. There should be no editing additional config files by hand. Part of this project will be to determine what all needs to be included in the master config file. The file will likely be created by the veyepar system which will be responsible for transforming the conference schedule into a consistent format and providing a UI to manage such things as:
Talks in the room that the attendees know as “UB2.252A (Lameere)” will have a url of www.timvideos.com/fosdem2014/lameere ad an irc channel of #fosdem-lameere; and the video mixer machine has a mac address of 00:11:22..., content source machines: [22:33:44... 44:55:66...], host names: lameere-mixer, lameere-source-1 and -2, and somehow specifying if the encoders will be EC2, RackSpace or local vm's.

Pieces

• collecting infrastructure
  • dvswitch or gst-switch
  • flumotion collector (watchdog, register, etc)
• streaming infrastructure (EC2)
  • timvideos frontend website
  • timvideos backend website
  • timvideos support (IRC log bot, etc)
  • flumotion encoder (watchdog, register, etc)

Further reading

Existing build systems that do quite a bit of what is needed:
https://github.com/yoe/fosdemvideo/blob/master/doc/README
https://github.com/CarlFK/veyepar/blob/master/setup/nodes/pxe/README.txt
https://github.com/timvideos/streaming-system/blob/master/tools/setup/runall.sh

Knowledge Prerequisites

• Good understanding of deploying machines and systems such as PXE boot.

Contacts

• Potential Mentors: @CarlFK @codersquid @iiie
• Mailing list: [email protected]

[mithro]

@maxstr Does your work with #21 make some of this simplier?

[rahulroxx]

@mithro i would like to do this as my gsoc'16 project

[mithro]

Please read our Summer of Code page at https://code.timvideos.us/summer-of-code/, especially the FAQ section.

[CarlFK]

This is mostly complete:
https://salsa.debian.org/debconf-video-team/ansible

It lacks a few of the described features, but I think those should be logged as bugs against the respective projects.

It doesn't have the build a farm of VMs tests, but I think that's ok. you can use qemu to test nodes, and some of the key pieces need real hardware to really test (hdmi2usb, usb audio.)

GitLab

DebConf videos / ansible

Ansible playbooks for the DebConf Video Team's machines - https://debconf-video-team.pages.debian.net/ansible/