Self host part 2

.gitignore

@@ -0,0 +1,25 @@
+node_modules
+*.pem
+pass
+.pass
+hellorunnable
+dump.rdb
+erl_crash.dump
+npm-debug.log
+ca.srl
+.DS_Store
+ansible/roles/hipache/templates/runnable*
+enviroments/**/k8/**/configMaps/*cert*
+dock.sh
+dock-pool.sh
+environments/**/secrets
+# Leaving for now while PR is merged
+environments/*/k8
+# Meant to not break anything. Might remove later
+environments/runnable-on-prem-test
+*.retry
+*.tfstate*
+terraform/credentials.tfvars
+terraform/.build
+ansible/secrets/*
+.idea

Dockerfile

@@ -0,0 +1,38 @@
+FROM ubuntu:14.04
+
+RUN apt-get update -y
+RUN DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y -q unzip build-essential python-pip python-dev python-yaml libxml2-dev libxslt1-dev zlib1g-dev git curl sshpass openssh-client
+RUN pip install --upgrade pyyaml jinja2 pycrypto
+
+RUN curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash - && \
+    apt-get install -y nodejs
+
+RUN curl -O https://releases.hashicorp.com/vault/0.6.3/vault_0.6.3_linux_amd64.zip && \
+    unzip ./vault_0.6.3_linux_amd64.zip -d /bin && \
+    chmod +x /bin/vault
+
+RUN git clone git://github.com/ansible/ansible.git --recursive /opt/ansible
+
+RUN cd /opt/ansible && \
+    git checkout v2.1.3.0-1 && \
+    git submodule update --init --recursive && \
+    bash -c 'source ./hacking/env-setup'
+
+ENV PATH        /opt/ansible/bin:$PATH
+ENV PYTHONPATH  /opt/ansible/lib:$PYTHONPATH
+ENV MANPATH     /opt/ansible/docs/man:$MANPATH
+
+ADD ./ssh /root/.ssh
+RUN echo 'eval `ssh-agent`' >> /root/start.sh
+RUN echo 'ssh-add /root/.ssh/id_rsa' >> /root/start.sh
+RUN echo 'npm start' >> /root/start.sh
+RUN chmod +x /root/start.sh
+
+ADD ./ansible/ /ansible
+RUN cd /ansible && npm install
+
+ADD ./deployer/ /deployer
+RUN cd /deployer && npm install
+
+WORKDIR /deployer
+CMD /root/start.sh

PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,23 @@
+We should treat our k8 files as compiled files, since we don't actually (http://blog.andrewray.me/dealing-with-compiled-files-in-git/). We should probably wait some time until we actually have more confidence in our compilation.
+
+[//]: # (Let's get your best description here about what's happend! Here's a list as well, if you like:)
+
+* I removed this function
+* I fixed all these things, etc.
+
+#### Dependencies
+
+- [ ] list dependencies (eg, PR from another branch or repo; tags or versions required prior to deployment)
+
+#### Tests
+
+> Test any modifications on one of our environments.
+
+- [ ] tested on _environment_ by _someone_
+
+#### Deployment (post-merge)
+
+> Ensure that all environments have the given changes.
+
+- [ ] deployed to gamma
+- [ ] deployed to delta

README.md

@@ -1,4 +1,155 @@
 devops-scripts
 ==============
 
-devops-scripts
+Scripts for managing our deployments.
+
+# How to Deploy at Runnable
+## Setup
+
+Before you can deploy you'll need to install the appropriate tools, scripts, and keys on your local machine.
+To do so, execute the following steps:
+
+1. Install Ansible v2.2.1.0 (the deploy automation tool we use to deploy projects to production)
+Installation: http://docs.ansible.com/intro_installation.html
+Upgrading: `sudo pip install ansible==2.2.1.0` or http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip
+
+2. Install JMESPath: 
+`pip install jmespath-terminal`
+
+3. Get the latest devops-scripts (the recipes that we use to deploy various projects)
+https://github.com/CodeNow/devops-scripts
+
+4. Change to the devops scripts repo directory and run the following command:
+`ln -s /<local-path-to-devops-scripts>/ssh/config ~/.ssh/config`
+
+5. Obtain the "Ansible Secrets" zip for the environment you want to deploy (or create the new environment following [./environments/README.md](./environments/README.md))
+
+6. Unzip file obtained above into `devops-scripts/environments/${YOUR_ENV}/secrets`
+
+7. Copy the `*.pem` files from `devops-scripts/ansible/secrets` to your `~/.ssh` directory
+
+8. Install two required tools onto your machine:
+```bash
+brew update && brew install vault daemon
+```
+
+At this point you should be capable of deploying; keep reading to find out how to actually perform a deploy!
+
+## Deploying Services
+- **IMPORTANT:** always pull latest devopts-scripts (`git pull origin master`)
+- **IMPORTANT:** Before you deploy a new version of any project make sure to determine which version of the project is currently deployed. This way you can quickly revert to the last stable release if something goes wrong after pushing a new version.
+
+### Step 1: Determine the Current Deploy Version
+To determine the latest deploy tag for a project please check the project's repository on
+github and look for the latest release tag (should be in the form `vX.Y.Z`). Once you've located the tag,
+copy it down somewhere that is easily and quickly accessible (you may need to use it quickly if something goes wrong).
+
+### Step 2: Deploy the Project via `ansible-playbook`
+
+- **WARNING:** If you were unable to determine the last deploy tag for a project and cannot revert **STOP**.
+  Ask someone on the team for help before continuing.
+- **IMPORTANT:** All commands should be run from the `devops-script/ansible` directory.
+
+#### Ansible Vault
+
+Please note that there are playbook that require encrypted [ansible vault](http://docs.ansible.com/ansible/playbooks_vault.html) files. If you see the following error:
+
+```bash
+ERROR: A vault password must be specified to decrypt # snip
+```
+
+you will need to re-run the playbook with:
+
+```bash
+--ask-vault-pass
+```
+
+#### Latest Tag
+Build and deploy a service to the latest tag of its repository. This will build
+the docker image needed to run the container on our infrastructure.
+
+#### Branch or Tag
+Build and deploy a service to a specific branch or tag on its repository. This performs a build
+of the docker image needed to run the service on our architecture.
+
+##### Command
+```
+ansible-playbook -i ../[inventory_dir] [service-playbook] -e @../environments/[gamma-or-delta]/main.yml -e git_branch=[branch-or-tag] -t deploy
+```
+
+##### Arguments
+- `[inventory_dir]` - The environment inventory files (servers and variables). Should be one of the following:
+  - `/enviroments/stage` - Runnable sandbox staging environment services
+  - `/environments/gamma` - Gamma services (internal use only; production mirror)
+  - `/environments/delta` - Delta services (real production)
+- `[main-var-file]` - The file with the main variables for the environment
+- `[service-playbook]` - The playbook for the service you wish to deploy, ex:
+  - `api.yml` - Deploys both the api and the api-workers services
+  - `shiva.yml` - Deploys the shiva micro-service
+  - `charon.yml` - Deploys a specific version of charon DNS to all docks
+- `[branch-or-tag]` - The branch or tag you wish to deploy, ex:
+  - `-e git_branch=v1.9.9` (version tag)
+  - `-e git_branch=my-feature-branch` (branch)
+  - `-e git_branch=3928745892364578623` (specific commit)
+
+##### Rebuild and Deploy Tag or Branch (No Cache)
+Forces a rebuild of a docker image for the given service at the given branch or tag and then deploys the
+newly created image. This is useful when a previously deployed branch has new changes that need to
+be deployed to an environment.
+
+Generally this command is only used with `gamma-hosts/` as it is often used to update code
+being tested in the production mirror.
+
+##### Command
+```
+ansible-playbook -i ../[inventory_dir] [service-playbook] -e @../environments/[gamma-or-delta]/main.yml] -e git_branch=[branch-or-tag] -t deploy -e build_args=--no-cache
+```
+
+##### Arguments
+- `[inventory_dir]` - The environment inventory files (servers and variables).
+- `[main-var-file]` - The file with the main variables for the environment
+- `[service-playbook]` - The playbook for the service you wish to deploy.
+- `[branch-or-tag]` - The branch or tag you wish to deploy.
+
+## Reverting
+If, for some reason, the new deploy is not operating as expected you can quickly revert by referencing the tag you collected in Step 1.
+Simply run the appropriate deploy command in the previous section with the last release tag and the new deploy will be reverted.
+
+## Deploy Songs
+
+- **IMPORTANT:** Make sure to play the song loud and proud when deploying!
+
+It is the custom at Runnable to play a song to the entire team when deploying. For each of the repositories here are the respective songs:
+
+| Service | Deploy Song Link |
+| ------- | ---------------- |
+| api / api-workers | [Push it - Rick Ross](https://www.youtube.com/watch?v=qk2jeE1LOn8) |
+| arithmancy | [onerepublic - Counting Stars](https://www.youtube.com/watch?v=hT_nvWreIhg) |
+| big poppa | [Big Poppa - The Notorious B.I.G.](https://www.youtube.com/watch?v=phaJXp_zMYM) |
+| charon | [Enter Sandman - Metallica](https://www.youtube.com/watch?v=CD-E-LDc384) |
+| clio | [Billy Joel - We Didn't Start the Fire](https://www.youtube.com/watch?v=eFTLKWw542g) |
+| cream | [C.R.E.A.M. - Wu-Tang Clan](https://www.youtube.com/watch?v=PBwAxmrE194) |
+| customerbot | [Trailer Park Boys Theme](https://www.youtube.com/watch?v=dI6Drn3OA70) |
+| deployer | [Rollout - Ludacris](https://www.youtube.com/watch?v=t21DFnu00Dc) |
+| detention | [Unbreakable Kimmy Schmidt](https://youtu.be/CV9xF8CjhJk?t=21s) |
+| docker-listener | [Call Me Maybe - Carly Rae Jepsen](https://www.youtube.com/watch?v=fWNaR-rxAic) |
+| drake | [Drake - Hotline Bling](https://www.youtube.com/watch?v=uxpDa-c-4Mc)
+| filibuster | [He's a Pirate - Pirates Of The Caribbean](https://www.youtube.com/watch?v=yRh-dzrI4Z4) |
+| Full Stack Deploy (`all.yml`) | [The Cleveland Orchestra (George Szell conducting) Ludwig von Beethoven Symphony No. 9 "Chorale (Ode To Joy)" Opus 125 IV.](https://www.youtube.com/watch?v=4g5770gaais) |
+| github-proxy | [Proxy - Martin Garrix](https://www.youtube.com/watch?v=NWB6-PJw4Mk) |
+| khronos | [Time After Time - Cyndi Lauper](https://www.youtube.com/watch?v=VdQY7BusJNU) |
+| krain | [Men at Work - Down Under](https://www.youtube.com/watch?v=XfR9iY5y94s) |
+| link | [Zelda Main Theme Song](https://www.youtube.com/watch?v=cGufy1PAeTU) |
+| mavis | [Fairy Tail Theme song](https://www.youtube.com/watch?v=R4UFCTMrV-o) |
+| navi | [Ocarina of Time: Lost Woods The Legend of Zelda](https://www.youtube.com/watch?v=iOGpdGEEcJM) |
+| optimus | [Original Transformers Opening Theme](https://www.youtube.com/watch?v=nLS2N9mHWaw) |
+| pheidi | [Chariots of Fire Theme](https://www.youtube.com/watch?v=CSav51fVlKU) |
+| runnable-angular | [Push It To The Limit - Scarface](https://www.youtube.com/watch?v=9D-QD_HIfjA) |
+| sauron | [Sauron Theme Song from LOTR](https://www.youtube.com/watch?v=V_rk9VBrXMY) |
+| Security Groups | [Out Of The Woods - Tayor Swift](https://www.youtube.com/watch?v=JLf9q36UsBk)
+| shiva | [FFXIV Shiva Theme](https://www.youtube.com/watch?v=noJiH8HLZw4) |
+| starlord | [Blue Swede - Hooked on a Feeling](https://www.youtube.com/watch?v=NrI-UBIB8Jk) |
+| swarm-deamon | [Pink Floyd - Another Brick In The Wall](https://www.youtube.com/watch?v=5IpYOF4Hi6Q) |
+| swarm-manager | [Eric Prydz VS Pink Floyd - 'Proper Education'](https://www.youtube.com/watch?v=IttkDYE33aU) |
+| varnish | [Karate Kid Theme Song](https://www.youtube.com/watch?v=VIYqtkdMxQg) |
+| vault / vault-values | [Seal - Kiss From A Rose](https://www.youtube.com/watch?v=zP3so2hY4CM) |

ansible/.gitignore

@@ -0,0 +1,2 @@
+_cache
+*.pyc

ansible/README.md

@@ -0,0 +1,15 @@
+
+Ansible provides a framework for our administration and deployment. It requires an organization for scripts and variables.  By design it uses SSH to connect to all hosts before it executes the actions. As such it can be run from any machine. All Ansible provided functionality is idempotent and it strongly encourage custom scripts match that standard.
+
+Here is the organization of the files in `devops-scripts/ansible`
+
+* `*-hosts` - Files naming all the servers
+* `*.yml` - The top level ansible actions.  These files describe how a host has vars and roles executed on it. 
+* `/group_vars` - yml files that define variables and values for your ansible scripts. This mostly maps one to one with machine types in AWS. They’re a key value map. 
+* `/library` - Third party libraries and scripts. 
+* `/roles` - A set of folders containing the ansible roles. A role defines the executable actions by ansible.  The center pieces is the `/tasks/main.yml`. It defines name actions and requirements. 
+The role can have several sub folders.
+  * `/handlers` - ??? 
+  * `/defaults` - ???
+  * `/meta` - contains dependencies
+  * `/template` - templates for any files that need to be generate and delivered. 

ansible/agreeable-egret.yml

@@ -0,0 +1,11 @@
+---
+- hosts: agreeable-egret
+  vars_files:
+    - group_vars/alpha-agreeable-egret.yml
+  roles:
+  - role: notify
+
+  - role: builder
+
+  - role: k8-deployment
+  - role: k8-service

ansible/ansible.cfg

@@ -0,0 +1,3 @@
+[defaults]
+# Required so `sudo: yes` does not lose the environment variables, which hold the ssh-agent socket
+sudo_flags = -HE

ansible/api-core.yml

@@ -0,0 +1,14 @@
+---
+- hosts: api
+  vars_files:
+    - group_vars/alpha-api-base.yml
+    - group_vars/alpha-api.yml
+  roles:
+  - role: notify
+    rollbar_token: "{{ api_rollbar_token }}"
+
+  - role: builder
+
+  - role: docker_client
+  - role: k8-deployment
+  - role: k8-service

ansible/api.yml

@@ -0,0 +1,3 @@
+- include: api-core.yml
+- include: socket-server.yml
+- include: workers.yml

ansible/app-services.yml

@@ -0,0 +1,6 @@
+- include: detention.yml git_branch="{{ detention_branch }}"
+- include: drake.yml git_branch="{{ drake_branch }}"
+- include: eru.yml git_branch="{{ eru_branch }}"
+- include: metis.yml git_branch="{{ astral_branch }}"
+- include: shiva.yml git_branch="{{ astral_branch }}"
+# run with: `--extra-vars "@current_versions.yml"`

ansible/arithmancy.yml

@@ -0,0 +1,11 @@
+---
+- hosts: arithmancy
+  vars_files:
+    - group_vars/alpha-arithmancy.yml
+  roles:
+  - role: notify
+    rollbar_token: "{{ arithmancy_rollbar_token }}"
+
+  - role: builder
+
+  - role: k8-deployment

ansible/base.yml

@@ -0,0 +1,23 @@
+---
+- hosts: localhost
+  connection: local
+  tasks:
+  - fail: msg="`host` (target host) needs to be defined to run this role"
+    when: host is not defined
+
+  - add_host:
+      name={{ host }}
+      groups=dock
+
+- hosts: "{{ host }}"
+  roles:
+  - { role: apt_update }
+  - { role: package-dock, tags: [ dock, package ] }
+  - { role: package-aws, tags: [ dock, package ] }
+  - { role: package_ntp }
+  - { role: build_essential }
+  - { role: docker, tags: [ docker ] }
+  - { role: datadog, tags: [ datadog ] }
+  - { role: ulimits, tags: [ ulimits ] }
+  - { role: loggly, tags: [ loggly, clean ] }
+  - { role: node }

ansible/bastion.yml

@@ -0,0 +1,9 @@
+---
+- hosts: bastion
+  roles:
+  - role: notify
+    tags: [ notify ],
+    app_name: bastion_sshd,
+    git_branch: latest,
+    name: bastion_sshd
+  - { role: bastion_sshd, tags: bastion-sshd }

ansible/big-poppa-http.yml

@@ -0,0 +1,13 @@
+---
+- hosts: big-poppa
+  vars_files:
+    - group_vars/alpha-big-poppa-base.yml
+    - group_vars/alpha-big-poppa-http.yml
+  roles:
+  - role: notify
+    rollbar_token: "{{ big_poppa_http_rollbar_token }}"
+
+  - role: builder
+
+  - role: k8-deployment
+  - role: k8-service

ansible/big-poppa-worker.yml

@@ -0,0 +1,12 @@
+---
+- hosts: big-poppa
+  vars_files:
+    - group_vars/alpha-big-poppa-base.yml
+    - group_vars/alpha-big-poppa-worker.yml
+  roles:
+  - role: notify
+    rollbar_token: "{{ big_poppa_worker_rollbar_token }}"
+
+  - role: builder
+
+  - role: k8-deployment

ansible/big-poppa.yml

@@ -0,0 +1,2 @@
+- include: big-poppa-http.yml
+- include: big-poppa-worker.yml

ansible/cadvisor.yml

@@ -0,0 +1,7 @@
+---
+- hosts: docks
+  vars_files:
+    - "group_vars/alpha-cadvisor.yml"
+  roles:
+  - { role: notify, tags: "notify" }
+  - { role: container_kill_start }