Skip to content

Commit f14fe4a

Browse files
author
Mark Lavin
committed
Adding docs on server setup. See #30 
1 parent 7554a6d commit f14fe4a

File tree

4 files changed

+273
-83
lines changed

4 files changed

+273
-83
lines changed

‎README.rst

+1-83
Original file line numberDiff line numberDiff line change
@@ -57,92 +57,10 @@ You should now be able to run the development server::
5757
python manage.py runserver
5858

5959

60-
Setup repository
61-
------------------------
62-
63-
Before your project can be deployed to a server, the code needs to be
64-
accessible in a git repository.
65-
66-
1. Add your project code to a git repo, hosted somewhere your server can clone it from.
67-
68-
2. Edit ``fabfile.py`` near the top and insert your repo's URL. E.g., change this::
69-
70-
env.repo = u'' # FIXME: Add repo URL
71-
72-
to this::
73-
74-
env.repo = u'[email protected]:account/reponame.git'
75-
76-
77-
78-
Server Provisioning
79-
------------------------
80-
81-
The first step in creating a new server is to create users on the remote server. You
82-
will need root user access with passwordless sudo. How you specify this user will vary
83-
based on the hosting provider. EC2 and Vagrant use a private key file. Rackspace and
84-
Linode use a user/password combination.
85-
86-
1. For each developer, add a user record along with their SSH key into the ``conf/pillar/devs.sls``.
87-
88-
2. Set the project name in ``conf/pillar/project.sls`` and the environment's domain in
89-
``conf/pillar/<environment>/env.sls`` if it has not already been set.
90-
91-
3. Add any environment secrets to the ``conf/pillar/<environment>/secrets.sls``. This file is not in the source
92-
control by default but there are example ``secrets.ex`` files to use as a starting point.
93-
94-
4. Provision the box using the Salt bootstrap::
95-
96-
fab -H <fresh-server-ip> -u <root-user> <environment> provision
97-
98-
This will provision the box for the initial deploy (create users and install/configure necessary pacakges).
99-
100-
5. Add the IP to the appropriate environment function. You can now run the initial deploy::
101-
102-
fab <environment> deploy
103-
104-
105-
Vagrant Testing
106-
------------------------
107-
108-
You can test the provisioning/deployment using `Vagrant <http://vagrantup.com/>`_.
109-
Using the Vagrantfile you can start up the VM. This requires the ``precise32`` box::
110-
111-
vagrant up
112-
113-
With the VM up and running, you can create the necessary users.
114-
Put the developers' keys in ``conf/users`` as before, then
115-
use these commands to create the users. The location of the key file
116-
(/usr/lib/ruby/gems/1.8/gems/vagrant-1.0.2/keys/vagrant)
117-
may vary on your system. If installed via apt then this may be located in
118-
/usr/share/vagrant/keys/vagrant. Running ``locate keys/vagrant`` might
119-
help find it::
120-
121-
fab -H 33.33.33.10 -u vagrant -i /usr/lib/ruby/gems/1.8/gems/vagrant-1.0.2/keys/vagrant vagrant provision
122-
fab vagrant deploy
123-
124-
It is not necessary to reconfigure the SSH settings on the vagrant box.
125-
126-
The vagrant box forwards
127-
port 80 in the VM to port 8080 on the host box. You can view the site
128-
by visiting localhost:8080 in your browser.
129-
130-
You may also want to add::
131-
132-
33.33.33.10 dev.example.com
133-
134-
to your hosts (/etc/hosts) file.
135-
136-
You can stop the VM with ``vagrant halt`` and
137-
destroy the box completely to retest the provisioning with ``vagrant destroy``.
138-
139-
For more information please review the Vagrant documentation.
140-
141-
14260
Deployment
14361
------------------------
14462

145-
For future deployments, you can deploy changes to a particular environment with
63+
You can deploy changes to a particular environment with
14664
the ``deploy`` command. This takes an optional branch name to deploy. If the branch
14765
is not given, it will use the default branch defined for this environment in
14866
``env.branch``::

‎docs/provisioning.rst

+143
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
Server Provisioning
2+
========================
3+
4+
5+
Overview
6+
------------------------
7+
8+
{{ project_name|title }} is deployed on the following stack.
9+
10+
- OS: Ubuntu 12.04 LTS
11+
- Python: 2.7
12+
- Database: Postgres
13+
- Application Server: Gunicorn
14+
- Frontend Server: Nginx
15+
- Cache: Memcached
16+
17+
These services are configured to run together on a single machine. Each environment
18+
(``staging`` or ``production``) should run on a separate machine. `Supervisord <http://supervisord.org/>`_
19+
manages the application server process.
20+
21+
22+
Initial Setup
23+
------------------------
24+
25+
Before your project can be deployed to a server, the code needs to be
26+
accessible in a git repository. Once that is done you should update the ``env.repo`` in
27+
the ``fabfile.py``. E.g., change this::
28+
29+
env.repo = u'' # FIXME: Add repo URL
30+
31+
to this::
32+
33+
env.repo = u'[email protected]:account/reponame.git'
34+
35+
You also need to set the project name in `conf/pillar/project.sls``. This should
36+
match the ``env.project`` in ``fabfile.py``. For the environment you want to setup
37+
you will need to set the ``domain`` in ``conf/pillar/<environment>/env.sls``.
38+
39+
You will also need add the developer's user names and SSH keys to ``conf/pillar/devs.sls``. Each
40+
user record should match the format::
41+
42+
example-user:
43+
groups: [admin, login]
44+
public_key:
45+
- ssh-rsa <Full SSH Key would go here>
46+
47+
Additional developers can be added later but you will need to create at least on user for
48+
yourself.
49+
50+
51+
Managing Secrets
52+
------------------------
53+
54+
Secret information such as passwords and API keys should never be committed to the
55+
source repository. Instead aach environment manages is secrets in ``conf/pillar/<environment>/secrets.sls``.
56+
These ``secrets.sls`` files are excluded from the source control and need to be passed
57+
to the developers out of band. There are example files given in ``conf/pillar/<environment>/secrets.ex``.
58+
They have the format::
59+
60+
secrets:
61+
DB_PASSWORD: 'XXXXXX'
62+
63+
Each key/value pair given in the ``secrets`` dictionary will be added to the OS environment
64+
and can retrieved in the Python code via::
65+
66+
import os
67+
68+
password = os.environ['DB_PASSWORD']
69+
70+
Secrets for other environments will not be available. That is the staging server
71+
will not have access to the production secrets. As such there is no need to namespace the
72+
secrets by their environment.
73+
74+
75+
Setup Checklist
76+
------------------------
77+
78+
To summarize the steps above you can use the following checklist
79+
80+
- ``env.repo`` is set in ``fabfile.py``
81+
- Developer user names and SSH keys have been added to ``conf/pillar/devs.sls``
82+
- Project name has been in ``conf/pillar/project.sls``
83+
- Environment domain name has been set in ``conf/pillar/<environment>/env.sls``
84+
- Environment secrets have been set in ``conf/pillar/<environment>/secrets.sls``
85+
86+
87+
Provision
88+
------------------------
89+
90+
Once you have completed the above steps you are ready to provision a new server
91+
for a given environment. You will need to be able to connect to the server
92+
as a root user. How this is done will depend on where the server is hosted.
93+
VPS providers such as Linode will give you a username/password combination. Amazon's
94+
EC2 uses a private key. These credentials will be passed as command line arguments.::
95+
96+
# Template of the command
97+
fab -H <fresh-server-ip> -u <root-user> <environment> provision
98+
# Example of provisioning 33.33.33.10 as a staging machine
99+
fab -H 33.33.33.10 -u root staging provision
100+
101+
Behind the scenes this will rsync the states/pillars in ``conf`` over to the
102+
server as well as check out the base states from the `margarita <https://github.com/caktus/margarita>`_
103+
repo. It will then use the `masterless salt-minion <http://docs.saltstack.com/topics/tutorials/quickstart.html>`_
104+
to ensure the states are up to date.
105+
106+
Note that because of the use of rsync it is possible to execute configuration changes which
107+
have not yet been committed to the repo. This can be handy for testing configuration
108+
changes and allows for the secrets to be excluded from the repo but it's a double-edged sword.
109+
You should be sure to commit any configuration changes to the repo when they are ready.
110+
111+
Once a server has been created for its environment it should be added to the ``env.hosts``
112+
for the given environment. In our example we would add::
113+
114+
def staging():
115+
env.environment = 'staging'
116+
env.hosts = ['33.33.33.10', ]
117+
118+
At this point we can run the first deploy::
119+
120+
fab staging deploy
121+
122+
This will do the initial checkout of the repo source, install the Python requirements,
123+
run syncdb/migrate and collect the static resources.
124+
125+
126+
Updates
127+
------------------------
128+
129+
During the life of the project you will likely need to make updates to the server
130+
configuration. This might include new secrets add to the pillar, new developers
131+
added to the project or new services which need to be installed. Configuration updates
132+
can be made by calling the ``provision`` command again.::
133+
134+
# Template of the command
135+
fab <environment> provision
136+
# Reprovision the staging server
137+
fab staging provision
138+
139+
In this case we do not need to connect as the root user. We connect as our developer
140+
user. We also do not need to specify the host. It will use the ``env.hosts`` previously
141+
set for this environment.
142+
143+
For more information testing the provisioning see the doc:`vagrant guide </vagrant>`.

‎docs/server-setup.rst

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
Server Setup
2+
========================
3+
4+
5+
Provisioning
6+
------------------------
7+
8+
The server provisioning is managed using `Salt Stack <http://saltstack.com/>`_. The base
9+
states are managed in a `common repo <https://github.com/caktus/margarita>`_ and additional
10+
states specific to this project are contained within the ``conf`` directory at the root
11+
of the repository.
12+
13+
For more information see the doc:`provisioning guide </provisioning>`.
14+
15+
16+
Layout
17+
------------------------
18+
19+
Below is the server layout created by this provisioning process::
20+
21+
/var/www/
22+
{{ project_name }}/
23+
env/
24+
log/
25+
public/
26+
static/
27+
media/
28+
29+
``/var/www/{{ project_name }}/`` contains source code of the project. ``/var/www/env/``
30+
is the `virtualenv <http://www.virtualenv.org/>`_ for Python requirements. ``/var/www/log/``
31+
stores the Nginx, Gunicorn and other logs used by the project. ``/var/www/public/``
32+
holds the static resources (css/js) for the project and the uploaded user media.
33+
``/var/www/public/static/`` and ``/var/www/public/media/`` map to the ``STATIC_ROOT`` and
34+
``MEDIA_ROOT`` settings.
35+
36+
37+
Deployment
38+
------------------------
39+
40+
For deployment each developer connects to the server as their own user. Each developer
41+
has SSH access via their public key. These users are created/managed by the Salt
42+
provisioning. The deployment itself is automated with `Fabric <http://docs.fabfile.org/>`_.
43+
To deploy a developer simply runs::
44+
45+
# Deploy updates to staging
46+
fab staging deploy
47+
# Deploy updates to production
48+
fab production deploy
49+
50+
Each environment (``staging`` or ``production``) is tied to a particular Git branch managed
51+
by ``env.branch`` in the ``fabfile.py``. Deploying a different branch can be done by
52+
passing the branch name to the deploy command::
53+
54+
# Deploy new-feature to staging
55+
fab staging deploy:new-feature
56+
57+
Developers should coordinate to ensure that they do not deploy different branches on
58+
top of one another.
59+
60+
New python requirements add to the ``requirements/`` files and new South migrations
61+
are detected by grepping the Git diff on deploy. This works fairly well but if they
62+
need to be manually updated that can be done via Fabric::
63+
64+
# Installs new requirements from requirements/production.txt
65+
fab staging update_requirements
66+
# Runs syncdb/migrate
67+
fab staging syncdb

‎docs/vagrant.rst

+62
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
Vagrant Testing
2+
========================
3+
4+
5+
Starting the VM
6+
------------------------
7+
8+
You can test the provisioning/deployment using `Vagrant <http://vagrantup.com/>`_.
9+
Using the included Vagrantfile you can start up the VM. This requires Vagrant 1.2+ and
10+
the ``precise32`` box. The box will be installed if you don't have it already.::
11+
12+
vagrant up
13+
14+
The general provision workflow is the same as in the previous doc:`provisioning guide </provisioning>`
15+
so here are notes of the Vagrant specifics.
16+
17+
18+
Provisioning the VM
19+
------------------------
20+
21+
The ``fabfile.py`` contains a ``vagrant`` environment with the VM's IP already added.
22+
The rest of the environment is made to match the ``staging`` environment. If you
23+
have already configured the ``conf/pillar/staing/env.sls`` and ``conf/pillar/staing/secrets.sls``
24+
then you can continue provisioning the VM.
25+
26+
To connect to the VM for the first time you need to use the private key which ships
27+
with the Vagrant install. The location of the file may vary on your platform depending
28+
on which version you installed and how it was installed. You can use ``locate`` to find it::
29+
30+
# Example locate with output
31+
$ locate keys/vagrant
32+
/opt/vagrant/embedded/gems/gems/vagrant-1.2.2/keys/vagrant
33+
/opt/vagrant/embedded/gems/gems/vagrant-1.2.2/keys/vagrant.pub
34+
35+
You can then call the initial provision using this key location for the ``-i`` option::
36+
37+
fab -u vagrant -i /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/keys/vagrant vagrant provision
38+
39+
After that has finished you can run the initial deploy::
40+
41+
fab vagrant deploy
42+
43+
44+
Testing on the VM
45+
------------------------
46+
47+
With the VM fully provisioned and deployed you can access the VM on localhost port 8089. Since
48+
the Nginx configuration will only listen for the domain name in ``conf/pillar/staing/env.sls``
49+
you will need to modify your ``/etc/hosts`` configuration to view it. You will need to add::
50+
51+
127.0.0.1 <domain>
52+
53+
where ``<domain>`` matches the domain in ``conf/pillar/staing/env.sls``. For example lets use
54+
staging.example.com::
55+
56+
127.0.0.1 staging.example.com
57+
58+
In your browser you can now view staging.example.com:8089 and see the VM running the full
59+
web stack.
60+
61+
Note that this ``/etc/hosts`` entry will prevent you from accessing the true staging.example.com.
62+
When your testing is complete you should remove or comment out this entry.

0 commit comments

Comments
 (0)