Skip to content

Commit 94d9f71

Browse files
daveoconnordaveoconnor
committed
Added nix for developer setup (#1379)
1 parent b74c1c9 commit 94d9f71

File tree

7 files changed

+349
-11
lines changed

7 files changed

+349
-11
lines changed

‎.envrc

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# we have checks to account for first time setup
2+
if [ -f ".env" ]; then
3+
dotenv
4+
fi
5+
if [ -f "$HOME/.config/nix/nix.conf" ]; then
6+
echo "Entering nix shell environment."
7+
use flake
8+
fi

‎.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,7 @@ media
147147
python.log
148148

149149
# direnv
150-
.envrc
150+
.direnv
151151
.local_env
152152

153153
.DS_Store

‎docs/nix_based_setup.md

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Boost.org Website
2+
3+
## Overview
4+
5+
A Django based website that will power a new Boost website. See the [documentation](./docs/README.md) for more information about maintaining this project.
6+
7+
Links:
8+
9+
- https://www.stage.boost.cppalliance.org/ - staging
10+
- https://www.boost.org/ - production
11+
12+
---
13+
14+
## Local Development Setup
15+
16+
This project uses Python 3.11, Docker, and Docker Compose.
17+
18+
This document describes how to set up a development environment using Nix, which is a package manager that allows for reproducible builds and development environments, like a better encapsulated declarative cross-platform Homebrew.
19+
20+
For a basic rundown on Nix, this video could be useful https://www.youtube.com/watch?v=yQwW8dkuHqw
21+
22+
1. Install the following according to the instructions for your platform if not already installed:
23+
1. Direnv - https://direnv.net/docs/installation.html (don't install OS packaged version, must be >= 2.35.0) and then configure your shell to add the hook as per the direnv docs.
24+
2. Docker Engine
25+
* Linux - https://docs.docker.com/engine/install/
26+
* MacOS - https://orbstack.dev/ or https://github.com/abiosoft/colima ?
27+
* Windows - ?
28+
3. Just - https://just.systems/man/en/packages.html
29+
4. Nix - https://nixos.org/download/ (multi-user, y to all options)
30+
2. Clone this https://github.com/boostorg/website-v2.git repository to your machine.
31+
3. cd into the repository directory.
32+
4. In a terminal run `just bootstrap` in the root of the checked out repository to install the necessary development dependencies and generate the .env file.
33+
* This will ask you to log in to your google account to allow permissions for the production database load later. Ask Sam for permissions on the database backup drive with your email address.
34+
5. Update the generated .env file with the necessary environment variables. Where you can't retrieve these yourself, you can ask someone for some in #boost-website on the slack server at https://ccplang.slack.com. The minimum that must be set is:
35+
* GITHUB_TOKEN - a personal access token for the GitHub API, from your profile
36+
* STATIC_CONTENT_AWS_ACCESS_KEY_ID - ask for this
37+
* STATIC_CONTENT_AWS_SECRET_ACCESS_KEY - ask for this
38+
5. Run `just setup` to build services, and build the JS and CSS assets. If docker fails with permissions errors, reboot your machine.
39+
6. Run `just load_production_data` to download live data from the backup server.
40+
7. Run `docker compose up` to start the server.
41+
42+
### Social Login with django-allauth
43+
44+
Follow these instructions to use the social logins through django-allauth on your local machine.
45+
46+
See https://testdriven.io/blog/django-social-auth/ for more information.
47+
48+
#### Github
49+
- Go to https://github.com/settings/applications/new and add a new OAuth application
50+
- Set `http://localhost:8000` as the Homepage URL
51+
- Set `http://localhost:8000/accounts/github/login/callback/` as the Callback URL
52+
- Click whether you want to enable the device flow
53+
- <img src="https://user-images.githubusercontent.com/2286304/252841283-9a846c68-46bb-4dac-8d1e-d35270c09f1b.png" alt="The GitHub screen that registers a new OAuth app" width="400">
54+
- On completion copy the Client ID and Client Secret to the `.env` file as values of `GITHUB_OAUTH_CLIENT_ID` and `GITHUB_OAUTH_CLIENT_SECRET`.
55+
- Run `direnv allow` and restart your docker containers.
56+
57+
Setup should be complete and you should be able to see an option to "Use Github" on the sign up page.
58+
59+
To test the flow including authorizing Github for the Boost account, log into your GitHub account settings and click **Applications** in the left menu. Find the "Boost" authorization and delete it. The next time you log into Boost with this GitHub account, you will have to re-authorize it.
60+
61+
<img src="https://user-images.githubusercontent.com/2286304/204642346-8b269aaf-4693-4351-9474-0a998b97689c.png" alt="The 'Authorized OAuth Apps' tab in your GitHub Applications" width="400">
62+
63+
This setup process is not something that can currently be automated through terraform because of a lack of relevant Github API endpoints to create Oauth credentials.
64+
65+
#### Google
66+
67+
More detailed instructions at:
68+
69+
https://docs.allauth.org/en/latest/socialaccount/providers/google.html
70+
71+
1. Update the `.env` file with values for:
72+
1. `TF_VAR_google_cloud_email` (the email address of your Google Cloud account)
73+
2. `TF_VAR_google_organization_domain` (usually the domain of your Google Cloud account, e.g. "boost.org" if you will be using a @boost.org email address)
74+
3. `TF_VAR_google_cloud_project_name` (optional, default: localboostdev) - needs to change if destroyed and a setup is needed within 30 days
75+
2. Run `just development-tofu-init` to initialize tofu.
76+
3. Run `just development-tofu-plan` to confirm the planned changes.
77+
4. Run `just development-tofu-apply` to apply the changes.
78+
5. Go to https://console.developers.google.com/
79+
1. Search for the newly created project, named "Boost Development" (ID: localboostdev by default).
80+
2. Type "credentials" in the search input at the top of the page.
81+
3. Select "Credentials" under "APIs & Services".
82+
1. Click "+ CREATE CREDENTIALS"
83+
2. Select "OAuth Client ID"
84+
3. Select Application Type: "Web application"
85+
4. Name: "Boost Development" (arbitrary)
86+
5. For "Authorized Javascript Origins" use:`http://localhost:8000`
87+
6. For "Authorized Redirect URIs" use:
88+
* `http://localhost:8000/accounts/google/login/callback/`
89+
* `http://localhost:8000/accounts/google/login/callback/?flowName=GeneralOAuthFlow`
90+
7. Save
91+
6. From the page that's displayed, update the `.env` file with values for the following:
92+
- `GOOGLE_OAUTH_CLIENT_ID` should be similar to "k235bn2b1l1(...)asdsk.apps.googleusercontent.com"
93+
- `GOOGLE_OAUTH_CLIENT_SECRET` should be similar to "LAJACO(...)KLAI612ANAD"
94+
7. Run `docker compose down && docker compose up` and restart your docker containers.
95+
96+
Point 5 above can not be automated through terraform because of a lack of relevant Google Cloud API endpoints to create Oauth credentials.
97+
98+
Setup should be complete and you should be able to see an option to "Use Google" on the sign up page.
99+
100+
#### Additional Notes on allauth login flows:
101+
**Working locally**: If you need to run through the login flows multiple times, create a superuser so you can log into the admin. Then, log into the admin and delete your "Social Account" from the admin. This will test a fresh connection to GitHub for your logged-in GitHub user.
102+
103+
## Syncing EmailData Locally (optional)
104+
105+
To work with mailinglist data locally, the django application expects to be
106+
able to query a copy of the hyperkitty database from HYPERKITTY_DATABASE_NAME.
107+
Then, `just manage sync_mailinglist_stats` management command can be run.
108+
109+
## Debugging
110+
For local development there is Django Debug Toolbar, and the option to set a debugger.
111+
112+
In your env:
113+
- Django Debug Toolbar, enabled by default, can be disabled by setting DEBUG_TOOLBAR=False
114+
- IDE Debugging, disabled by default, can be enabled by uncommenting `PYTHONBREAKPOINT` in your .env file.
115+
116+
### Set Up Pycharm
117+
You can set up your IDE with a new "Python Debug Server" configuration as:
118+
119+
<img src="images/pycharm_debugger_settings.png" alt="PyCharm Debugger Settings" width="400">
120+
121+
### Debugger Usage
122+
To use the debugger add `breakpoint()` on a line in the code before you want to start debugging and then add breakpoints by clicking on the gutter. The debugger will stop at these point, you can then step/inspect the variables.
123+
124+
125+
## Troubleshooting
126+
127+
### Docker
128+
Keep in mind if there are issues with docker that the host docker daemon on your machine and the docker daemon in the nix setup may not match. It's a good idea to keep both up to date.
129+
130+
### Direnv
131+
when you switch to the directory you may see direnv exporting a bunch of environment variables as below.
132+
133+
The installer configures direnv to suppress those but it's a recent configuration option, so may be worth checking for an update if you see them.
134+
135+
## Disk space
136+
Should you find you're running short on disk space, to delete previous versioned store data you can run `nix-collect-garbage -d`. Reentering the directory will then reinstall all the current dependencies again. It's probably a good idea to run that periodically.
137+
138+
```shell
139+
direnv: export +ALLOWED_HOSTS +AR +AS...
140+
```

‎flake.lock

Lines changed: 61 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎flake.nix

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
{
2+
description = "Boost.org development environment.";
3+
4+
inputs = {
5+
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
6+
flake-utils.url = "github:numtide/flake-utils";
7+
};
8+
9+
outputs = { self, nixpkgs, flake-utils, ... }@inputs:
10+
flake-utils.lib.eachDefaultSystem (system:
11+
let
12+
pkgs = import nixpkgs {
13+
inherit system;
14+
};
15+
# https://nixos.wiki/wiki/Google_Cloud_SDK
16+
gdk = pkgs.google-cloud-sdk.withExtraComponents( with pkgs.google-cloud-sdk.components; [
17+
gke-gcloud-auth-plugin
18+
]);
19+
# Install a Ruby gem from rubygems.org
20+
asciidoctorBoostGem = pkgs.stdenv.mkDerivation rec {
21+
pname = "asciidoctor-boost";
22+
version = "0.1.7";
23+
sha = "ce139448812a9848219ce4cdb521c83c16009406a9d16efbc90bb24e94a46c24";
24+
25+
src = pkgs.fetchurl {
26+
url = "https://rubygems.org/downloads/${pname}-${version}.gem";
27+
sha256 = "${sha}";
28+
};
29+
dontUnpack = true;
30+
nativeBuildInputs = [ pkgs.ruby ];
31+
buildPhase = "true"; # Nothing to compile.
32+
installPhase = ''
33+
# Create a temporary gem directory
34+
mkdir -p $out
35+
# Set GEM_HOME to install gems locally under $out.
36+
export GEM_HOME=$out
37+
# Install the gem into GEM_HOME.
38+
${pkgs.ruby}/bin/gem install ${src} --no-document --ignore-dependencies
39+
'';
40+
meta = {
41+
description = "Asciidoctor Boost Ruby Gem installed from rubygems.org";
42+
homepage = "https://rubygems.org/gems/asciidoctor-boost";
43+
license = "BSL-1.0";
44+
};
45+
};
46+
47+
in {
48+
devShells.default = pkgs.mkShell {
49+
buildInputs = with pkgs; [
50+
# general system
51+
# e.g. this could contain docker client if we wanted that to be consistent,
52+
# though we need the daemon on the host anyway so it's redundant
53+
# general project
54+
awscli
55+
gdk
56+
just
57+
opentofu
58+
# frontend
59+
nodejs_22 # matches Dockerfile, due for upgrade?
60+
yarn
61+
# backend
62+
asciidoctor
63+
asciidoctorBoostGem
64+
pre-commit
65+
python311 # matches Dockerfile, due for upgrade?
66+
python311.pkgs.black
67+
python311.pkgs.isort
68+
python311.pkgs.pip-tools
69+
];
70+
# Host system installation workflow goes into the bootstrap justfile target.
71+
# Project specific installation and execution workflow should go here.
72+
shellHook = ''
73+
if [ ! -f .git/hooks/pre-commit ]; then
74+
pre-commit install
75+
fi
76+
if [ ! -d .venv ]; then
77+
python3.11 -m venv .venv
78+
. .venv/bin/activate
79+
pip install -r requirements.txt -r requirements-dev.txt
80+
else
81+
. .venv/bin/activate
82+
fi
83+
if [ ! -f .env ]; then
84+
cp env.template .env
85+
echo ".env created, you should update its contents"
86+
fi
87+
# google cloud login
88+
gcloud auth list --format="value(account)" | grep -q . || {
89+
echo "Not logged in. Running gcloud auth login..."
90+
gcloud auth login
91+
}
92+
'';
93+
};
94+
}
95+
);
96+
}

‎justfile

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,43 @@ ENV_FILE := ".env"
1212
# ----
1313

1414
@bootstrap: ## installs/updates all dependencies
15-
#!/usr/bin/env bash
16-
set -euo pipefail
17-
if [ ! -f "{{ENV_FILE}}" ]; then
18-
echo "{{ENV_FILE}} created"
19-
cp env.template {{ENV_FILE}}
15+
command -v direnv >/dev/null 2>&1 || { echo >&2 "Direnv is required but not installed. see: https://direnv.net/docs/installation.html - Aborting."; exit 1; }
16+
command -v nix >/dev/null 2>&1 || { echo >&2 "Nix is required but not installed. see: https://nixos.org/download.html - Aborting."; exit 1; }
17+
command -v just >/dev/null 2>&1 || { echo >&2 "Just is required but not installed. see: https://just.systems/man/en/packages.html - Aborting."; exit 1; }
18+
command -v docker >/dev/null 2>&1 || { echo >&2 "Docker is required but not installed. see: docs for links - Aborting."; exit 1; }
19+
20+
shell_name=$(basename "$SHELL") && \
21+
echo $shell_name && \
22+
if [ "$shell_name" = "zsh" ] && command -v zsh >/dev/null; then \
23+
zsh -i -c 'echo ${precmd_functions} | grep -q _direnv_hook' || { echo "❌ direnv hook is NOT installed in Zsh"; exit 1; }; \
24+
elif ([ "$shell_name" = "pwsh" ] || [ "$shell_name" = "powershell" ]) && command -v "$shell_name" >/dev/null; then \
25+
"$shell_name" -NoProfile -Command '$function:prompt.ToString() | grep -q direnv' || { echo "❌ direnv hook is NOT installed in PowerShell"; exit 1; }; \
26+
else \
27+
echo "ℹ️ Unsupported shell for checking direnv hook: $shell_name. Ensure you have the direnv shell hook eval set up correctly if there are problems."; \
2028
fi
21-
docker compose --file {{COMPOSE_FILE}} build --force-rm
29+
30+
if [ ! -d $HOME/.config/direnv/direnv.toml ]; then \
31+
mkdir -p $HOME/.config/direnv; \
32+
printf "[global]\nhide_env_diff = true\nload_dotenv = true\n" > $HOME/.config/direnv/direnv.toml; \
33+
fi
34+
if [ ! -d $HOME/.config/nix ]; then \
35+
mkdir -p $HOME/.config/nix; \
36+
printf "experimental-features = nix-command flakes\n" > $HOME/.config/nix/nix.conf; \
37+
fi
38+
# check if the docker group exists, create if not
39+
if [ ! $(getent group docker) ]; then \
40+
echo "ℹ️ Adding docker group..."; \
41+
sudo groupadd docker; \
42+
fi
43+
44+
# check if user is in docker group, add if not
45+
if [ $(id -Gn | grep -c docker) -eq 0 ]; then \
46+
echo "ℹ️ Adding docker group"; \
47+
sudo usermod -aG docker $USER; \
48+
echo "ℹ️ Added docker user. Please close the shell and open a new one."; \
49+
fi
50+
echo "Bootstrapping complete, update your .env and run 'just setup'"
51+
echo "If you have issues with docker permissions running just setup try restarting your machine."
2252

2353
@rebuild: ## rebuilds containers
2454
docker compose kill
@@ -31,7 +61,7 @@ ENV_FILE := ".env"
3161

3262
@build: ## builds containers
3363
docker compose pull
34-
DOCKER_BUILDKIT=1 docker compose build
64+
docker compose build
3565

3666
@cibuild: ## invoked by continuous integration servers to run tests
3767
python -m pytest
@@ -49,6 +79,8 @@ alias shell := console
4979
@setup: ## sets up a project to be used for the first time
5080
docker compose --file {{COMPOSE_FILE}} build --force-rm
5181
docker compose --file docker-compose.yml run --rm web python manage.py migrate --noinput
82+
npm install
83+
npm run build
5284

5385
@test_pytest: ## runs pytest
5486
-docker compose run --rm -e DEBUG_TOOLBAR="False" web pytest -s --create-db

0 commit comments

Comments
 (0)