Ease development with Docker

README.md

@@ -18,103 +18,17 @@ If you are a PG&E customer you can link your account now!  If you are not a PG&E
 
 # Development
 
-## Environment Setup (Ubuntu 20.04)
+## Environment Setup (Docker)
 
-Process notes here: https://github.com/JPHutchins/open-energy-view/issues/31
+You can get an environment setup easily using Docker. Make sure you have Docker and Docker Compose installed.
 
-The following notes are for setting up the environment with a Windows 10 host and Ubuntu 20.04 on WSL2.  Please submit a PR if you find necessary adaptations on your environment.
-
-Personally I use VSCode from the Windows host utilizing the "Remote - SSH" and "Remote - WSL" extensions.
-
-### Clone this repository
 ```
 git clone [email protected]:JPHutchins/open-energy-view.git
 cd open-energy-view
+docker-compose up
 ```
-### Install backend dependencies
-* **Install python requirements**
-
-  Note: check your python3 version
-  ```
-  sudo apt install python3.8-venv build-essential python3-dev
-  ```
-* **Create the virtual environment and install packages**
-  ```
-  python3 -m venv venv
-  source venv/bin/activate
-  pip3 install -r requirements.txt
-  ```
-* **Install and configure rabbitmq**
-  * Install erlang:
-    ```
-    sudo apt update
-    sudo apt install software-properties-common apt-transport-https
-    wget -O- https://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc | sudo apt-key add -
-    echo "deb https://packages.erlang-solutions.com/ubuntu focal contrib" | sudo tee /etc/apt/sources.list.d/rabbitmq.list
-    sudo apt update
-    sudo apt install erlang
-    ```
-  * Install rabbitmq
-    ```
-    curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.deb.sh | sudo bash
-    sudo apt install rabbitmq-server
-  * Start rabbitmq-server
-    ```
-    sudo service rabbitmq-server start
-    ```
-  * Verify that rabbitmq-server is running
-    ```
-    sudo service rabbitmq-server status
-    ```
-  * Configure rabbitmq
-    ```
-    sudo rabbitmqctl add_user jp admin
-    sudo rabbitmqctl set_user_tags jp administrator
-    sudo rabbitmqctl add_vhost myvhost
-    sudo rabbitmqctl set_permissions -p myvhost jp ".*" ".*" ".*"
-    ```
-
-### Install frontend dependencies and build
-* **Install nvm** (if you don't have it)
-
-  notes: https://github.com/nvm-sh/nvm
-  ```
-  curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
-  ```
-* **Install npm**
-  ```
-  nvm install 10.19.0
-  ```
-* **Install frontend packages**
-  ```
-  cd open_energy_view/frontend
-  nvm use 10
-  npm install
-  ```
-* **Build frontend**
-  
-  Assumes you are at path: `*/open-energy-view/open_energy_view/frontend`
-  ```
-  nvm use 10
-  npm run build
-  ```
-
-### Run the development server and workers
-* **Start the server and workers**
-  * Open four terminals (example from VSCode)
-
-  ![Four-Terminals](/docs/four-terminals.png)
-  * First terminal: `./run-wsgi-dev`
-  * Second terminal: `./run-io-worker`
-  * Third terminal: `./run-cpu-worker`
-* **Open the development site in a browser**
-  * Fourth terminal: `ip a`
-  * Note the IP address of your WSL2 instance, in this case `172.31.30.203`
-
-  ![ip-a](/docs/ip-a.png)
-  * On your host OS, open a Chrome or Firefox web browser and navigate to `http://<YOUR_WSL2_IP>:5000`
-
-  ![browser-address](/docs/browser-address.png)
+
+On your host OS, open a Chrome or Firefox web browser and navigate to `http://localhost:5000`
 
 ### Example account setup
 For first time setup you must register a user to your local database. Use something easily memorable and keep in mind that you can register as many users as you need while testing.

docker-compose.yml

@@ -0,0 +1,103 @@
+version: "3.9"
+services:
+  rabbitmq:
+    container_name: oev-rabbit
+    image: rabbitmq:3.11-alpine
+    ports:
+      - "5672:5672"
+      - "15672:15672"
+    hostname: "rabbitmq" # Important for how persistence works.
+    volumes:
+      - oev-rabbit:/var/lib/rabbitmq
+  rabbitmqsetup:
+    image: rabbitmq:3.11-alpine
+    restart: "no"
+    entrypoint: ["bash", "-c", "sleep 10 && rabbitmqctl -n rabbit@rabbitmq add_user jp admin && rabbitmqctl -n rabbit@rabbitmq set_user_tags jp administrator && rabbitmqctl -n rabbit@rabbitmq add_vhost myvhost && rabbitmqctl -n rabbit@rabbitmq set_permissions -p myvhost jp \".*\" \".*\" \".*\""]
+    volumes:
+      - oev-rabbit:/var/lib/rabbitmq
+    depends_on:
+      - rabbitmq # Make use of the cookie in the main container so we can run rabbitmqctl commands.
+  app:
+    container_name: oev
+    build:
+      context: ./
+      dockerfile: ./docker/Dockerfile-app
+    ports:
+      - "5000:5000"
+    environment:
+      - FLASK_CONFIG=config.DevConfig
+      - FLASK_APP=open_energy_view
+      - FLASK_ENV=production
+      - SECRET_KEY=dev
+      - JWT_SECRET_KEY=dev
+      - JWT_BLACKLIST_ENABLED=True
+      - JWT_TOKEN_LOCATION=cookies
+      - JWT_ACCESS_COOKIE_PATH=/
+      - JWT_REFRESH_COOKIE_PATH=/api/web/token/refresh
+      - JWT_COOKIE_CSRF_PROTECT=True
+      - PROD_JWT_COOKIE_SECURE=True
+      - DEV_JWT_COOKIE_SECURE=False
+      - PROD_DATABASE_URI=sqlite:///../test/data/energy_history_test.db
+      - DEV_DATABASE_URI=sqlite:///../test/data/energy_history_test.db
+      - PGE_CLIENT_ID=client_id
+      - PGE_CLIENT_SECRET=client_secret
+      - GOOGLE_CLIENT_ID=GOOGLE_CLIENT_ID
+      - GOOGLE_CLIENT_SECRET=GOOGLE_CLIENT_SECRET
+      - OAUTHLIB_INSECURE_TRANSPORT=True
+      - CERT_PATH=test/cert/cert.crt
+      - KEY_PATH=test/cert/private.key
+      - API_RESPONSE_KEY=xS5MqJ6N9CyH-hvqAGrmBVAxFMOyauMpdrdqCZa1eqo=
+      - PORT=5000
+    depends_on:
+      - rabbitmqsetup
+  cpu-worker:
+    restart: unless-stopped
+    container_name: oev-cpu-worker
+    build:
+      context: ./
+      dockerfile: ./docker/Dockerfile-worker
+    environment:
+      - HOSTNAME=celery.cpus@%h
+      - QUEUES=cpu
+      - CONCURRENCY=1
+      - POOL=prefork
+      - FLASK_CONFIG=config.DevConfig
+      - FLASK_APP=open_energy_view
+      - FLASK_ENV=production
+      - SECRET_KEY=dev
+      - JWT_SECRET_KEY=dev
+      - JWT_BLACKLIST_ENABLED=True
+      - JWT_TOKEN_LOCATION=cookies
+      - JWT_ACCESS_COOKIE_PATH=/
+      - JWT_REFRESH_COOKIE_PATH=/api/web/token/refresh
+      - JWT_COOKIE_CSRF_PROTECT=True
+      - PROD_JWT_COOKIE_SECURE=True
+      - DEV_JWT_COOKIE_SECURE=False
+      - PROD_DATABASE_URI=sqlite:///../test/data/energy_history_test.db
+      - DEV_DATABASE_URI=sqlite:///../test/data/energy_history_test.db
+      - PGE_CLIENT_ID=client_id
+      - PGE_CLIENT_SECRET=client_secret
+      - GOOGLE_CLIENT_ID=GOOGLE_CLIENT_ID
+      - GOOGLE_CLIENT_SECRET=GOOGLE_CLIENT_SECRET
+      - OAUTHLIB_INSECURE_TRANSPORT=True
+      - CERT_PATH=test/cert/cert.crt
+      - KEY_PATH=test/cert/private.key
+      - API_RESPONSE_KEY=xS5MqJ6N9CyH-hvqAGrmBVAxFMOyauMpdrdqCZa1eqo=
+      - PORT=5000
+    depends_on:
+      - app
+  io-worker:
+    restart: unless-stopped
+    container_name: oev-io-worker
+    build:
+      context: ./
+      dockerfile: ./docker/Dockerfile-worker
+    environment:
+      - HOSTNAME=celery.io@%h
+      - QUEUES=io
+      - CONCURRENCY=500
+      - POOL=gevent
+    depends_on:
+      - app
+volumes:
+  oev-rabbit:

docker/Dockerfile-app

@@ -0,0 +1,14 @@
+FROM node:10.19.0-slim AS ui
+ADD open_energy_view/frontend /frontend
+WORKDIR /frontend
+RUN npm ci && npm run build
+
+FROM python:3.8-slim
+ADD . /app
+WORKDIR /app
+RUN apt update && apt install -y build-essential
+RUN pip3 install -r ./requirements.txt
+RUN apt purge -y build-essential
+COPY --from=ui /frontend/dist /app/open_energy_view/frontend/dist
+COPY --from=ui /frontend/node_modules /app/open_energy_view/frontend/node_modules
+ENTRYPOINT uwsgi --http 0.0.0.0:"$PORT" --gevent 100 --wsgi-file wsgi.py --callable app

docker/Dockerfile-worker

@@ -0,0 +1,2 @@
+FROM open-energy-view_app
+ENTRYPOINT celery worker --app open_energy_view.celery_tasks --hostname="$HOSTNAME" --queues="$QUEUES" --loglevel=info --pool="$POOL" --concurrency="$CONCURRENCY"

open_energy_view/celery.py

@@ -2,7 +2,7 @@
 
 
 celery = Celery(
-    "tasks", backend="rpc://", broker="amqp://jp:admin@localhost:5672/myvhost",
+    "tasks", backend="rpc://", broker="amqp://jp:admin@rabbitmq:5672/myvhost",
 )
 
 

open_energy_view/celery_tasks.py

@@ -98,7 +98,7 @@ def add(x, y):
 @celery.task(bind=True, name="get_jp")
 def get_jp(self):
     response = requests.get(
-        "http://slowwly.robertomurray.co.uk/delay/2500/url/https://www.jphutchins.com"
+        "https://httpstat.us/200?sleep=2500"
     )
     return "BOOM"
 
@@ -135,25 +135,25 @@ def fetch_task(self, published_period_start, interval_block_url, headers, cert):
 @celery.task(bind=True, name="fake_fetch")
 def fake_fetch(self):
     test_xml = [
-        "/home/jp/open-energy-view/test/data/espi/espi_2_years.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-16.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-17.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-18.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-19.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-20.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-21.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-22.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-23.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-24.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-25.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-26.xml",
-        "/home/jp/open-energy-view/test/data/espi/Single Days/2019-10-27.xml",
+        "/app/test/data/espi/espi_2_years.xml",
+        "/app/test/data/espi/Single Days/2019-10-16.xml",
+        "/app/test/data/espi/Single Days/2019-10-17.xml",
+        "/app/test/data/espi/Single Days/2019-10-18.xml",
+        "/app/test/data/espi/Single Days/2019-10-19.xml",
+        "/app/test/data/espi/Single Days/2019-10-20.xml",
+        "/app/test/data/espi/Single Days/2019-10-21.xml",
+        "/app/test/data/espi/Single Days/2019-10-22.xml",
+        "/app/test/data/espi/Single Days/2019-10-23.xml",
+        "/app/test/data/espi/Single Days/2019-10-24.xml",
+        "/app/test/data/espi/Single Days/2019-10-25.xml",
+        "/app/test/data/espi/Single Days/2019-10-26.xml",
+        "/app/test/data/espi/Single Days/2019-10-27.xml",
     ]
     test_xml.reverse()
 
     for xml_path in test_xml:
         response = requests.get(
-            "http://slowwly.robertomurray.co.uk/delay/2500/url/https://www.jphutchins.com"
+            "https://httpstat.us/200?sleep=2500"
         )
         with open(xml_path) as xml_reader:
             xml = xml_reader.read()

open_energy_view/resources.py

@@ -35,7 +35,7 @@
 from .celery_tasks import get_jp, insert_espi_xml_into_db, process_data
 from .celery import celery
 
-IP_AND_PORT = os.environ.get("IP_AND_PORT")
+PORT = os.environ.get("PORT")
 
 PGE_BULK_ID = 51070
 PGE_CLIENT_ID = os.environ.get("PGE_CLIENT_ID")
@@ -428,8 +428,7 @@ def get(self):
 
 class FakeOAuthStart(Resource):
     def get(self):
-        # TODO: get host IP on Dev mode - my WSL is not working on localhost...
-        return redirect(f"http://{IP_AND_PORT}/#/fake_oauth")
+        return redirect(f"http://127.0.0.1:{PORT}/#/fake_oauth")
 
 
 class AddFakeSourceFromFakeOAuthOLD(Resource):