add docker run for the entire app + fix env var and container naming

* add script to run docker containers under the same network and project

* add env var example for docker containers

* update engine url env var name

* Update .env.example

* Update .env.example

* Delete services/enterprise/.test.env

* Deleted database info

* final updates and fixes to env vars and docker compose local development

---------

Co-authored-by: Juan Valacco <[email protected]>
Co-authored-by: Dishen <[email protected]>
Co-authored-by: dishenwang2023 <[email protected]>

README.md

@@ -34,10 +34,26 @@ This repository contains four components under `/services` which can be used tog
 1. Engine: The core natural language-to-SQL engine. If you would like to use the dataherald API without users or authentication, running the engine will suffice.
 2. Enterprise: The application API layer which adds authentication, organizations and users, and other business logic to Dataherald. 
 3. Admin-console: The front-end component of Dataherald which allows a GUI for configuration and observability. You will need to run both engine and enterprise for the admin-console to work.
-4. Slack: A slackbot which allows users from a slack channel to interact with dataherald. Requires both engine and enterprise to run.
+4. Slackbot: A slackbot which allows users from a slack channel to interact with dataherald. Requires both engine and enterprise to run.
 
 For more information on each component, please take a look at their `README.md` files.
 
+## Running locally
+
+Each component in the `/services` directory has its own `docker-compose.yml` file. To set up the environment, follow these steps:
+
+1. **Set Environment Variables**:
+   Each service requires specific environment variables. Refer to the `.env.example` file in each service directory and create a `.env` file with the necessary values. 
+   > For the Next.js front-end app is `.env.local`
+2. **Run Services**:
+   You can run all the services using a single script located in the root directory. This script creates a common Docker network and runs each service in detached mode.
+
+Run the script to start all services:
+
+```bash
+sh docker-run.sh
+```
+
 ## Contributing
 As an open-source project in a rapidly developing field, we are open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
 

docker-run.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# Create Docker network
+docker network create dataherald_network
+
+# Bring up services with Docker Compose
+docker-compose -p dataherald -f services/engine/docker-compose.yml up --build -d
+docker-compose -p dataherald -f services/enterprise/docker-compose.yml up --build -d
+docker-compose -p dataherald -f services/slackbot/docker-compose.yml up --build -d
+docker-compose -p dataherald -f services/admin-console/docker-compose.yml up --build -d

services/admin-console/.env.example

@@ -1,21 +1,22 @@
 NODE_ENV="development" # development | production
 
 # API URL
-NEXT_PUBLIC_API_URL=
+NEXT_PUBLIC_API_URL='http://localhost:3001'
 
 # AUTH 0 CONFIG
 AUTH0_BASE_URL='http://localhost:3000'
 AUTH0_SCOPE='openid profile email offline_access'
 AUTH0_SECRET='use [openssl rand -hex 32] to generate a 32 bytes value'
-AUTH0_ISSUER_BASE_URL='https://{yourDomain}'
+AUTH0_ISSUER_BASE_URL='{yourIssuer}'
 AUTH0_API_AUDIENCE='{yourAudience}'
 AUTH0_CLIENT_ID='{yourClientId}'
 AUTH0_CLIENT_SECRET='{yourClientSecret}'
 
 # (OPTIONAL) Posthog Analytics 
+NEXT_PUBLIC_POSTHOG_DISABLED='true'
 NEXT_PUBLIC_POSTHOG_KEY=
 NEXT_PUBLIC_POSTHOG_HOST="https://app.posthog.com"
 
 
 # (OPTIONAL) STRIPE
-NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=
+NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=

services/admin-console/dev.Dockerfile

@@ -1,5 +1,7 @@
 FROM node:18-alpine
 
+LABEL Author="Juan Sebastian Valacco"
+
 WORKDIR /app
 
 # Install dependencies based on the preferred package manager
@@ -12,6 +14,10 @@ COPY . .
 # Uncomment the following line to disable telemetry at run time
 ENV NEXT_TELEMETRY_DISABLED 1
 
+# Docker network URL for the API -- used for nextjs server side API calls inside the docker network
+# The browser needs to access the API from the exposed port in the docker host (i.e.: localhost:3001)
+ENV DOCKER_API_URL='http://api:3001' 
+
 # Note: Don't expose ports here, Compose will handle that for us
 
 # Start Next.js in development mode based on the preferred package manager

services/admin-console/docker-compose.yml

@@ -1,6 +1,6 @@
 services:
   next-app:
-    container_name: next-app
+    container_name: console
     build:
       context: .
       dockerfile: dev.Dockerfile
@@ -9,6 +9,11 @@ services:
     volumes:
       - ./src:/app/src
       - ./public:/app/public
-    restart: always
     ports:
       - 3000:3000
+    restart: always
+    networks:
+      - dataherald_network
+networks:
+  dataherald_network:
+    external: true

services/admin-console/prod.Dockerfile

@@ -1,5 +1,7 @@
 FROM --platform=linux/amd64 node:18-alpine AS base
 
+LABEL Author="Juan Sebastian Valacco"
+
 # Install dependencies only when needed
 FROM base AS deps
 # Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.

services/admin-console/src/config.ts

@@ -1,4 +1,5 @@
-export const API_URL = process.env.NEXT_PUBLIC_API_URL
+const isServer = typeof window === 'undefined';
+export const API_URL = isServer ? process.env.DOCKER_API_URL || process.env.NEXT_PUBLIC_API_URL : process.env.NEXT_PUBLIC_API_URL;
 export const AUTH = {
   hostname: process.env.AUTH0_BASE_URL,
   cliendId: process.env.AUTH0_CLIENT_ID,

services/engine/README.md

@@ -123,7 +123,7 @@ Fernet.generate_key()
 >We need to set it up externally to enable external clients running on docker to communicate with this app. 
 Run the following command:
 ```
-docker network create backendnetwork
+docker network create dataherald_network
 ```
 
 4. Build docker images, create containers and raise them. This will raise the app and mongo container
@@ -140,7 +140,7 @@ It should look like this:
 ```
 CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS         PORTS                      NAMES
 72aa8df0d589   dataherald-app   "uvicorn dataherald.…"   7 seconds ago   Up 6 seconds   0.0.0.0:80->80/tcp         dataherald-app-1
-6595d145b0d7   mongo:latest     "docker-entrypoint.s…"   19 hours ago    Up 6 seconds   0.0.0.0:27017->27017/tcp   dataherald-mongodb-1
+6595d145b0d7   mongo:latest     "docker-entrypoint.s…"   19 hours ago    Up 6 seconds   0.0.0.0:27017->27017/tcp   mongodb
 ```
 
 6. In your browser visit [http://localhost/docs](http://localhost/docs)

services/engine/docker-compose.yml

@@ -1,6 +1,6 @@
-version: "3.9"
 services:
-  app:
+  engine:
+    container_name: engine
     build:
       context: .
       dockerfile: Dockerfile
@@ -16,9 +16,10 @@ services:
     depends_on:
       - mongodb
     networks:
-      - backendnetwork
+      - dataherald_network
     env_file: .env
   mongodb:
+    container_name: mongodb
     image: mongo:latest
     restart: always
     ports:
@@ -32,7 +33,7 @@ services:
       MONGO_INITDB_ROOT_PASSWORD: "${MONGODB_DB_PASSWORD}"
       MONGO_INITDB_DATABASE: "${MONGODB_DB_NAME}"
     networks:
-      - backendnetwork
+      - dataherald_network
 networks:
-  backendnetwork:
+  dataherald_network:
     external: true

services/engine/docs/quickstart.rst

@@ -46,7 +46,7 @@ Starting Docker
 
 #. Create a Docker network for communication between services (the Dataherald engine and Mongo) using the following command 
 
-    ```docker network create backendnetwork```
+    ```docker network create dataherald_network```
 
 
 #. Build the docker images, create containers and raise them by running
@@ -61,7 +61,7 @@ Starting Docker
         
         CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS         PORTS                      NAMES
         72aa8df0d589   dataherald-app   "uvicorn dataherald.…"   7 seconds ago   Up 6 seconds   0.0.0.0:80->80/tcp         dataherald-app-1
-        6595d145b0d7   mongo:latest     "docker-entrypoint.s…"   19 hours ago    Up 6 seconds   0.0.0.0:27017->27017/tcp   dataherald-mongodb-1
+        6595d145b0d7   mongo:latest     "docker-entrypoint.s…"   19 hours ago    Up 6 seconds   0.0.0.0:27017->27017/tcp   mongodb
 
 
 #. You can also verify the engine is running by navigating to the Swagger-UI from your browser at `<http://localhost/docs>`_

services/enterprise/.env.example

@@ -1,15 +1,15 @@
-K2_CORE_URL=http://localhost:3001/api/v1
+ENGINE_URL=http://{DOCKER_CONTAINER_NAME:PORT}/api/v1 # example: http://engine/api/v1 (port 80 by default)
 MONGODB_DB_NAME=
 MONGODB_URI=
 
-AUTH0_DOMAIN=
-AUTH0_ISSUER=
-AUTH0_API_AUDIENCE=
+AUTH0_DOMAIN='{yourDomain}'
+AUTH0_ISSUER='{yourIssuer}'
+AUTH0_API_AUDIENCE='{yourAudience}'
 AUTH0_DISABED=False
 
 API_KEY_SALT=
 
-DEFAULT_K2_TIMEOUT=120
+DEFAULT_ENGINE_TIMEOUT=120
 
 S3_AWS_ACCESS_KEY_ID=
 S3_AWS_SECRET_ACCESS_KEY=
@@ -19,7 +19,7 @@ ENCRYPT_KEY =
 
 POSTHOG_API_KEY=
 POSTHOG_HOST=
-POSTHOG_DISABLED=False
+POSTHOG_DISABLED=True
 
 # To set the SSH key file
 SSH_PRIVATE_KEY_PASSWORD=

services/enterprise/README.md

@@ -12,7 +12,7 @@ We use Docker to create the environment ai-server.
 >We need to set it up externally to enable external clients running on docker to communicate with this app. 
 Run the following command:
 ```
-docker network create backendnetwork
+docker network create dataherald_network
 ```
 >If you are running ai-engine locally on docker, you can find container's IP by running this command:
 ```
@@ -103,7 +103,7 @@ Follow these steps to set up and run the Enterprise and Engine containers for lo
 
 1. Create the docker network if it doesn't exist
 ```
-docker network create backendnetwork
+docker network create dataherald_network
 ```
 
 2. Set the envvars in the .env file for `server` and `dataherald` projects
@@ -114,9 +114,9 @@ cp .env.example .env
 3. Make sure you use the same `ENCRYPT_KEY` for `server` and `dataherald`.
 
 
-4. For `K2_CORE_URL` check that you use as host the service name that is specified in docker-composer file, for example:
+4. For `ENGINE_URL` check that you use as host the service name that is specified in docker-composer file, for example:
 ```
-K2_CORE_URL=http://app/api/v1
+ENGINE_URL=http://app/api/v1
 ```
 
 5. Run the containers and execute the initialization script to generate data. It should create a real db_connection, 

services/enterprise/config.py

@@ -26,8 +26,8 @@
 class Settings(BaseSettings):
     load_dotenv()
 
-    engine_url: str = os.environ.get("K2_CORE_URL")
-    default_engine_timeout: int = os.environ.get("DEFAULT_K2_TIMEOUT")
+    engine_url: str = os.environ.get("ENGINE_URL")
+    default_engine_timeout: int = os.environ.get("DEFAULT_ENGINE_TIMEOUT")
     encrypt_key: str = os.environ.get("ENCRYPT_KEY")
     api_key_salt: str = os.environ.get("API_KEY_SALT")