WIP feat(docker-local): improve local docker configuration for quickstart

CONTRIBUTING.md

@@ -22,29 +22,29 @@ First, the following tools and frameworks need to be installed on your system pr
 Second, fork the repository, and then run the following commands to clone the repository locally.
 
 ```
-$ git clone https://github.com/{your-account}/rocketnotes.git
-$ cd rocketnotes
+git clone https://github.com/{your-account}/rocketnotes.git
+cd rocketnotes
 ```
 
-#### 1. Start DynamoDB and docker network
+#### 1. Start DynamoDB and S3 mock with docker
 ```
-$ docker-compose up -d
+docker-compose up dynamodb s3 -d
 ```
 #### 2. Init tables and create data for default user
 ```
-$ sh ./dynamodb-init.sh
+sh ./dynamodb-init.sh
 ```
 #### 3. Build and start Lambda functions with AWS SAM
 ```
-$ sam build
-$ sam local start-api --docker-network rocketnotes_local-serverless-network --warm-containers EAGER 
+sam build
+sam local start-api --docker-network rocketnotes_local-serverless-network --warm-containers EAGER 
 ```
 #### 4. Start Angular app
-```
-$ export BASE_API_URL="http://localhost:3000"
-$ cd webapp
-$ npm install
-$ npm run start
+```bash
+export BASE_API_URL="http://localhost:3000"
+cd webapp
+npm install
+npm run start
 ```
 Open http://localhost:4200 in your browser and you should see the Rocketnotes webapp with the cheat sheet of the default user displayed.
 

Dockerfile.sam

@@ -0,0 +1,11 @@
+FROM public.ecr.aws/sam/build-provided.al2:latest
+
+WORKDIR /var/task
+
+COPY template.yaml .
+COPY go.mod .
+COPY go.sum .
+COPY lambda-handler ./lambda-handler
+COPY .aws-sam ./.aws-sam
+
+EXPOSE 3002

INSTALLATION.md

@@ -7,15 +7,11 @@ The following tools need to be installed on your system prior to build and deplo
 - [Node.js >= 14.x](https://nodejs.org/download/release/latest-v14.x/)
 - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
 
-
-First fork the repository, and then run the following commands to clone the repository locally.
-
 ```
-$ git clone https://github.com/{your-account}/rocketnotes.git
-$ cd rocketnotes
-$ npm install
+git clone https://github.com/fynnfluegge/rocketnotes.git
+cd rocketnotes
+npm install
 ```
-
 </br>
 
 ## AWS hosting
@@ -38,51 +34,62 @@ How to create a hosted zone and configure Route 53 as a DNS service with your do
 
 ### Deploy all AWS resources
 The following environment variables are required for the deployment:
-```console
-$ export AWS_ACCOUNT="<YOUR_AWS_ACCOUNT_ID>"
-$ export AWS_REGION="<YOUR_AWS_REGION>"
-$ export COGNITO_USER_POOL_ID="<YOUR_COGNITO_USER_POOL_ID>"
-$ export COGNITO_APP_CLIENT_ID="<YOUR_COGNITO_APP_CLIENT_ID>"
-$ export DOMAIN_NAME="<YOUR_DOMAIN_NAME>"
-$ export DOMAIN="<YOUR_DOMAIN>"
-$ export SUBDOMAIN="<YOUR_SUBDOMAIN>" # <- use "www" here if you don't have a subdomain configured in your hosted zone (e.g app)
+```bash
+export AWS_ACCOUNT="<YOUR_AWS_ACCOUNT_ID>"
+export AWS_REGION="<YOUR_AWS_REGION>"
+export COGNITO_USER_POOL_ID="<YOUR_COGNITO_USER_POOL_ID>"
+export COGNITO_APP_CLIENT_ID="<YOUR_COGNITO_APP_CLIENT_ID>"
+export DOMAIN_NAME="<YOUR_DOMAIN_NAME>"
+export DOMAIN="<YOUR_DOMAIN>"
+export SUBDOMAIN="<YOUR_SUBDOMAIN>" # <- use "www" here if you don't have a subdomain configured in your hosted zone (e.g app)
 ```
 > **_NOTE:_** <YOUR_DOMAIN_NAME> is your domain **without** extension like ".com" while <YOUR_DOMAIN> is your domain **with** extension in this context.
 
 Once your environment variables are specified run:
 ```
-$ cd cdk
-$ cdk deploy
+cd cdk
+cdk deploy
 ```
 
 The first deployment will take some minutes, since all the resources and lambda functions need to be initially created. If the deployment was successfull the api url should be logged in the console as `HTTP API endpoint URL`.
 
 ### Build webapp
 The Angular webapp need to be bundled in production mode.
 The following environment variables are required for the production build:
-```console
-$ export REDIRECT_SIGN_IN="<YOUR_DOMAIN_URL>"
-$ export REDIRECT_SIGN_OUT="<YOUR_DOMAIN_URL/logout>"
-$ export AUTH_GUARD_REDIRECT="<AUTH_GUARD_REDIRECT_URL>" # <- "https://<YOUR_DOMAIN_NAME>.auth.<AWS_REGION>.amazoncognito.com/login?response_type=code&client_id=<YOUR_COGNITO_APP_CLIENT_ID>&redirect_uri=https://<YOUR_SUBDOMAIN>.<YOUR_DOMAIN>"
-$ export API_URL="<YOUR_API_URL>" # <- HTTP API endpoint URL from deployment console log
+```bash
+export REDIRECT_SIGN_IN="<YOUR_DOMAIN_URL>"
+export REDIRECT_SIGN_OUT="<YOUR_DOMAIN_URL/logout>"
+export AUTH_GUARD_REDIRECT="<AUTH_GUARD_REDIRECT_URL>" # <- "https://<YOUR_DOMAIN_NAME>.auth.<AWS_REGION>.amazoncognito.com/login?response_type=code&client_id=<YOUR_COGNITO_APP_CLIENT_ID>&redirect_uri=https://<YOUR_SUBDOMAIN>.<YOUR_DOMAIN>"
+export API_URL="<YOUR_API_URL>" # <- HTTP API endpoint URL from deployment console log
 ```
 Once your environment variables are specified run:
 ```
-$ cd webapp
-$ npm install
-$ npm run build
+cd webapp
+npm install
+npm run build
 ```
 
 ### Deploy webapp
 Finally, the Angular app can be deployed to S3 with again:
 ```
-$ cd cdk
-$ cdk deploy
+cd cdk
+cdk deploy
 ```
 This deployment will only deploy the webapp build to the S3 bucket and will be much faster than the previous one.
 
 </br>
 
 ## Run on your local machine with Docker
-> **_NOTE:_** This is under development in [#PR46](https://github.com/fynnfluegge/rocketnotes/pull/46). As a workaround you can visit [Getting started](CONTRIBUTING.md#getting-started) to run locally in dev mode.
-
+```
+cd rocketnotes
+docker-compuse up -d
+```
+Docker-compose will create and start four containers with a docker network:
+- the DynamoDB with a volume listening on port 8041
+- the S3 mock listening on port 9091
+- the Angular app listening on port 3001
+- all the lambda functions in a single container listening on port 3002  
+
+On initial startup it may take a moment.
+Once it's done execute `sh ./dynamodb-init.sh` as a last step to initialize the dynamodb.
+Now you can open `http://localhost:3001` in the browser and you should see the initially created Cheat Sheet document 🚀

cdk/rocketnotes.go

@@ -258,7 +258,7 @@ func RocketnotesStack(scope constructs.Construct, id string, props *RocketnotesS
 	awscdklambdagoalpha.NewGoFunction(stack, jsii.String("POST-Document"), &awscdklambdagoalpha.GoFunctionProps{
 		FunctionName: jsii.String("POST-Document"),
 		Runtime:      awslambda.Runtime_PROVIDED_AL2(),
-		Entry:        jsii.String("../lambda-handler/save-document-event-handler"),
+		Entry:        jsii.String("../lambda-handler/save-document-handler"),
 		Events: &[]awslambda.IEventSource{
 			awslambdaeventsources.NewSqsEventSource(queue, &awslambdaeventsources.SqsEventSourceProps{
 				BatchSize: jsii.Number(1),
@@ -346,7 +346,7 @@ func RocketnotesStack(scope constructs.Construct, id string, props *RocketnotesS
 				BatchSize: jsii.Number(1),
 			}),
 		},
-		Environment: &map[string]*string{"bucketName": bucket.BucketName()},
+		Environment: &map[string]*string{"BUCKET_NAME": bucket.BucketName()},
 		Role:        lambdaS3SqsDynamoDbRole,
 		MemorySize:  jsii.Number(1024),
 		Timeout:     awscdk.Duration_Millis(jsii.Number(900000)),
@@ -359,7 +359,7 @@ func RocketnotesStack(scope constructs.Construct, id string, props *RocketnotesS
 		Runtime:      awslambda.Runtime_PYTHON_3_9(),
 		Entry:        jsii.String("../lambda-handler/semantic-search-handler"),
 		Index:        aws.String("main.py"),
-		Environment: &map[string]*string{"bucketName": bucket.BucketName()},
+		Environment: &map[string]*string{"BUCKET_NAME": bucket.BucketName()},
 		Role:        lambdaS3Role,
 		MemorySize:  jsii.Number(1024),
 		Timeout:     awscdk.Duration_Millis(jsii.Number(900000)),
@@ -379,7 +379,7 @@ func RocketnotesStack(scope constructs.Construct, id string, props *RocketnotesS
 		Runtime:      awslambda.Runtime_PYTHON_3_9(),
 		Entry:        jsii.String("../lambda-handler/chat-handler"),
 		Index:        aws.String("main.py"),
-		Environment: &map[string]*string{"bucketName": bucket.BucketName()},
+		Environment: &map[string]*string{"BUCKET_NAME": bucket.BucketName()},
 		Role:        lambdaS3Role,
 		MemorySize:  jsii.Number(1024),
 		Timeout:     awscdk.Duration_Millis(jsii.Number(900000)),

docker-compose.yaml

@@ -1,8 +1,10 @@
 services:
   dynamodb:
-    build: .
+    build:
+      context: .
+      dockerfile: Dockerfile.dynamodb
     hostname: dynamodb-local
-    container_name: dynamodb-local
+    container_name: rocketnotes-dynamodb
     command: -jar DynamoDBLocal.jar -sharedDb -dbPath /home/dynamodblocal/data/
     ports:
       - "8041:8000"
@@ -11,10 +13,45 @@ services:
     networks:
       - local-serverless-network
 
+  s3:
+    image: adobe/s3mock:latest
+    container_name: rocketnotes-s3
+    environment:
+      - initialBuckets=faissIndexBucket
+    ports:
+      - 9091:9090
+    networks:
+      - local-serverless-network
+
+  api:
+    build:
+      context: .
+      dockerfile: Dockerfile.sam
+    command:
+      bash -c "sam local start-api --host 0.0.0.0 --container-host host.docker.internal --port 3002 --docker-volume-basedir ${PWD}/.aws-sam/build --debug
+      --docker-network rocketnotes_local-serverless-network"
+    container_name: rocketnotes-api
+    mem_limit: "1g"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ${PWD}:${PWD}:ro
+    platform: linux/amd64
+    ports:
+      - "3002:3002"
+
+  webapp:
+    build: webapp/.
+    container_name: rocketnotes-webapp
+    platform: linux/amd64
+    ports:
+      - "3001:3001"
+    environment:
+      API_URL: "http://localhost:3002"
+
 volumes:
   dynamodb-volume:
-   driver: local
+    driver: local
 
 networks:
-   local-serverless-network:
-     driver: "bridge"
+  local-serverless-network:
+    driver: "bridge"

dynamodb-init.sh

@@ -11,12 +11,27 @@ aws dynamodb create-table --endpoint-url http://localhost:8041 --table-name tnn-
 --provisioned-throughput ReadCapacityUnits=2,WriteCapacityUnits=2 \
 > /dev/null 2>&1
 
+
+aws dynamodb update-table \
+--endpoint-url http://localhost:8041 \
+--table-name tnn-Documents \
+--attribute-definitions AttributeName=userId,AttributeType=S \
+--global-secondary-index-updates \
+"[{\"Create\":{\"IndexName\":\"userId-index\",\"KeySchema\":[{\"AttributeName\":\"userId\",\"KeyType\":\"HASH\"}],\"Projection\":{\"ProjectionType\":\"ALL\"},\"ProvisionedThroughput\":{\"ReadCapacityUnits\":1,\"WriteCapacityUnits\":1}}}]" \
+> /dev/null 2>&1
+
+aws dynamodb create-table --endpoint-url http://localhost:8041 --table-name tnn-Vectors \
+--attribute-definitions AttributeName=id,AttributeType=S \
+--key-schema AttributeName=id,KeyType=HASH \
+--provisioned-throughput ReadCapacityUnits=2,WriteCapacityUnits=2 \
+> /dev/null 2>&1
+
 # create document tree
 aws dynamodb put-item --endpoint-url http://localhost:8041 --table-name tnn-Tree \
 --item '{"id": {"S": "4afe1f16-add0-11ed-afa1-0242ac120002"},"documents": {"L": [{"M": {"id": {"S": "5b6ae09e-c32a-45ee-bb3b-1c65fc943a9c"},"children": {"NULL": true},"name": {"S": "Cheat Sheet"},"parent": {"S": "root"},"pinned": {"BOOL": false}}}]},"pinned": {"NULL": true},"trash": {"NULL": true}}' \
 > /dev/null 2>&1
 
 # create document
 aws dynamodb put-item --endpoint-url http://localhost:8041 --table-name tnn-Documents \
---item '{"id": {"S": "5b6ae09e-c32a-45ee-bb3b-1c65fc943a9c"},"content": {"S": "<p align=center>\nThis document provides a general introduction to Markdown syntax and some extensive features, and is intended to serve as a handy cheat sheet for you.\n</p>\n\n# H1\n## H2\n### H3  \n**bold text**  \n*italicized text*  \n~~The world is flat.~~\n\n1. First item\n2. Second item\n3. Third item\n- First item\n- Second item\n- Third item\n\n[some_link](https://www.example.com)\n\n![alt text](https://picsum.photos/200)\n\n---\n\n| Syntax | Description |\n| ----------- | ----------- |\n| Header | Title |\n| Paragraph | Text |\n\n---\n\n\n```\n$ docker run -t -i --rm ubuntu bash\n```\n\n```javascript\nvar doSomeStuff = function(data) {\n     var searchForm = $(#some_form);\n     var searchParms = [];\n\n     for (var i = 0; i < data; i++) {\n         searchParms[i] = data[i];\n     }\n};\n```\n\n---\n"},"lastModified": {"NULL": true},"parentId": {"NULL": true},"title": {"S": "Cheat Sheet"},"userId": {"NULL": true}}' \
+--item '{"id": {"S": "5b6ae09e-c32a-45ee-bb3b-1c65fc943a9c"},"content": {"S": "<p align=center>\nThis document provides a general introduction to Markdown syntax and some extensive features, and is intended to serve as a handy cheat sheet for you.\n</p>\n\n# H1\n## H2\n### H3  \n**bold text**  \n*italicized text*  \n~~The world is flat.~~\n\n1. First item\n2. Second item\n3. Third item\n- First item\n- Second item\n- Third item\n\n[some_link](https://www.example.com)\n\n![alt text](https://picsum.photos/200)\n\n---\n\n| Syntax | Description |\n| ----------- | ----------- |\n| Header | Title |\n| Paragraph | Text |\n\n---\n\n\n```\n$ docker run -t -i --rm ubuntu bash\n```\n\n```javascript\nvar doSomeStuff = function(data) {\n     var searchForm = $(#some_form);\n     var searchParms = [];\n\n     for (var i = 0; i < data; i++) {\n         searchParms[i] = data[i];\n     }\n};\n```\n\n---\n"},"lastModified": {"NULL": true},"parentId": {"NULL": true},"title": {"S": "Cheat Sheet"},"userId": {"S": "NULL"}}' \
 > /dev/null 2>&1

lambda-handler/chat-handler/main.py

@@ -8,10 +8,16 @@
 from langchain_community.vectorstores import FAISS
 from langchain_openai import ChatOpenAI, OpenAIEmbeddings
 
-s3 = boto3.client("s3")
+is_local = os.environ.get("LOCAL", False)
+s3_args = {}
+
+if is_local:
+    s3_args["endpoint_url"] = "http://s3:9090"
+
+s3 = boto3.client("s3", **s3_args)
 
 documents_table_name = "tnn-Documents"
-bucket_name = os.environ["bucketName"]
+bucket_name = os.environ["BUCKET_NAME"]
 
 
 def handler(event, context):

lambda-handler/hello-world-handler/local.go

@@ -0,0 +1,44 @@
+// This is a simple AWS Lambda function that returns a JSON response with a message "Hello World"
+// Used for integration testing with sam local and docker-compose
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+
+	"github.com/aws/aws-lambda-go/events"
+	"github.com/aws/aws-lambda-go/lambda"
+)
+
+type Response struct {
+  Message string `json:"message"`
+}
+
+func init() {
+}
+
+func handleRequest(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
+
+  response := Response{
+    Message: "Hello Wooooorld",
+  }
+
+	b, err := json.Marshal(response)
+	if err != nil {
+		fmt.Println(err)
+	}
+
+
+	return events.APIGatewayProxyResponse{
+		StatusCode: 200,
+		Headers: map[string]string{
+			"Access-Control-Allow-Origin": "*", // Required for CORS support to work locally
+		},
+		Body: string(b),
+	}, nil
+}
+
+func main() {
+	lambda.Start(handleRequest)
+}