Impossible to run start-api and start-lambda in parallel on Linux

[fbeister]

Description

It seems to be impossible to run local start-api and local start-lambda in parallel in --docker-network host mode to achieve a scenario where we have an API Lambda chain-calling two other lambdas:

API ---(A)---> Lambda 1 ---(B)---> Lambda 2 ---(C)---> Lambda 3

• A) is covered by start-api itself
• B) is covered by running start-api with --docker-network host in order for it to see the local start-lambda on port 3001
• C) needs start-lambda to be run with --docker-network host, too but this fails with tcp :9001: bind: address already in use

Steps to reproduce

I use the following script to start my SAM environment:

JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/" sam local start-lambda --docker-network host 2>&1 | sed -e 's/^/[LAMBDA] /' &
JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/" sam local start-api --docker-network host 2>&1 | sed -e 's/^/[API] /'

Observed result

[API] Found one Lambda function with name 'LambdaTestingMultitest'
[API] Invoking lambdas.testing.multitest.TestingMultitest::handleRequest (java11)
[API] No environment variables found for function 'LambdaTestingMultitest'
[API] Environment variables overrides data is standard format
[API] Loading AWS credentials from session with profile 'None'
[API] Resolving code path. Cwd=/tmp/.aws-sam/build, CodeUri=LambdaTestingMultitest
[API] Resolved absolute path to code is /tmp/.aws-sam/build/LambdaTestingMultitest
[API] Code /tmp/.aws-sam/build/LambdaTestingMultitest is not a zip/jar file
[API] Skipping building an image since no layers were defined
[API] 
[API] Fetching lambci/lambda:java11 Docker container image......
[API] Mounting /tmp/.aws-sam/build/LambdaTestingMultitest as /var/task:ro,delegated inside runtime container
[API] Starting a timer for 30 seconds for function 'LambdaTestingMultitest'
[LAMBDA] Found one Lambda function with name 'LambdaTestingMultitestintermediate'
[LAMBDA] Invoking lambdas.testing.multitestintermediate.TestingMultitestintermediate::handleRequest (java11)
[LAMBDA] No environment variables found for function 'LambdaTestingMultitestintermediate'
[LAMBDA] Environment variables overrides data is standard format
[LAMBDA] Loading AWS credentials from session with profile 'None'
[LAMBDA] Resolving code path. Cwd=/tmp/.aws-sam/build, CodeUri=LambdaTestingMultitestintermediate
[LAMBDA] Resolved absolute path to code is /tmp/.aws-sam/build/LambdaTestingMultitestintermediate
[LAMBDA] Code /tmp/.aws-sam/build/LambdaTestingMultitestintermediate is not a zip/jar file
[LAMBDA] Skipping building an image since no layers were defined
[LAMBDA] 
[LAMBDA] Fetching lambci/lambda:java11 Docker container image......
[LAMBDA] Mounting /tmp/.aws-sam/build/LambdaTestingMultitestintermediate as /var/task:ro,delegated inside runtime container
[LAMBDA] Starting a timer for 30 seconds for function 'LambdaTestingMultitestintermediate'
[LAMBDA] 2020/02/27 10:13:23 listen tcp :9001: bind: address already in use
[LAMBDA] 2020-02-27 11:13:23 127.0.0.1 - - [27/Feb/2020 11:13:23] "POST /2015-03-31/functions/LambdaTestingMultitestintermediate/invocations HTTP/1.1" 200 -
[API] START RequestId: 4fb8d051-c0b0-1df1-8733-25c99015a36b Version: $LATEST
[API] [INFO ] lambdas.util.Lambdas.<clinit>(Lambdas.java:21) Running on local SAM, using local SAM Lambda entrypoint
[API] END RequestId: 4fb8d051-c0b0-1df1-8733-25c99015a36b
[API] REPORT RequestId: 4fb8d051-c0b0-1df1-8733-25c99015a36b	Init Duration: 272.94 ms	Duration: 3239.25 ms	Billed Duration: 3300 ms	Memory Size: 512 MB	Max Memory Used: 111 MB	
[API] 2020-02-27 11:13:23 127.0.0.1 - - [27/Feb/2020 11:13:23] "GET /testing/multitest HTTP/1.1" 200 -
[API] 2020-02-27 11:13:24 127.0.0.1 - - [27/Feb/2020 11:13:24] "GET /favicon.ico HTTP/1.1" 403 -

Expected result

Call (C) going through instead of the docker container not being spawned

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu Linux 18.04.4
2. sam --version: SAM CLI, version 0.43.0

[ashleyfrieze]

Port 9001 isn't a default port for either start api or start lambda. I am able to make this work, but there is a trick to it. You need to make it possible for the lambda in B to be able to reach the SAM running start lambda. This can be achieved by running sam inside a container inside the docker network in which your lambdas launch, or by working out how to reach back to the real linux host from the docker container, by using the host network.

Once I have lambdas running lambdas, I run SAM inside docker too - https://codingcraftsman.wordpress.com/2019/02/26/play-it-again-sam/ <- dockerfile in here shows how to run sam inside docker.

[jfuss]

This is a duplicate of #510 (Lambda to Lambda).

@fbeister
On Linux this is much easier, as you can attach the containers to the host network directly and then you can directly access your localhost. When you configure your lambda function invokes for this, you need the endpoint to be localhost with the port of the start-lambda. On Windows or Mac, you need to use the host.docker.internal with the port as the endpoint.

I am not sure why you are getting :9001: bind: address already in use but this is something to due with your system and you already using that port.

@ashleyfrieze You do not need to run SAM CLI inside docker to make this work. You need to configure the endpoint properly though.

Closing as there is already an issue open to make clear docs on this and is possible to do as @ashleyfrieze and I have described.

[bajh]

I know this issue is closed but I'm having this issue and it seems like 9001 is the lambda runtime port?

aws-sam-cli/samcli/local/go-bootstrap/aws-lambda-mock.go

Line 30 in de4e122

var apiBase = "http://127.0.0.1:9001/2018-06-01"