Send GET request to "sam local start-api" from a Python Function "sam local invoke X"

[gccalvin]

Hello. This is most likely a misunderstanding on my part.

I am using a Github Action workflow running ubuntu-latest, with python 3.9, with aws-actions/setup-sam@v2.

After running sam build and validate, I run the following:

- name: Test API and Function
      run: |
        docker network create local-lambda
        sam local start-api --docker-network local-lambda &
        sam local invoke VisitorCounterIT --no-event --docker-network local-lambda

My function (VisitorCounterIT) looks like:

import requests

def lambda_handler(event, context):
    r = requests.get('http://127.0.0.1:3000/CounterUpdate')
    return r

I know lambda needs to return JSON, but the issue is that it can't reach the local-api:

Error:  ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=3000): Max retries exceeded with url: /CounterUpdate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f704dc9e3d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

However, I can curl the api with no issues through the vm. Also, I've commented out the local docker DynamoDB database for debugging, but I was able to connect from the function to the DynamoDB through the endpoint_url='http://dynamodb:8000'

Is there an endpoint url I should be using to send this GET request?

I've tried messing with some of the container parameters and not using the docker network, but I still can't get pass this. I also tried the host.docker.internal, localhost, and local docker IP, and the I get the same error. --container-host-interface 0.0.0.0 also doesn't help.

Any help would be appreciated, thank you.

[jfuss]

@gccalvin When you provide --docker-network this is the docker network that the container will attach too when the local lambda function is invoked. sam local start-api runs on the localhost of the machine (not within the docker-network). So in order for the VisitorCounterIT to communicate to the local API gateway, you can follow https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host.

[gccalvin]

Hello @jfuss
Thank you for providing that link to the documentation. From what I gathered I should try host.docker.internal within my Python function. This still didn't work for me.

I removed the docker network, which means start-api runs on localhost and sam local invoke runs on a docker container? I still couldn't connect with localhost, 127.0.0.1, host.docker.internal:
Error: ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=3000): Max retries exceeded with url: /CounterUpdate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f42859e6940>: Failed to establish a new connection: [Errno -2] Name or service not known'))

I found this. Which mentioned 172.17.0.1 being used for the default network bridge. This time I got the following error:
Error: ConnectionError: HTTPConnectionPool(host='172.17.0.1', port=3000): Max retries exceeded with url: /CounterUpdate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5f37919940>: Failed to establish a new connection: [Errno 111] Connection refused'))

I also tried the following with no luck: (using host.docker.internal in the function)
sam local invoke VisitorCounterIT --no-event --container-host 172.17.0.1 --container-host-interface 0.0.0.0 --debug

My function can send GET requests to other servers, like google.com, and a curl on the vm to 127.0.0.1 has no issues.

Are there any extra networking arguments I should be passing to my sam commands to get the function to communicate with the Ubuntu vm? Or do you know of a working case where a sam local invoke could send requests to the sam local start-api? I can provide more info if needed. Thank you.

[jfuss]

@gccalvin In my experience with Docker, the networking for it is super confusing.

I found this

I would be cautious. Unless Docker has that specifically documented, it can change and by one of the comments it depends on how you have things setup. "This DOES NOT work on all cases. If you have other networks, a new interface is going to be created: 172.17.0.1, 172.18.0.1, 172.19.0.1 and so on (try ifconfig to list all interfaces). You have to manually obtain the IP for your network"

removed the docker network, which means start-api runs on localhost and sam local invoke runs on a docker container?

To be precise, start-api (the endpoint) runs on your localhost but the lambda functions invoked runs in Docker. For sam local invoke the lambda function you invoke runs in Docker. So what has to happen, for what you want, is the container needs to be able to communicate to your localhost on your machine.

My function can send GET requests to other servers, like google.com, and a curl on the vm to 127.0.0.1 has no issues.

This is not surprising and has to do how the containers do DNS (more or less). So a lookup for google can resolve but a lookup to you machines localhost will not (from the container) without the IPAddress to route the request.

On the VM to 127.0.0.1, that also makes sense since the VM would be able to do the lookup for where your localhost is on that machine.

#510 (comment) might help? Maybe what you are missing is --host?

[gccalvin]

Hello @jfuss

Thank you for the information, and linking that issue. However, in their case, they are using sam local start-lambda wheras I'm using sam local invoke. So the --host parameter isn't available to me.

However, I've come to the conclusion that there is no need for a SAM function to locally test my main lambda function, when I can just call it as a regular python function.

I removed my sam local invoke and replaced it with a call to my python function, and ensured that sam local start-api had the right --docker-network parameter passed. Since the python function is running on the localhost, not a container, it could send a request with the following: r = requests.get('http://127.0.0.1:3000/CounterUpdate')

I'm still not completely sure why the function launched in a docker from sam local invoke had so much trouble reaching the api, but I no longer need it for my use case.

Thanks for your help!