Support one Local Lambda calling another Local Lambda scenarios

[suneelthakur1980]

We are in process writing an integration test for lambda.

Here is the situation LambdaA calls LambdaB internally.
Now I am invoking sam lambda A which internally call lmbdaB... ON AWS console this is working but not able to work it from SAM local.

However, if lambaB invocation is removed from LambdaA, both lambda works fine in SAM LOCAL.

Is this supported by SAM local, any workaround to move forward?
code snippets

var AWS = require('aws-sdk');
AWS.config.region = 'eu-west-1';
var lambda = new AWS.Lambda();

exports.handler = function(event, context,callback) {

var params = {
FunctionName: 'lambda-B', // the lambda function we are going to invoke
InvocationType: 'RequestResponse',
LogType: 'Tail',
Payload: JSON.stringify(event,null,null)
};
lambda.invoke(params, function(err, data) {

const dataPayload=data.Payload;
console.log("------dataPayload-----"+dataPayload);
console.log("------Now it should work-----"+JSON.parse(dataPayload).errorType);
if (err) { context.fail(err);
} else {
context.succeed('Lambda_SIGNATURE said '+ JSON.stringify(data));
}

[sanathkr]

Actually, this exact scenario is being enabled by #508. More details on this soon, but the tl;dr version is, we will provide a new command (possibly, sam local start-lambda) that will setup a local HTTP endpoint where you can send Lambda Invokes to.

This allows you to perform Lambda calling Lambda locally by just setting a different endpoint something like:

var lambda = new AWS.Lambda()
if(process.env.AWS_SAM_LOCAL) {
  // only run inside local lambda runner
   var ep = new AWS.Endpoint('localhost:3000');
   var lambda = new AWS.Lambda({endpoint: ep})
}

[suneelthakur1980]

Thanks, Sanath for the reply.
That means current SAM Local version 0.4.0 does not support invoking one lambda form another.
Also if we have complex scenerio where LambdaA -->invoke-->LambdaB --> LambdaC then we will have two endpoints for each lambda? Can we have something like register lambda feature in docker image where we could invoke lambda-like lambda.invoke(params, function(err, data) ..

[sanathkr]

Yes, currently it is not supported, but it will be in the future. When it does, the feature will work similar to what you described. There will be only one endpoint.

[ToddHoff]

Does this work in SAM CLI, version 0.8.1?
I'm starting with:
sam local start-lambda -t sam.yaml

And configure lambda:
var ep = new AWS.Endpoint('127.0.0.1:3001');
this.LAMBDA = new AWS.Lambda({endpoint: ep});

And I sam local shows this error when invoke a lambda function from node on that endpoint:

2019-01-01 12:19:50 127.0.0.1 - - [01/Jan/2019 12:19:50] code 400, message Bad request version ('¤±\x8eé\x11C\x03D\x81\x17q=Ϣɴq/r9ïÉÖD:\x00"\x1a\x1a\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x00')
2019-01-01 12:19:50 127.0.0.1 - - [01/Jan/2019 12:19:50] "ü0¿ó��ç�OüÛà!¹lÚ,���JÕ3��~½O8o �ÝK櫓�éCD�q=Ϣɴq/

Is there something else that needs to be done?

[jadamsdb]

I was able to get this to work on 0.6.0. I am running on mac, so I had to create my lambda with the following endpoint:

new Lambda({ region: 'us-east-1', endpoint: 'http://docker.for.mac.localhost:3001' })

The issue I have now is that it doesn't appear that InvocationType of Event is supported. I wanted to use it as a fire and forget async call from one lambda to another (L1 --(async)-> L2), both running locally. When I update the InvocationType to be Event, I get a response of

invocation-type: Event is not supported. RequestResponse is only supported.

[ejhayes]

@jadamsdb Event invocation type is part of #749

[surbhi029]

I have created two lambdas -validateuserpermission and updateuserpermission. Here is the situation updateuserpermission calls validateuserpermission internally.
Now I did sam local start-lamda and then sam local invoke updateuserpermission which internally call validateuserpermission. I am getting below error

"Could not connect to the endpoint URL: "http://127.0.0.1:3001/2015-03-31/functions/validateuserpermission/invocations\""

ON AWS console this is working but not able to work it from SAM local.

However, if validateuserpermission invocation is removed from updateuserpermission , both lambda works fine in SAM LOCAL.

Could you please help me in resolving this issue so that i would be able to test this situation.

[renfeng]

@surbhi029 127.0.0.1 in your endpoint_url is the virtual host for sam local, which is not the host as you'd expect. --host command line argument allows start-lambda to bind to a specific hostname or ip. You can try one that is accessible to your sam local virtual host.

See https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-lambda.html

[surbhi029]

@renfeng how can i find a hostname or ip that is accessible to my sam local virtual host.

[jfuss]

@surbhi029 If you on mac or windows, use https://host.docker.internal:<port> as the endpoint. If you are on linux, you can attach the container to your host network (and then able to talk to localhost), by adding --docker-network host to the command.

[hburrows]

@jfuss Can you please provide me a little more guidance on the proper configuration to get lambda_A to call lambda_B working within sam local start-lambda. In my situation, when lambda-A invokes lambda-B I'm getting an Inaccessible host: 'host.docker.internal'. This service may not be available in the 'us-west-2' region. error. I've tried a variety of endpoint permutations (http vs https, 127.0.0.1, 192.168.99.100 (which is my local docker host ip), etc.)

This is essentially my setup:

const lambda = new AWS.Lambda({
  apiVersion: '2015-03-31',
  endpoint: 'https://host.docker.internal:3001'
});

lambda.invoke({
  FunctionName: 'lambda_B',
  InvocationType: 'Event',
  Payload: JSON.stringify({"data":"for lambda-b"})
})

I start sam local ... without any extra arguments:

sam local start-lambda

And then invoke with something like this:

aws lambda invoke --function-name lambda_a --endpoint-url "http://127.0.0.1:3001" --no-verify-ssl --payload '{"data":"for lambda-a"}' out.txt

In my lambda_a function if I instantiate my change Lambda connection as such

const lambda = new AWS.Lambda({ apiVersion: '2015-03-31' });

I get no errors but my lambda_b function never gets called (or if it does no output appears in the sam local start-lambda console so it doesn't appear to get called)

[hburrows]

OK. I figured out how to make this work. The following is what worked for me for anyone else encountering the same friction.

First and VERY IMPORTANT... if you're using a Mac or Windows make sure you're using Docker Desktop and NOT Docker Toolbox. Docker Toolbox doesn't support host.docker.internal. You could always find your docker host's IP and hard-code but that's extra work and brittle.

Second... below is example code that works. At the time of me writing this, an invocation-type of 'Event' is not supported but hopefully soon (#749). You cannot use SSL so specify sslEnabled: false and make sure your endpoint isn't referencing https.

const lambda = new AWS.Lambda({
  apiVersion: '2015-03-31',
  endpoint: 'http://host.docker.internal:3001',
  sslEnabled: false
})

await lambda.invoke({
  FunctionName: funcName,
  InvocationType: 'RequestResponse',
  Payload: JSON.stringify({"arguments": "for other function"
})

Assuming you've configured any required environment variables correctly you should be in business.

[JLarky]

var ep = new AWS.Endpoint('localhost:3000');

@sanathkr this should be

var ep = new AWS.Endpoint('http://localhost:3001');

[ngarg-panw]

Hi,
I am very new to working with AWS lambda through Java code. My requirement is the same: calling on lambda from another using sam local.
This is the Java code i have (made changes according to some of the posts above):

        .withEndpointConfiguration(
            new AwsClientBuilder.EndpointConfiguration("http://host.docker.internal:3001", "")).build();
    InvokeRequest request = new InvokeRequest();
    JSONObject body = new JSONObject();
    body.put("body", payload);

    request.withFunctionName(converterLambdaARN)
        .withInvocationType(InvocationType.RequestResponse)
        .withPayload(body.toString());

    InvokeResult result = lambdaClient.invoke(request);```

My template.yml looks like this:
```Resources:
  ScannerFunction:
    # This resource creates a Lambda function.
    Type: AWS::Serverless::Function
    Properties:
      Runtime: java8
      Handler: abc.xyz.awslambda.LambdaInputHandler::handleRequest
      CodeUri: core
      Events:
        ScannerEndpoint:
          Type: Api
          Properties:
            Path: /
            Method: any
          ConversionEndpoint:
            Type: Api
            Properties:
              Path: /conversion
              Method: any```

The above is not working. Looking forward for some pointers.
Thanks

[lbassin]

Even though running start-api in the host network works as a workaround it would be great to have this feature directly in the CLI, is there any update on the subject?

[pkrisko]

To anyone still having this issue... what solved it for us was that some of our lambdas are exposed through API Gateway whereas others are invoked directly (by other lambdas and not exposed through APIGW). We needed to do both sam local start-api and sam local start-lambda, and then configure the internal lambda call to hit the start-lambda server port.

[chaseTfreeman]

@pkrisko -- does this require that you change all your "directly-invoked" lambdas back to their ARNs before deploying?
Like you, I have a mixed use of API gateway invocations and direct ARN-based invocations (client.invoke). From what I understand in your post, I have two options currently:

1. I can test all of this locally if I swap the lambda.invoke() for HTTP requests to the API gateway endpoints of those same lambdas. OR
2. I run start-api && start-lambda and change the ARNs to the lambda_server_port/functionName

Am I understanding that correctly?

EDIT: Fixed my own understanding. I'm using python3, so I needed to define my Boto3 lambda client to use the localhost endpoint for start-lambda. I didn't catch the example in the documentation here (in light gray text under AWS sdk--duh!): https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-lambda.html

With this, I only need to define my boto3/lambda client and in my client.invoke()s I can simply use the FunctionName rather than the ARN. For others that find this be sure to define your region_name correctly.

lambda_client = boto3.client('lambda',
                            region_name='us-east-1',
                            endpoint_url='http://docker.for.mac.localhost:3001')

EDIT2:
While I can finally test lambda-to-lambda invocation locally, when I remove the docker/localhost from the boto3 client definition and deploy, AWS is complaining that it cannot find the lambda invocation in my parent function when I use the FunctionName in client.invoke()

response = lambda_client.invoke(
        FunctionName = 'FunctionNameHere',
        Payload = json.dumps(data_here)
        )

If I change the FunctionName back to the functions ARN I am able to deploy and test the api gateway endpoint successfully using the parent > child function.

[jb3rndt]

As I was trying to achieve a similar thing as the original poster I found that the information in this post are somewhat outdated and want to share my findings.
Setup wise I'm using Linux and sam version 1.23.0.

Starting off with the scenario, I want to call LambdaA through API gateway running locally with sam local start-api. LambdaA then directly invokes LambdaB via boto3.client('lambda').invoke() (python). LambdaB itself is started via sam local start-lambda. Calling LambdaB didn't work.
The first thing I tried was putting both LambdaA and LambdaB into the host network with sam local start-api --docker-network host and sam local start-lambda --docker-network host respectively, as suggested for Linux. Additionally I tried to use localhost as the endpoint for LambdaB like so: boto3.client("lambda", endpoint_url = "http://localhost:3001").
Doing this, I could not even correctly invoke LambdaA through API gateway anymore.

After a way too long investigation I found in #2436 (comment) that using the host network cannot work anymore since sam version 1.13.
After some reading in docker networks, I finally tried to use the docker0 network adapter ip (usually 172.17.0.1) as the endpoint for invoking LambdaB (boto3.client("lambda", endpoint_url = "http://172.17.0.1:3001")). By additionally starting LambdaB with sam local start-lambda --host 0.0.0.0 (Edit: 172.17.0.1 works as well) instead of localhost as the host, I finally got it working.

I cannot explain in depth what is going on as I'm not used to docker, especially networking, yet. What i know is that every container gets put into the bridge network by default, so that is where the lambda containers go. So to address one container from the other we have to use the network adapter ip and most importantly make LambdaB use 0.0.0.0 as host to reveal its interfaces.

I hope this solves this problem for some others.

[chaseTfreeman]

@jb3rndt thank you for the in depth analysis and for sharing your findings. I will give it a shot when I have the time.

But wow! So many hurdles to jump through to have a reasonable local testing environment. I wish AWS serverless team would better acknowledge these drawbacks.

[yonihod]

@hburrows I can't seem to make this work on nodejs

trying to figure this out for so long now.

[sanchezz985]

Hi everyone,

Same issue here in 2021 😅 ... I'm a linux user and I tried the @jb3rndt solution but I can't seem to make this work, at least programatically.

What I want :

1. Invoke lambdaA through API gateway with sam local start-api
2. Invoke lambdaB with sam local start-lambda --host 172.17.0.1 (docker0 network adapter ip) from lambdaA programatically by using the AWS SDK for Node JS.

ApiGateway ===> LambdaA ===> Lambda B

What I got

Step 1 it's ok, I can invoke lambdaA
Step 2 fails, my code tries to connect with lambdaB but execution time it's consumed and I get a message like "Invalid lambda response received: Lambda response must be valid json" and a 502 code.

The code I'm using to invoke lambdaB is:

const AWS = require('aws-sdk');

const lambda = new AWS.Lambda({
  apiVersion: '2015-03-31',
  endpoint: 'http://172.17.0.1:3001', // docker0 network adapter ip
  sslEnabled: false,
  region: 'us-west-2',
  accessKeyId: 'any', // I also tried with my aws credentials
  secretAccessKey: 'any' // I also tried with my aws credentials
});

lambda.invoke({
  FunctionName: 'lambdaB'
}, (err, res) => {
  console.log(res);
});

But the best part comes when I try to invoke lambdaB with aws cli and it works 😅

aws lambda invoke --function-name "lambdaB" --endpoint-url "http://172.17.0.1:3001" --no-verify-ssl --payload '{}' out.txt

What's the difference between aws cli and invoke lambdaB from code ? any help ?

I also tried to connect my containers to a new docker network using docker network create sam-local and I got the same, I could access only with aws cli.

I don't know, maybe I missed something. any suggestions will be appreciated 😃 .

[jb3rndt]

I could not exactly figure out why yours is not working. I gave my best to create a minimal example that should be working. I hope you and other people can find out what is going different in your code and the example and find out why stuff is not working.
You can find a working example here: https://github.com/jb3rndt/local-l2l-invocation.

Some things that might be different to your setup:

• Both lambdas are not running simultaneously at the time they get invoked?
• Conflicting networks? In the example, everything is running in a vm which reduced conflict potential with other applications. So if you get your code running in there, you might have to take another look into your network setup

Generally, I found that the error "Invalid lambda response received: Lambda response must be valid json" is mostly caused by timeouts (in my case at least) which in turn are mostly caused by network endpoints that do not respond. In your case it is not clear to me whether lambdaB got invoked and did throw the 502 or whether lambdaB did not even get invoked 😅.

So I hope that helps.
Anyone looking into the example feel free to contribute with PRs to improve it.

[cezarmezzalira]

Hey @jb3rndt, thank you for your investigation and your reply.

I'm using linux and SAM CLI, version 1.35.0

To me, what worked is to call the target lambda using the command sam build --use-container && sam local start-lambda --host 0.0.0.0 -n env.json .
And, to call the target lambda in another lambda, I'm using this host URL: http://172.17.0.1:3001, basically points to docker IP 172.17.0.1.

Thank you again for your advise!

[HaroldEstn]

Hey @jb3rndt, thank you for your investigation and your reply.

I'm using linux and SAM CLI, version 1.35.0

To me, what worked is to call the target lambda using the command sam build --use-container && sam local start-lambda --host 0.0.0.0 -n env.json . And, to call the target lambda in another lambda, I'm using this host URL: http://172.17.0.1:3001, basically points to docker IP 172.17.0.1.

Thank you again for your advise!

Thank you very much! @cezarmezzalira Changing the IP from http://127.0.0.1:3001 to Docker's IP http://172.17.0.1:3001 and running the Lambda commands within Docker's IP now allows me to invoke LambdaB locally from LambdaA.

Now I start de api server with sam local start-api -t template.yaml --skip-pull-image --port 8080 -n .env.json --host 172.17.0.1
And lambda server with sam local start-lambda -n .env.json --host 172.17.0.1 --port 3001 -t template.yaml --skip-pull-image
and set my lambda client with docker enpoint.

lambda_client = boto3.client(
        'lambda',
        endpoint_url='http://172.17.0.1:3001' if is_local else None,
        region_name='us-east-1'
        )

Now everything works as expected, and I can invoke multiple functions locally with no problems.
I really appreciate your help!