Bootstrap failure 12/14 #61

powerserg19 · 2022-03-05T11:24:58Z

This is not an issue but a plea for help. When I run synth it returns

Synthesizing this full stack, Stack Name: mytestStack Not making progress trying to resolve environmental context. Giving up. [Error at /mytestStack] Need to perform AWS calls for account 123456789012, but no credentials have been configured

I tried following this link #39 but to no avail. Ive also looked at the other issues

I am trying to run cdklocal in gitlab ci

localstack is up and running before I run synth. I can create a bucket etc..
here are the services that are available
curl localstack:4566/health % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 864 100 864 0 0 96000 0 --:--:-- --:--:-- --:--:-- 105k {"features": {"initScripts": "initialized"}, "services": {"acm": "available", "apigateway": "available", "cloudformation": "available", "cloudwatch": "available", "config": "available", "dynamodb": "available", "dynamodbstreams": "available", "ec2": "available", "es": "available", "events": "available", "firehose": "available", "iam": "available", "kinesis": "available", "kms": "available", "lambda": "available", "logs": "available", "opensearch": "available", "redshift": "available", "resource-groups": "available", "resourcegroupstaggingapi": "available", "route53": "available", "route53resolver": "available", "s3": "available", "secretsmanager": "available", "ses": "available", "sns": "available", "sqs": "available", "ssm": "available", "stepfunctions": "available", "sts": "available", "support": "available", "swf": "available"}, "version": "0.14.0"}

How do I satisfy the credentials cdklocal needs?
This is my latest try in the ci file

    - aws configure set aws_access_key_id foo
    - aws configure set aws_secret_access_key bar

    - echo "[default]" > ~/.aws/config
    - echo "region = us-east-1" >> ~/.aws/config
    - echo "sso_account_id = 123456789012" >> ~/.aws/config
    - echo "sso_role_name = 123456789012MyRoleName" >> ~/.aws/config
    - echo "output = json" >> ~/.aws/config

This is the resultant config and creds when i cat the files when the ci job runs

$ cat ~/.aws/config
[default]
region = us-east-1
sso_account_id = 123456789012
sso_role_name = 123456789012MyRoleName
output = json
$ cat ~/.aws/credentials
[default]
aws_access_key_id = foo
aws_secret_access_key = bar

These are variables

variables:
     AWS_DEFAULT_REGION: us-east-1
    AWS_ACCESS_KEY_ID: foo
    AWS_SECRET_ACCESS_KEY: bar
    STACK_NAME: mytestStack
    CDK_DEPLOY_REGION: us-east-1
    CDK_DEPLOY_ACCOUNT: 123456789012

This is how I am running synth

- cdklocal synthesize --profile default -v

here are the version of awscli and aws-cdk-local. Could Node be issue?

$ cdklocal --version
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!                                                                                  !!
!!  Node v11.15.0 has reached end-of-life and is not supported.                     !!
!!  You may to encounter runtime issues, and should switch to a supported release.  !!
!!                                                                                  !!
!!  As of the current release, supported versions of node are:                      !!
!!  - ^12.7.0                                                                       !!
!!  - ^14.5.0                                                                       !!
!!  - ^16.3.0                                                                       !!
!!  - ^17.3.0                                                                       !!
!!                                                                                  !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(node:1486) ExperimentalWarning: queueMicrotask() is experimental.
2.15.0 (build 151055e)
$ aws --version
aws-cli/1.19.112 Python/2.7.17 Linux/5.4.0-1009-aws botocore/1.20.112

Here is some of the verbose output

CDK toolkit version: 2.15.0 (build 151055e) Command line arguments: { _: [ 'synth' ], profile: 'default', v: 1, verbose: 1, deployAccount: 123456789012, deployRegion: 'us-east-1', lookups: true, 'ignore-errors': false, ignoreErrors: false, json: false, j: false, debug: false, ec2creds: undefined, i: undefined, 'version-reporting': undefined, versionReporting: undefined, 'path-metadata': true, pathMetadata: true, 'asset-metadata': true, assetMetadata: true, 'role-arn': undefined, r: undefined, roleArn: undefined, staging: true, 'no-color': false, noColor: false, validation: true, quiet: false, q: false, '$0': 'cdklocal' }

I am wondering if the REAL VPC I need to reference in my project is causing the issue. I x-ed out the leading 7 chars


Setting "vpc-provider:account=123456789012:filter.vpc-id=vpc-xxxxxxxxce5f18eb92:region=us-east-1:returnAsymmetricSubnets=true" context to {"$providerError":"Need to perform AWS calls for account 123456789012, but no credentials have been configured","$dontSaveContext":true}
Setting "CDK_DEFAULT_REGION" environment variable to us-east-1
context: { 'vpc-provider:account=123456789012:filter.vpc-id=vpc-xxxxxxxce5f18eb92:region=us-east-1:returnAsymmetricSubnets=true':
   { '$providerError':
      'Need to perform AWS calls for account 123456789012, but no credentials have been configured',
     '$dontSaveContext': true },

The text was updated successfully, but these errors were encountered:

whummer · 2022-03-05T14:30:29Z

Thanks for reporting @powerserg19 . Please note that we're using a different default account ID in LocalStack - it should be 000000000000 (not 123456789012). Can you try changing it, or commenting out the account ID part in general?

Also, is it required to specify a profile at all? Ideally cdklocal should work out of the box without having to manually customize the credentials (if that's not the case, we need to fix it!). Can you please share the log output of a plain vanilla run of cdklocal synthesize, without specifying a profile or using custom credentials in the environment? Thanks!

powerserg19 · 2022-03-05T23:34:45Z

@whummer First off - Thank you very much for your early response!
It finally synthesizes without specifying account. I had tried that early on in this effort and did not realize that my stack was not account and region agnostic. I made several changes to make it agnostic.
Thanks again for you help.
Much appreciated!

powerserg19 · 2022-03-07T20:33:43Z

@whummer I tried to run cdklocal deploy "nameofStack" I got un-resolvable account error. How do I depoy without account?
: Error: Unable to resolve AWS account to use. It must be either configured when you define your CDK Stack, or through the environment Unable to resolve AWS account to use. It must be either configured when you define your CDK Stack, or through the environment

powerserg19 · 2022-03-08T19:07:55Z

@whummer
I am perplexed by this. I created a simple stack and all it does is create a bucket. There is no reference to account or region. Synth passes but when I run cdklocal deploy its tell me that I need to specify an account. "Unable to resolve account to use"
If I specify 000000000000, it tells me credentials cannot be resolved

whummer · 2022-03-08T19:33:24Z

@powerserg19 Can you please share your LocalStack startup configuration, as well as the full logs of the last cdklocal ... run? Please note that the local STS service needs to be enabled - so make sure to either include it in the LocalStack SERVICES environment config, or not specify this config at all (which is preferred).

powerserg19 · 2022-03-08T20:34:37Z

@whummer
The first two txt files are java files. They are the CDK code that creates the stack.
The third one is my gitlab yml file. It doesn't specify services and I curl the localstack health endpoint which returns all the services running. Including STS. The fourth file is the CI job output. I called cdklocal deploy with the -v.
Thanks again!

SigmaplusCdk.txt
BucketTest.txt
gitlab-ci.txt
ci-job-log.txt

powerserg19 · 2022-03-09T15:23:27Z

@whummer
I realize this isn't a cdklocal issue
but If you look at the output, cdklocal trying to connect to 127.0.0.1:4566 to get the default account ID and its failing.

Looking up default account ID from STS
Unable to determine the default AWS account: { Error: connect ECONNREFUSED 127.0.0.1:4566
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1088:14)
  message: 'connect ECONNREFUSED 127.0.0.1:4566',

Im running this in gitlab so how can I configure localstack such that cdklocal can access the host?
Currrently I have this config for localstack

  services:
    - name: localstack/localstack:0.14.0
      alias: localstack

and I can reach the host via
localstack:4566/health

powerserg19 · 2022-03-09T15:39:23Z

@whummer
I understand now. I dont alias it :)

I if run this on may local machine it works. Meaning I start localstack, do cdklocal deploy and there are no issues. The stack is deployed successfully.
In gitlab, however I need to bind/network cdklocal docker that is running inside gitlab docker to localstack docker that is running inside gitlab docker as well. I am not sure how and who to configure for that

powerserg19 · 2022-03-09T17:32:24Z

@whummer
I figured out
LOCALSTACK_HOSTNAME: localhost

powerserg19 · 2022-03-09T18:32:41Z

@whummer

here is my latest yaml

    STACK_NAME: myStack
    LOCALSTACK_HOSTNAME: localstack


  services:
    - name: localstack/localstack:0.14.0
      alias: localstack

  script:
    - apt update && apt install git -y
    - npm install -g aws-cdk-local aws-cdk
    - pip install awscli
    - cdklocal --version
    - aws --version
    - cdklocal synth
    - cdklocal -v bootstrap "aws://unknown-account/unknown-region"
    - cdklocal -v deploy

bootstrap reaches 12/14 but then fails and not much error

DKToolkit | 12/14 | 6:45:27 PM | CREATE_FAILED        | AWS::CloudFormation::Stack | CDKToolkit 
Failed resources:
 ❌  Environment aws://unknown-account/unknown-region failed bootstrapping: Error: The stack named CDKToolkit failed to deploy: CREATE_FAILED (Deployment failed)
The stack named CDKToolkit failed to deploy: CREATE_FAILED (Deployment failed)
Error: The stack named CDKToolkit failed to deploy: CREATE_FAILED (Deployment failed)
    at waitForStackDeploy (/usr/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:309:11)
    at processTicksAndRejections (internal/process/task_queues.js:86:5)

powerserg19 · 2022-03-09T19:12:24Z

here is the full deploy result. I get the same behavior if i do

cdklocal bootstrap aws://000000000000/us-east-1 or cdklocal -v bootstrap aws://unknown-account/unknown-region

Waiting for stack CDKToolkit to finish creating or updating...
[AWS cloudformation 200 1.773s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
Stack CDKToolkit has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS (Deployment succeeded))
[AWS cloudformation 200 0.026s 0 retries] describeStackEvents({ StackName: 'CDKToolkit', NextToken: undefined })
CDKToolkit |  0/14 | 7:27:33 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | TrustedAccounts 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | TrustedAccountsForLookup 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | CloudFormationExecutionPolicies 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | FileAssetsBucketKmsKeyId 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | PublicAccessBlockConfiguration 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | FileAssetsBucketName 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | ContainerAssetsRepositoryName 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | Qualifier 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | HasTrustedAccounts 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | HasTrustedAccountsForLookup 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | HasCloudFormationExecutionPolicies 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | HasCustomFileAssetsBucketName 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | CreateNewKey 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | UseAwsManagedKey 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | HasCustomContainerAssetsRepositoryName 
CDKToolkit |  0/14 | 7:27:33 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | UsePublicAccessBlockConfiguration 
CDKToolkit |  1/14 | 7:27:34 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | FileAssetsBucketEncryptionKey 
CDKToolkit |  2/14 | 7:27:34 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | FileAssetsBucketEncryptionKeyAlias 
CDKToolkit |  3/14 | 7:27:35 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | StagingBucket 
CDKToolkit |  4/14 | 7:27:36 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | StagingBucketPolicy 
CDKToolkit |  5/14 | 7:27:36 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | ContainerAssetsRepository 
CDKToolkit |  6/14 | 7:27:36 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | FilePublishingRole 
CDKToolkit |  7/14 | 7:27:36 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | ImagePublishingRole 
CDKToolkit |  8/14 | 7:27:37 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | LookupRole 
CDKToolkit |  9/14 | 7:27:37 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | FilePublishingRoleDefaultPolicy 
[AWS cloudformation 200 0.166s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
[AWS cloudformation 200 0.017s 0 retries] describeStackEvents({ StackName: 'CDKToolkit', NextToken: undefined })
CDKToolkit | 10/14 | 7:27:39 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | CloudFormationExecutionRole 
CDKToolkit | 11/14 | 7:27:39 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | CdkBootstrapVersion 
CDKToolkit | 12/14 | 7:27:40 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | DeploymentActionRole 
CDKToolkit | 12/14 | 7:27:40 PM | CREATE_FAILED        | AWS::CloudFormation::Stack | CDKToolkit 
Failed resources:
 ❌  Environment aws://000000000000/us-east-1 failed bootstrapping: Error: The stack named CDKToolkit failed to deploy: CREATE_FAILED (Deployment failed)
The stack named CDKToolkit failed to deploy: CREATE_FAILED (Deployment failed)
Error: The stack named CDKToolkit failed to deploy: CREATE_FAILED (Deployment failed)
    at waitForStackDeploy (/usr/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:309:11)
    at processTicksAndRejections (internal/process/task_queues.js:86:5)
Cleaning up project directory and file based variables

whummer · 2022-03-09T20:39:39Z

Thanks for the update @powerserg19 . Looks like we're one step closer - the error you're seeing is coming from the CloudFormation deployment. Can you please enable DEBUG=1 in your LocalStack container environment, and then share the detailed container logs here? There is likely an issue with the deployment of one of your resources.. Also, please note that some advanced features may only be supported in our Pro version. Thanks

powerserg19 · 2022-03-09T21:22:48Z

@whummer
Attached is the output with DEBUG on. This is just the deployment of a vanilla example stack. Not my product stack. All it contains is bucket creation. I attached that code earlier in this thread. In this case "cdklocal deploy" works but it fails when I try bootstrap just the same as my product stack( which needs bootstrap because its large) . If you want the product ouput logs i can supply them though I may have to redact a few things.

Here is my CI file

before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  - apt-get -yq update && apt-get -yq install -y python3.7  python-pip
  - python3 -V

stages:
  - synth

synthisize project:
  tags:
    - java11
  stage: synth
  
  services:
    - name: localstack/localstack:0.14.0
      alias: localstack

  variables:
    AWS_DEFAULT_REGION: us-east-1
    STACK_NAME: testDeploy
    LOCALSTACK_HOSTNAME: localstack
    DEBUG: 1

  script:
    - apt update && apt install git -y
    - npm install -g aws-cdk-local aws-cdk
    - pip install awscli
    - cdklocal --version
    - aws --version
    - curl localstack:4566/health
    - cdklocal synth
    - cdklocal -v -v -v --debug bootstrap aws://000000000000/us-east-1
    - cdklocal -v deploy

ci-job-log.txt

powerserg19 · 2022-03-10T02:14:34Z

@whummer
I followed this thread localstack/localstack#5382
and set [email protected].
It passed the bootstrap part and it also deployed my vanilla stack which only consists of a bucket.
When I ran this against my product stack, it passed the bootstrap then almost immediately failed
This was the error

testgaia: deploying... ❌ testgaia failed: ReferenceError: BUCKET_NAME_OUTPUT is not defined BUCKET_NAME_OUTPUT is not defined ReferenceError: BUCKET_NAME_OUTPUT is not defined at ExistingToolkitInfo.get (/usr/lib/node_modules/aws-cdk-local/bin/cdklocal:151:43) at makeBodyParameter (/usr/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:423:38) at deployStack (/usr/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:248:31) at processTicksAndRejections (internal/process/task_queues.js:86:5)

powerserg19 · 2022-03-10T13:03:42Z

@whummer
What does BUCKET_NAME_OUTPUT not defined mean and how can I satisfy that ?

powerserg19 · 2022-03-10T14:55:40Z

@whummer
How much is the pro version and what level of support will I get? Also I would like to know what it will exercise. I can give you details of my stack if need be in a separate email. Thanks

powerserg19 closed this as completed Mar 9, 2022

powerserg19 reopened this Mar 9, 2022

powerserg19 changed the title ~~Trying to configure credentials so I can run synth~~ Bootstrap failure 12/14 Mar 9, 2022

powerserg19 closed this as completed Mar 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootstrap failure 12/14 #61

Bootstrap failure 12/14 #61

powerserg19 commented Mar 5, 2022 •

edited

Loading

whummer commented Mar 5, 2022 •

edited

Loading

powerserg19 commented Mar 5, 2022

powerserg19 commented Mar 7, 2022

powerserg19 commented Mar 8, 2022 •

edited

Loading

whummer commented Mar 8, 2022

powerserg19 commented Mar 8, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022

powerserg19 commented Mar 9, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022

powerserg19 commented Mar 9, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022 •

edited

Loading

whummer commented Mar 9, 2022

powerserg19 commented Mar 9, 2022 •

edited

Loading

powerserg19 commented Mar 10, 2022

powerserg19 commented Mar 10, 2022

powerserg19 commented Mar 10, 2022

Bootstrap failure 12/14 #61

Bootstrap failure 12/14 #61

Comments

powerserg19 commented Mar 5, 2022 • edited Loading

whummer commented Mar 5, 2022 • edited Loading

powerserg19 commented Mar 5, 2022

powerserg19 commented Mar 7, 2022

powerserg19 commented Mar 8, 2022 • edited Loading

whummer commented Mar 8, 2022

powerserg19 commented Mar 8, 2022 • edited Loading

powerserg19 commented Mar 9, 2022

powerserg19 commented Mar 9, 2022 • edited Loading

powerserg19 commented Mar 9, 2022

powerserg19 commented Mar 9, 2022 • edited Loading

powerserg19 commented Mar 9, 2022 • edited Loading

whummer commented Mar 9, 2022

powerserg19 commented Mar 9, 2022 • edited Loading

powerserg19 commented Mar 10, 2022

powerserg19 commented Mar 10, 2022

powerserg19 commented Mar 10, 2022

powerserg19 commented Mar 5, 2022 •

edited

Loading

whummer commented Mar 5, 2022 •

edited

Loading

powerserg19 commented Mar 8, 2022 •

edited

Loading

powerserg19 commented Mar 8, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022 •

edited

Loading

powerserg19 commented Mar 9, 2022 •

edited

Loading