awscli.clidriver crashes when importing openapi.json

[houmie]

Describe the bug

When trying to create an API from openapi spec, we get a crash without any explanation. Our openapi.json is auto generated by FastAPI and validated via https://editor-next.swagger.io
Based on the OpenAPI spec it is correct. So it must be something with AWS.

aws apigateway --debug import-rest-api --body 'file:///Users/houmie/Downloads/release/openapi.json'

We found the issue. See the section: Possible Solution

Expected Behavior

I expected it to import it successfully or at least show what is rejected by AWS API Gateway.

Current Behavior

Error:

2023-09-21 09:22:05,133 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"message":"Invalid OpenAPI input."}\n'
2023-09-21 09:22:05,133 - MainThread - botocore.hooks - DEBUG - Event needs-retry.api-gateway.ImportRestApi: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x105907050>>
2023-09-21 09:22:05,133 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-09-21 09:22:05,133 - MainThread - botocore.hooks - DEBUG - Event after-call.api-gateway.ImportRestApi: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x1058c9910>>
2023-09-21 09:22:05,133 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 460, in main
    return command_table[parsed_args.command](remaining, parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 595, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 798, in __call__
    return self._operation_caller.invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 929, in invoke
    response = self._make_client_call(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/clidriver.py", line 941, in _make_client_call
    response = getattr(client, xform_name(operation_name))(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/botocore/client.py", line 341, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/awscli/2.13.20/libexec/lib/python3.11/site-packages/awscli/botocore/client.py", line 697, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.BadRequestException: An error occurred (BadRequestException) when calling the ImportRestApi operation: Invalid OpenAPI input.

An error occurred (BadRequestException) when calling the ImportRestApi operation: Invalid OpenAPI input.

Reproduction Steps

When trying to create an API from openapi spec, we get a crash without any explanation. Our openapi.json is auto generated by FastAPI and validated via https://editor-next.swagger.io

Then imported simply by:
aws apigateway --debug import-rest-api --body 'file:///Users/houmie/Downloads/release/openapi.json'

Possible Solution

UPDATE: We have converted the spec to YAML and then AWS Online Importer was able to show us the error on the first line: "openapi":"3.1.0" after changing it to "openapi":"3.0.0". We were able to import it.

Additional Information/Context

No response

CLI version used

2.13.20

Environment details (OS name and version, etc.)

MacOS Ventura 13.5.2 (22G91)

[aBurmeseDev]

Hi @houmie - thank you for reaching out.

I noticed the edit that you were able to import it, and I wanted to confirm if you're still seeing the issue. One thing i noticed is that might be the culprit is using file:// instead of fileb:// as mentioned in the CLI docs to use fileb.

Let me know if you have any other questions!
John

[houmie]

Hi John,

Thank you. The issue is happening both with AWS-CLI and with the online API-GATEWAY importer. Both fail on the same issue. Latest FastAPI's swagger generates the openapi file as version 3.1.0

{"openapi":"3.1.0","info":{"title":"FastAPI","version":"0.1.0"}

This crashes AWS-CLI / Webform. But if I change it to

{"openapi":"3.0.0","info":{"title":"FastAPI","version":"0.1.0"}

then it works. I think the code is internally hardcoded to accept only up to version 3.0.0, anything higher crashes.
Ideally this should be caught and exited in a friendly way, instead of a hard crash. But up to you how you see fit.

Best Regards,
Houman

[kdaily]

Hi! This part of the debug logs lets you know that this is an error coming back from the server:

2023-09-21 09:22:05,133 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"message":"Invalid OpenAPI input."}\n'

If you look above that a few lines, you'll see the full response, including the server HTTP error code. Question: is there any other information coming back in the response or headers? Sometimes services return more details that are not parsed into the final error message.

I do agree that this is not very helpful. The API documentation page does not list the supported OpenAPI version supported, and this is the same documentation that is used when generating the AWS CLI documentation page:

• https://docs.aws.amazon.com/apigateway/latest/api/API_ImportRestApi.html
• https://docs.aws.amazon.com/cli/latest/reference/apigateway/import-rest-api.html

However, the usage guide does state this limitation:

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html

You can use API Gateway to import a REST API from an external definition file into API Gateway. Currently, API Gateway supports OpenAPI v2.0 and OpenAPI v3.0 definition files, with exceptions listed in Amazon API Gateway important notes for REST APIs.

The documentation and error codes are modeled and managed by the API Gateway team. We can let them know that this is a frustrating error and request that they improve the documentation or the error message.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[houmie]

@kdaily Yes, that would be helpful. Thanks

[aBurmeseDev]

I've forwarded this feedback to service team. Since there are no actionable item from AWS CLI team, I'm going to close this issue now. Thanks again for sharing this feedback.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.