aws sqs list-queues returns incomplete result

[captainfalcon23]

Describe the bug

As per AWS documentation - https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sqs/list-queues.html

list-queues is a paginated operation. Multiple API calls may be issued in order to retrieve the entire data set of results. You can disable pagination by providing the --no-paginate argument. 

It seems in an account where there are > 1000 queues, there are two issues:

1. AWS CLI does NOT automatically perform paging to get all the results (as it does will heaps/all other services)

It seems like this behaviour is similar to "--no-paginate" i.e whether i specify "--no-paginate" or not, the resulting json returned is the same

3. The "NextToken" is ONLY included if I specify "--max-items". If I do NOT specify "--max-items" there is NO "NextToken" in the results (and only 1000 are returned)

Expected Behavior

AWS CLI will return ALL SQS queues, and dymamically perform paging, as it does with other services.

In addition, NextToken should be provided in the response if I do NOT specify "--max-items"

Current Behavior

See "describe the bug" section.

Reproduction Steps

Try running the below in an account with > 1000 queues:

aws sqs list-queues --no-cli-pager |wc -l

Possible Solution

Implement CLI auto paging as per almost every other aws service

Additional Information/Context

No response

CLI version used

aws-cli/2.4.19 Python/3.8.8 Linux/5.10.102.1-microsoft-standard-WSL2 exe/x86_64.centos.7 prompt/off

Environment details (OS name and version, etc.)

Centos 7.9 WSL

[captainfalcon23]

I just discovered if I run this:

aws sqs list-queues --max-items 999 --next-token XXXX

AWS CLI gives the following error:

**Cannot specify --no-paginate** along with pagination arguments: --max-items

So it seems "no-paginate" has been hard coded?

[tim-finnigan]

Hi @captainfalcon23 thanks for reaching out. It looks like the ListQueues API only returns a maximum of 1000 results: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ListQueues.html

If you do not set MaxResults, the response includes a maximum of 1,000 results. If you set MaxResults and there are additional results to display, the response includes a value for NextToken

It notes that in the CLI documentation as well:

QueueUrls → (list)

A list of queue URLs, up to 1,000 entries, or the value of MaxResults that you sent in the request.

Can you use --queue-name-prefix to filter based on the prefix you’re looking for?

[captainfalcon23]

Hi @tim-finnigan

Thanks for the reply. Yep, I did see that in the documentation, but my expectation is that AWS CLI is going to perform the pagination and get ALL the results by using the NextToken. Please see my comment below my original post.

There are a few issues:

1. AWS CLI does NOT automatically perform paging to get all the results (as it does will heaps/all other services)
2. The "NextToken" is ONLY included if I specify "--max-items". If I do NOT specify "--max-items" there is NO "NextToken" in the results (and only 1000 are returned)
3. Using "NextToken" + "--max-items" fails with the error, even though I am NOT providing the "--no-paginate" option i.e it seems like this has been hard coded in the AWS CLI.

**Cannot specify --no-paginate** along with pagination arguments: --max-items

I quote from the S3 example I mentioned - https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/list-objects-v2.html:

list-objects-v2 is a paginated operation. Multiple API calls may be issued in order to retrieve the entire data set of results. You can disable pagination by providing the --no-paginate argument.

and

Returns some or all (up to 1,000) of the objects in a bucket with each request.

In the case of S3api, we can clearly see that it is making multiple API requests to get the full results. This is an example from one of my test buckets:

aws --profile XXX s3api list-objects  --bucket XXX --region ap-southeast-2 --output text --no-cli-pager |wc -l

**67732**

So - this issue is about the fact that the behaviour between the two AWS CLI commands is different, even though the behaviour of the underlying AWS API is the same (returns a max of 1000 results.

[tim-finnigan]

Hi @captainfalcon23 thanks for following up. That error you mentioned was also reported in #2555, with this comment offering an explanation on that.

Also for more context here is the CLI documentation on pagination: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html

There are inconsistencies between service APIs but we don’t have control over the API logic or parameter names.

[captainfalcon23]

Thanks for the update @tim-finnigan

Upon reading the issue you mentioned, basically, what is happening is, I assume is when “max-items” is specified, the “no-paginate” option is getting silently added.

And the other issue is AWS broken api not returning next token. If they fix their api, the cli command should ideally just work (again I assume).

Do you know of the best way to raise the issue directly with aws to fix their sqs api?

[tim-finnigan]

I'll reach out to the SQS team regarding this but can't guarantee if or when they would update their API behavior. You can also try opening a support case with AWS to help escalate this if it is a blocker for you.

[tim-finnigan]

P63886091

[captainfalcon23]

Thanks so much @tim-finnigan!

[tim-finnigan]

We haven't received any update yet but I'm going to transfer this to our cross-SDK repository as the issue relates to the SQS ListQueues API rather than the CLI directly.

[JoshMcCullough]

Same issue here. We have a LOT of queues and it seems there's no way for me to programmatically do things with them. It would be so, so nice if we could filter list-queues with more than just "starts with" e.g. wildcard. Also, the paging issues. :)

[JohnAtOlo]

I can't find any way to use the next token or CLI to get a queue count past 1,000.

[tim-finnigan]

Checking in to highlight the API documentation again: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ListQueues.html

Returns a list of your queues in the current region. The response includes a maximum of 1,000 results. If you specify a value for the optional QueueNamePrefix parameter, only queues with a name that begins with the specified value are returned.

The listQueues methods supports pagination. Set parameter MaxResults in the request to specify the maximum number of results to be returned in the response. If you do not set MaxResults, the response includes a maximum of 1,000 results. If you set MaxResults and there are additional results to display, the response includes a value for NextToken. Use NextToken as a parameter in your next request to listQueues to receive the next page of results.

Pagination behavior may vary slightly between the CLI and SDKs, but the underlying API is the limiting factor here. And the current behavior is unlikely to change due to the need to maintain backwards compatibility with previous versions. We can help provide guidance for individual issues as I think there are some nuances to the different things that have been reported here. We recommend reaching out through AWS Support for further assistance if you have a support plan, or create an issue in the individual repository for the CLI/SDK you are using if you have encountering general pagination issues.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[captainfalcon23]

Hi @tim-finnigan
I acknowledge the limitation of the SQS API to only return a maximum of 1000 results. This is the same as EC2, IAM and many other services.

The docs state

If you set MaxResults and there are additional results to display, the response includes a value for NextToken. Use NextToken as a parameter in your next request to listQueues to receive the next page of results

So, if there are say 1500 queues to return, I expect the first API call to return 1000 + a NextToken, then I expect AWS CLI to automatically use the next token to fetch the next page of 500.

I am totally confused why you believe this is an SQS problem, when it seems like a problem with the CLI and it's implementation for this service.

I am also confused _where _ I should reraise this issue. It was originally raised under aws/aws-cli, then moved to aws/aws-sdk. There is no specific repos for the AWS services that I can see, and the aws-cli is what I am having the problem with.

Could you assist?

[tim-finnigan]

Hi @captainfalcon23 thanks for following up and apologies for any confusion on my end. I think this comment summarizes the issue well: https://repost.aws/questions/QUi7KyD3R_RlWYTqlBGvZaGQ/how-to-get-a-list-of-all-sqs-queues#ANeUpycZ2DRJKHPcGOBeQWyw

This behavior does seem to contradict the CLI pagination documentation: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html#cli-usage-pagination-serverside

By default, the AWS CLI uses a page size determined by the individual service and retrieves all available items. For example, Amazon S3 has a default page size of 1000. If you run aws s3api list-objects on an Amazon S3 bucket that contains 3,500 objects, the AWS CLI automatically makes four calls to Amazon S3, handling the service-specific pagination logic for you in the background and returning all 3,500 objects in the final output.

I'm going to transfer this issue back to the CLI repository for further investigation. Thanks for your patience here. In the meantime you can use the approach mentioned on the post linked above or use an SDK like Boto3 to get all of the results:

import boto3

client = boto3.client('sqs', region_name='us-west-2')

paginator = client.get_paginator('list_queues')

operation_parameters = {'MaxResults': 1000}

page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:
    print(len(page['QueueUrls']))

[tim-finnigan]

Hi @captainfalcon23 - I wanted to check in again regarding this earlier comment, it looks like the issue there is that --starting-token should be used rather than --next-token when there is a NextToken to return more results. (This is covered more in the pagination documentation). It's just a confusing error that Cannot specify --no-paginate along with pagination arguments: --max-items shows up when using --next-token.

I should also note that only using --page-size does return the full results beyond the 1000 queue request limit, for example: aws sqs list-queues > queues.txt --page-size 100. So I think the investigation from here should be into why the full results are only returned with --page-size is used.

To clarify: As I described in boto/boto3#3406 (comment), specifying PageSize works because it sets MaxResults, which does provide a NextToken value. So passing --max-results directly will also return a NextToken.

[captainfalcon23]

Hi @tim-finnigan
I just retested this today - I created 1300 dummy SQS queues in my account.

[~]$ aws --profile XXX-internal sqs list-queues --region ap-southeast-2 --no-cli-pager --output text|wc -l
1000
[~]$ aws --profile XXX-internal sqs list-queues --region ap-southeast-2 --no-cli-pager --output text --page-size 100|wc -l
1300

EXPECTATION:
CLI auto paginates and retrieves the full list for : aws --profile XXX-internal sqs list-queues --region ap-southeast-2 --no-cli-pager --output text|wc -l

So yeah it looks like specifying page-size makes it work, so the problem definitely lies within the aws-cli and unable to automatically do pagination without further input.

Regarding the point of --next-token vs --starting-token, that looks like another bug. It seems --next-token is not documented in the CLI help, nor the AWS docs, but exists as a CLI argument

[tim-finnigan]

Checking in again — the --starting-token parameter is documented for the list-queues command. An example for using it would be running this to get NextToken:

aws sqs list-queues --max-items 1000 --page-size 1000

And then using that token like this to get additional queues:

aws sqs list-queues --max-items 1000 --page-size 1000 --starting-token "...TQT09In0="

But if you just specify --page-size, then it will continue fetching results until all are returned:

aws sqs list-queues > queues.json --page-size 100

(And if you add --debug to your command, then you can see how the API requests are for MaxResults set to 100 while there is still a NextToken in the response.)

So this use case can be met with the above approach. Maybe the AWS CLI docs on pagination could better clarify this behavior, in which case I recommend using the Provide feedback link on that page to send feedback directly.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.