SQS get_paginator for list_queues only returning 1000 items

[AMMullan]

Describe the bug

When using the paginator with list_queues I can only get 1000 max results unless I explicily define PageSize - never seen this on any other service and to me, this feels like a bug?

Expected Behavior

Returns complete list of SQS Queues

Current Behavior

Returns 1000

Reproduction Steps

#
# Pagination - Broken
#
queues = [
    queue
    for page in sqs.get_paginator('list_queues').paginate()
    for queue in page.get('QueueUrls', [])
]
print('Paginator:', len(queues)) # Prints 1000

#
# Pagination - Works
#
queues = [
    queue
    for page in sqs.get_paginator('list_queues').paginate(
        PaginationConfig={
            'PageSize': 1000
        }
    )
    for queue in page.get('QueueUrls', [])
]
print('Paginator:', len(queues)) # Prints 2097

#
# NextToken - Works
#
queue_count = 0
params = {
    'MaxResults': 1000
}
while True:
    queues = sqs.list_queues(**params)

    for queue in queues.get('QueueUrls', []):
        queue_count += 1

    next_token = queues.get('NextToken')
    if next_token:
        params['NextToken'] = next_token
    else:
        break
print('NextToken:', queue_count) # Prints 2097

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3/1.23.6 botocore/1.26.6

Environment details (OS name and version, etc.)

Ubuntu 22.04

[tim-finnigan]

Hi @AMMullan thanks for reaching out. I commented on a related issue here a few months ago: #3078 (comment). As mentioned there only 1000 results are returned by default but you can use pagination to iterate through all of the queues. This is noted here in the documentation as well:

[AMMullan]

Thanks @tim-finnigan but that's what get_paginator is for...?

I've never had to add the PageSize on to get_paginator for any other service to get complete results (i.e. ec2.get_paginator('describe_instances').paginate() returns all instances with no extra params required.

[tim-finnigan]

Hi @AMMullan but in the case of describe_instances do you have over 1000 instances? The MaxResults for that API call is also 1000. This issue you brought up actually came up recently the AWS CLI repository as well: aws/aws-cli#8160. As mentioned there I reached out to the SQS team about increasing the max limit of a request but I have not received a response yet.

[AMMullan]

@tim-finnigan - that's a good point, I never have more than 1000 instances. I do have more than 1000 IAM Roles and the get_paginator is required for that and returns them all BUT there is no mention on the doc page for that about a limit.

I think I'm just confused by the fact that the paginator needs additional pagination, unless you pass in the PageSize; in which case it work as you'd expect. Am I being simple here and not understanding what the point of get_paginator is? I thought I'd been using it correctly for the last few years but perhaps not :/

[tim-finnigan]

Hi @AMMullan thanks for following up. In the SQS Developer Guide it notes:

If you do not set MaxResults, the response includes a maximum of 1,000 results and the NextToken value in the response is null.

So I think what's happening is that setting PageSize inherently sets MaxResults which allows for all results to return.

I could also confirm the behavior by passing MaxResults directly to retrieve all queues using an approach similar to the example in the boto3 paginator documentation:

import boto3

# Create a client
client = boto3.client('sqs', region_name='us-west-2')

# Create a reusable Paginator
paginator = client.get_paginator('list_queues')

operation_parameters = {'MaxResults': 1000}

# Create a PageIterator from the Paginator
page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:
    print(len(page['QueueUrls']))

Each service team owns their pagination models and there are some inconsistencies in how those models are defined. I tried testing the IAM ListRoles paginator for comparison but I'm only able to create up to 1000 roles in my account.

[AMMullan]

Thanks @tim-finnigan - at least now I know for future ref to always include the PageSize/MaxResults in my pagination config :)