[ADAP-547] [Feature] Pagination to allow >10k schemas per database

[jaypeedevlin]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

#571 used pagination to support retrieving >10k objects in a schema. The same pattern can be used to enable >10k schemas in a database.

I propose to use the same technique for the latter as the former (which I've confirmed Snowflake supports).

Describe alternatives you've considered

Overriding the macro in our project.

Who will this benefit?

Anyone using dbt at scale — specifically we run into this from time to time with our databases that create an environment per PR for dev.

Are you interested in contributing this feature?

Yes!

Anything else?

No response

[dbeatty10]

Thanks for opening this @jaypeedevlin !

✅ Green light for you to use the same technique as #572 but applied to >10k schemas.

In a ideal world, you will be able to refactor snowflake__get_paginated_relations_array slightly so that its logic can be reused rather than copy-pasted.

The crux would be generalizing the paginated_sql portion.

Ultimately, that will be up to you and the code reviewer to work out.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[jaypeedevlin]

Bump

[ttishihara]

Hello, are there any updates for this or any new related issues?

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.