Skip to content

simplify registry flow#456

Merged
zzstoatzz merged 10 commits intomainfrom
revamp-registry-flow
Feb 24, 2025
Merged

simplify registry flow#456
zzstoatzz merged 10 commits intomainfrom
revamp-registry-flow

Conversation

@zzstoatzz
Copy link
Copy Markdown
Collaborator

@zzstoatzz zzstoatzz commented Jan 6, 2025

effectively closes #323

most of the diff is the uv.lock file

this pr:

  • removes github3 dependency
  • stops using run_deployment to spin up separate infra for specific collections and instead just uv run the subflow (that handles a single collection) in a subprocess
  • adds pre-commits to CI and adds the JSON schema validation as a local hook
  • fixes the worker metadata sync to prefect core by moving it all into python

@zzstoatzz zzstoatzz requested a review from a team as a code owner January 6, 2025 03:15
@zzstoatzz zzstoatzz force-pushed the revamp-registry-flow branch 2 times, most recently from f9584c7 to b6fa7dc Compare January 6, 2025 03:22
whoops

lol

update deps
@zzstoatzz zzstoatzz force-pushed the revamp-registry-flow branch from b6fa7dc to a6e4031 Compare January 6, 2025 03:31
@zzstoatzz zzstoatzz added the enhancement New feature or request label Jan 6, 2025
ok what

grrrr

pre-commit and package
@zzstoatzz zzstoatzz force-pushed the revamp-registry-flow branch from ecd9764 to 3463c2f Compare January 6, 2025 04:38
Comment on lines +26 to +30
"secret-str",
"secret-int",
"secret-dict",
"secret-list",
"secret-float",
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: handle this case in Block where we have something like Secret[str]

Comment thread .pre-commit-config.yaml
Comment on lines +14 to +19
- id: json-schema-validation
name: JSON Schema Validation
entry: uv run src/prefect_collection_registry/view_schema_validation.py
language: system
pass_filenames: false
types: [json]
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run JSON schema validation as a pre-commit hook

@zzstoatzz zzstoatzz force-pushed the revamp-registry-flow branch 2 times, most recently from 171ed88 to e6d9b0b Compare January 6, 2025 07:46
add note

push code

update entrypoint

whoops

subprocess needs it too
@zzstoatzz zzstoatzz force-pushed the revamp-registry-flow branch from 9fb5ce4 to c13ff5d Compare January 6, 2025 07:54
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary now that schema validation is a pre-commit hook?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think so yes bc there's no guarantee someone installs the pre-commit

Comment thread prefect.yaml


image: prefecthq/prefect:3.1.11-python3.12
command: uv run --with git+https://github.com/PrefectHQ/prefect-collection-registry.git@main python -m prefect.engine
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you overriding the command here?

Copy link
Copy Markdown
Collaborator Author

@zzstoatzz zzstoatzz Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to install the dependencies for the flow, putting uv sync in a pull step wasn't working for some reason I'd like to figure out later

Comment thread src/prefect_collection_registry/cli.py Outdated
Comment thread src/prefect_collection_registry/update_collection_metadata.py Outdated
Comment thread src/prefect_collection_registry/utils.py
Comment on lines +226 to +229
response = await client.get(
url=f"https://api.github.com/repos/{repo_owner}/{repo_name}/contents/{path}",
headers={"Accept": "application/vnd.github+json"},
)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like you have a lot of different ways to get files from GitHub.

Copy link
Copy Markdown
Collaborator Author

@zzstoatzz zzstoatzz Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes one of them was a duplicate, thanks for catching that, but the other 2 serve different purposes

@zzstoatzz zzstoatzz merged commit a9436e6 into main Feb 24, 2025
@zzstoatzz zzstoatzz deleted the revamp-registry-flow branch February 24, 2025 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

use collection manager for metadata scraping

2 participants