merge queue: embarking main (52b5efb) and #811 together#907
Closed
mergify[bot] wants to merge 21 commits intomainfrom
Closed
merge queue: embarking main (52b5efb) and #811 together#907mergify[bot] wants to merge 21 commits intomainfrom
mergify[bot] wants to merge 21 commits intomainfrom
Conversation
Extend the model discovery script to probe Gemini models on Vertex AI in addition to Anthropic models. Uses the correct publisher-specific endpoints: generateContent for Google models, rawPredict for Anthropic. KNOWN_MODELS is now a list of (model_id, publisher, provider) tuples and the probe/version-resolution helpers accept a publisher parameter to construct the correct API URLs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the hardcoded KNOWN_MODELS list with API-driven discovery via
the Model Garden list endpoint (v1beta1/publishers/{publisher}/models).
Models are filtered by configurable prefix patterns and exclusion
regexes, with a seed list fallback when the API is unavailable.
Add version limiting (MAX_VERSIONS_PER_FAMILY=2) to keep only the
N most recent versions per model family, dropping older models like
claude-opus-4-1 automatically.
Log a summary table showing every model from each publisher with its
final disposition (KEEP, EXCLUDE, SKIP, DROP).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…from version limiting Move the collections.defaultdict import from inside keep_latest_versions to the top of the file per PEP 8. Provider default models (defaultModel + providerDefaults values from models.json) are now exempt from version limiting and always kept, ensuring the platform's default models are never dropped even when more than MAX_VERSIONS_PER_FAMILY versions exist in a family. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…paths Add PublisherConfig TypedDict for the PUBLISHERS constant so tooling catches key-name typos. Type discover_models manifest param as dict[str, object] instead of bare dict. URL-encode model_id in resolve_version and vertex_id in _build_probe_request for consistency with list_publisher_models which already quotes the publisher parameter. Restore the inline comment explaining why 403/404 responses bail without retry in resolve_version. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Model Garden list API already returns versionId for each model. Extract it from the list response instead of making a separate GET call per model via resolve_version. This eliminates the 403 errors on models that don't expose version info through the individual GET endpoint, and reduces API calls from N+1 to 1 per publisher. The roles/serviceusage.serviceUsageConsumer IAM role is no longer required — the script now only needs roles/aiplatform.user for probe requests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 3-attempt retry loop with exponential backoff for transient HTTP errors (429, 5xx, timeouts), matching the pattern used in probe_model. Non-retryable errors (403, 404) bail immediately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move urllib.request into the stdlib import block per PEP 8. Rename "DROP (version limit)" to "SKIP (version limit)" in the summary table since manifest entries are never deleted — the model is simply not processed. Add explicit ValueError for unknown publishers in _build_probe_request instead of silently defaulting to Anthropic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sion filter Model flags are now created enabled with 100% rollout instead of disabled at 0%. Add EnabledByDefault field to FlagSpec and enableFlagInEnv helper that calls the Unleash environment toggle API. Update parse_model_family to handle Gemini's semver naming convention (e.g. gemini-2.5-flash -> family gemini-flash, version (2, 5)) and strip temporal qualifiers (preview, exp, date stamps) so preview variants group with their stable counterpart. Add min_version to PublisherConfig to exclude old model generations. Google models with version <= 2.0 are now filtered out. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…name min_version Add publisher guard before the retry loop in probe_model so an unknown publisher returns "unknown" instead of crashing with a ValueError from _build_probe_request. Handle io.ReadAll error in enableFlagInEnv instead of silently discarding it, so error responses include a diagnostic when the body can't be read. Rename min_version to version_cutoff — the field excludes models at or below the given version, making it a ceiling not a floor. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
19 tests covering parse_model_family (Claude trailing digits, Gemini semver, qualifier stripping), model_id_to_label, and keep_latest_versions (version limiting, protected models, semver grouping, empty input). Uses stdlib unittest with no external deps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add unit tests for enableFlagInEnv covering the happy path and non-2xx error path. Move io.ReadAll inside the error branch so the response body is only consumed when needed. Apply version_cutoff filtering to seed models so they respect the same version floor as API-discovered models. Fix keep_latest_versions docstring — gemini-2.5-flash now parses to a version with the semver fix, so it's not a valid example of a versionless model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace total=False on PublisherConfig with NotRequired on the only optional field (version_cutoff). This makes publisher, provider, prefixes, and exclude required at the type level, preventing a latent KeyError if a future config omits them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The EnabledByDefault flag and enableFlagInEnv changes have been moved to feat/model-flags-enabled-by-default for a separate PR. This branch now contains only model-discovery script changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… test Restructure the retry loop so transient errors break on the final attempt instead of returning inline. The post-loop if-data-is-None guard is now the single reachable failure path with a proper error message including the last exception. Add missing assertion for single-version family in test_versionless_always_kept. Add test_seed_models_respect_version_cutoff to verify seed models are filtered by version_cutoff when the list API is unavailable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move GHA-only scripts to .github/scripts/ where they belong. Update workflow reference and DEFAULT_MANIFEST path. Address PR review feedback: - Document the pagination safety limit (20 pages × 100 = 2000 max) - Use @patch for SEED_MODELS in tests instead of direct mutation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎉 This pull request has been checked successfully and will be merged soon. 🎉
Branch main (52b5efb) and #811 are embarked together for merge.
This pull request has been created by Mergify to speculatively check the mergeability of #811.
You don't need to do anything. Mergify will close this pull request automatically when it is complete.
Required conditions of queue
defaultfor merge:#approved-reviews-by >= 1[🛡 GitHub repository ruleset ruleprotect-main]#changes-requested-reviews-by = 0[🛡 GitHub repository ruleset ruleprotect-main]Required conditions to stay in the queue:
#approved-reviews-by >= 1[🛡 GitHub repository ruleset ruleprotect-main]#approved-reviews-by>=1#changes-requested-reviews-by = 0[🛡 GitHub repository ruleset ruleprotect-main]#changes-requested-reviews-by=0check-success=End-to-End Testscheck-success=test-local-dev-simulationcheck-success=validate-manifestscommented-reviews-by=coderabbitai[bot]