docs: data-driven system-catalog reference, phase 3 (document mz_introspection per_worker variants) - #185
Draft
antiguru wants to merge 4 commits into
Draft
docs: data-driven system-catalog reference, phase 3 (document mz_introspection per_worker variants)#185antiguru wants to merge 4 commits into
antiguru wants to merge 4 commits into
Conversation
An aggregating global view can change a column's type relative to its per-worker sibling (e.g. a numeric SUM(...) global count vs a bigint per-worker count), so a per-worker variant is not safely derived as base columns plus worker_id. A variant now carries its own full column list (name and type, no meaning), rendered as a 2-column table and lint-checked name-sorted against the catalog.
Per-worker variant descriptions claimed "the same data ... but reported per worker," which is wrong for aggregating globals (many parents do SUM/MIN/COUNT GROUP BY worker, so the per-worker relation is the un-aggregated source, not the same data per worker). Reword to an umbrella description that covers both filter- and aggregate-derived globals, and apply it to the 23 existing per_worker variants. Also fix the generator's marker removal to consume an adjacent blank line when the marker sits alone between two blanks, and collapse the resulting doubled blank lines already committed to mz_introspection.md. Add add_variants coverage for the existing-variant (no duplication) and orphan (no parent) skip paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Phase 3 of the system-catalog docs data migration.
Stacked on phase 2 (base branch
catalog-docs-data-phase2); retarget down the stack as earlier phases land.Description
Documents the 22
mz_introspection_per_workerrelations whose parent is already documented, by adding each as avarianton its parent's YAML entry and removing itsRELATION_SPEC_UNDOCUMENTEDmarker.Refines the variants model: a variant carries its own full column list (name and type) sourced from the catalog, rendered as a 2-column table and lint-checked name-sorted.
This is necessary because an aggregating global view changes column types relative to its per-worker sibling (for example
countisnumericin the global butbigintper worker), so a variant cannot reuse the base column types.Variant columns are extracted from the live catalog, and
bin/sqllogicteston the regenerated file proves every declared column name and type matches.Three per-worker relations whose parent is undocumented or lives in
mz_internalstay asRELATION_SPEC_UNDOCUMENTEDmarkers, pending phases 4 and 5.Verification
ci/test/lint-docs-catalog.shpasses,bin/sqllogictestpasses on the regeneratedtest/sqllogictest/autogenerated/mz_introspection.slt(54/54), andhugo+ link checks are clean. The generator has unit tests (ci/test/test_gen_per_worker_variants.py).