feat: plugin-keyed config options (typed/blob split, file-syntax layer)#21
Merged
Conversation
Add the forward-facing config surface for plugin-owned parse options: a generic, plugin-keyed plugins.<name> section that downstream consumers read. - Config.Plugins holds repo-level, plugin-specific defaults keyed by the consuming plugin name. - Project.Plugins holds the per-project raw_options blob - the persisted, user-editable options a plugin owns. Config never interprets it. Both are map[string]map[string]any and opaque to config: the plugin that matches the key owns the schema. Mark the structured terraform.* / aws.* fields (var_files, vars, source_map, aws region/stack_id/stack_name/account_id) deprecated, pointing each at its plugins.<name> home. They are still read for backwards compatibility. terraform.workspace and terraform.cloud stay typed top-level fields (not folded): they are caller-sourced and read outside the plugin, so they travel via GenericOptions rather than the opaque blob.
Populate the plugins.<name> blob during config generation and carry the plugin-authored raw_options through identification. - plugin.Environment and autodetect.Project gain a RawOptions []byte carried from the IdentifyEnvironments RPC (proto bumped to v1.158.0 for the field). - Generation persists options under plugins.<name>, keyed by the consuming plugin (CDK folds into cloudformation via pluginKeyForType). When a plugin authored a raw_options blob it is persisted verbatim; otherwise, for the terraform family, config sideloads the var files it attributed locally into plugins.<name>.tfVarsFiles. This sideload is the one deliberate special case, expected to go away once IdentifyAllProjects moves that attribution plugin-side. The deprecated terraform.var_files field is no longer written. - terraform.workspace stays a top-level field (caller-sourced, read outside the plugin), not part of the blob. - ConfigSHA now hashes the plugins blob so generated configs still reflect var-file changes once terraform.* is no longer written.
Persist and read plugin parse options as one heading per plugin (terraform:, aws:, kubernetes:, ...) with no `plugins:` block, while the in-memory API keeps a plugin-keyed Plugins blob. The config file stays unbreaking; only where fields live in the API moves. configfile package (file-syntax layer): - configfile.Config/Project model the on-disk shape as structural fields plus opaque `Headings` captured as raw yaml.Node via custom (Un)MarshalYAML. It knows nothing about heading semantics and imports only yaml, so there is no import cycle with package config. Semantic bridge (fromConfigFile/toConfigFile) + Config.(Un)MarshalYAML: - On read, each heading is split: terraform: -> typed workspace/cloud/spacelift on ProjectTerraform + everything else -> Plugins[type]; aws: -> Plugins[cloudformation]; other headings -> Plugins[name]. On write, the inverse merges typed + blob back into one heading. Config.(Un)MarshalYAML route every yaml (un)marshal through this layer (generation re-parse, env-var replacement, callers), so output is always the heading shape and the Plugins yaml tags are dropped. - Keying helpers (blobKeyForHeading/headingForBlobKey/terraformBlobKey) collapse the terraform family to the terraform: heading and map aws<->cloudformation, matching how the CLI/runner look the blob up. Legacy 0.1/0.2 (configlegacy.go): - Relocated verbatim out of config.go / config_file.go into a self-contained, deletable file, producing *Config directly (same package, no cycle). It folds its typed var files/vars/aws into the blob so consumers find them and the env-var round-trip preserves them. Deprecated typed fields are populated mirrors: - Terraform.VarFiles/.Vars and AWS are filled from the canonical blob on load (mirrorBlobToDeprecated); the blob is the source of truth and the only thing serialized. ConfigSHA hashes the blob, not the mirrors. Repo-level defaults (one convention for every plugin): - Repo-level plugin defaults live under <plugin>.defaults:, matching terraform's existing terraform.defaults. foldRepoPluginDefaults deep-merges them into each project they apply to, per-project wins (terraform-family share the terraform default; cloudformation covers cdk). terraform.defaults splits like a project's terraform: heading - typed keys (cloud/spacelift/workspace) into TerraformDefaults, everything else into the terraform blob default; because the split extracts the typed keys, they and the blob keys are disjoint and recombine under defaults: on write with no duplicate key. A repo-level key placed directly (not under defaults:) is not treated as a default. - The deprecated terraform.source_map (a list) is transformed into each terraform-family project's regex_source_map (a map) blob key - the new CLI/runner read only the blob, so this avoids regressing existing source maps. Validation: an unknown top-level/project key whose value is not a mapping is a typo of a structural field (a plugin blob is always a mapping), so it is rejected with a clear error rather than silently captured - restoring typo detection without needing a plugin list. ConfigWithAutodetect is removed (configfile captures the autodetect: node itself). Config version stays 0.3.
liamcervante
force-pushed
the
plugin-options
branch
from
July 14, 2026 13:35
f5a62d0 to
cdea9fb
Compare
liamcervante
marked this pull request as ready for review
July 14, 2026 13:47
liamg
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes plugins the source of truth for their own parse options. Config persists each project's options as an opaque
raw_optionsblob keyed by the consuming plugin and forwards it verbatim — it never interprets the blob. Anything read outside the plugin (terraform workspace, TF-cloud identity, credentials) stays a typed field. The config file stays unbreaking — only where a field lives in the in-memory API moves; the on-disk shape is unchanged. Config version stays0.3.Pairs with infracost/parser#185 (plugins emit/read
raw_options), infracost/cli#185 and infracost/runner#1303 (read the blob + typedGenericOptions), and infracost/proto#82 (GenericOptions.workspace+terraform_cloud_configuration).Design
Three layers, cleanly separated:
configfilepackage (file-syntax layer) — models the on-disk shape only: structural fields + opaqueHeadingscaptured as rawyaml.Nodes. Knows nothing about heading semantics; imports only yaml, so there is no import cycle with packageconfig.fromConfigFile/toConfigFile) +Config.(Un)MarshalYAML— interprets the headings. On read, each heading is split (typed keys → typed fields; everything else →Plugins[key]blob); on write, the inverse recombines them into one heading. Routing every yaml (un)marshal through this layer means output is always one-heading-per-plugin with noplugins:block.configlegacy.go) — relocated verbatim into a self-contained, deletable file that produces*Configdirectly.How options are classified
workspace/cloud/spacelift) → travel viaGenericOptions/credentials, extracted out of the blob.raw_options.terraform/terragrunt/cisco_stacks) shares theterraform:heading on disk but keys the blob by exact project type;aws:⇄cloudformation(CDK resolves there too).Repo-level defaults (one convention for every plugin)
Repo-level plugin defaults live under
<plugin>.defaults:, matching terraform's existingterraform.defaults. They deep-merge into each project they apply to, per-project winning.terraform.defaultssplits typed/blob just like a project heading (the typed keys are extracted, so they and the blob keys are disjoint and recombine underdefaults:on write with no duplicate key). The deprecatedterraform.source_map(a list) is transformed into each terraform-family project'sregex_source_map(a map) blob key.Backwards compatibility
Terraform.VarFiles/.Vars,AWS) are populated on load as read-only mirrors of the canonical blob.Testing
go build ./...,go vet ./...,gofmtclean.go test ./...green, including the_WithPluginstests that run against the real downloaded plugin binaries.defaults:merge, legacy fold, and the non-mapping-key rejection.Notes for reviewers
/code-review; findings addressed (vars-mirror aliasing, non-mapping-key validation) or dissolved by the repo-leveldefaults:redesign (the duplicate-key marshal case).