Please build rule plugins

[Tatskaari]

Please tries to provide first class support for third party build rule authors. There's very little difference between the built in rules vs. rules written externally, however there's still some shortcomings. Most of this is around configuration. In this issue, I outline some of these shortcomings and suggest a formalised approach to third party rule repos called plugins. The long term plan for Please is to move all the bundled languages out of Please and into these plugins.

Terminology

Host repo - The repo that is including the plugin
Subrepo - Please has the ability in embed another repo inside the host repo. This can be done through the subrepo() builtin. The other repo is usually a Please repo, however it can also be a non-Please repo in which case we provide some build rules to interact with it. In either case, the repo is added to the host repo's build graph and can be access through a special type of build label with the @subrepo_name//some:target or ///subrepo_name//some:target syntax.
Plugin repo - A type of subrepo that is intended to extend Please with some additional functionality e.g. to support another language. These typically more advanced than other subrepos as they often need to define configuration to be set from the host repo.

Plugin configuration

As it stands, rule authors are expected to set default values for their configuration in the build definitions:

# @foo//build_defs:foo

def foo_library(...):
    return build_rule(
        ...
        tools = [CONFIG.FOOC_TOOL],
    )
 
CONFIG.setdefault("FOOC_TOOL", "fooc")

These values can then be overridden in the host repo's .plzconfig:

[BuildConfig]
fooc-tool = @foo//tools:fooc ; Notice the change in casing here vs. the built in rules

This approach is rather primitive and has some shortcomings when compared to the built in rules:

• There's no way to define the mapping between the key in the build language to the key in the config file. This mapping isn't always what you'd want or expect, i.e. keys would tpyically map from FOOC_TOOL to FoocTool which isn't possible with this approach.
• There's no way to add a config value which don't have a default value, or are mandatory.
• There's no name-spacing between sections. Using [BuildConfig] for everything could potentially result in config key collisions between plugins but mostly just feels second class.

I propose adding new config sections to define plugin's configuration. On the plugin side, plugins will be able to define a namespace for their configuration in their .plzconfig:

[PluginDefinition]
name = foo ; Creates a new configuration namespace for this plugin. 

We can then add fields that this plugin requires through a repeatable "PluginConfig" section:

[PluginConfig "FOOC_TOOL"] ; This key here matches the key in the build lang i.e. "CONFIG.FOO.FOOC_TOOL"
ConfigKey = FoocTool ; This determines the key that can be used in the host repo's .plzconfig file 
DefaultValue = fooc ; This field is optional. If not set, Please will create the key in the CONFIG map but set it to `None`
Optional = false ; Whether this config value can be empty i.e. None

The host repo can then specify values to these config values in their .plzconfig through a new Plugin section:

[Plugin "foo"] ; The name here must match the plugin name in the plugin's `.plzconfig` above
FoocTool = @foo//tools:fooc ; Note that the key here matches the format you would expect coming form the built in rules

The host repo is then expected to include the plugin as a subrepo e.g. though the github_repo() rule:

//plugins:foo
github_repo(
    name = "foo",
    repo = "thought-machine/please-foolang",
    revision = "v1.0.0",
)

This plugin can then be used via subinclude("@plugins/foo//build_defs:foo") or similar.

Validation

Typically, if extra config values are provided in a .plzconfig, Please will log warnings for these. This is because we need to parse older config files when querying changes between branches. This isn't the case for plugins as plugins can be updated in lock step with their .plzconfig files. I propose that problems in config for plugins should be fatal i.e. we should log errors, and stop the build in the case of unrecognised or missing plugin config.

The build rules can then perform additional validation to make sure the values for these config fields make sense. They can use fail() and warning() to report any problems to the user.

Referring to the current subrepo

Another difficulty in subrepos is it can be hard to set default config values that work both from inside the plugin repo, and from the host repo. For example, say we have some sort of language auto-formatter //tools:foo_fmt defined inside out plugin repo. We might want to set this as a default value to avoid every consumer having to set this manually. To do this we need to know the subrepo name that the host repo has included us as. To solve this, I propose we implement @self as a way to refer to the current subrepo name, or nothing in the case that we aren't inside a subrepo:

[PluginConfig "...."] 
...
DefaultValue = @self//tools:... 

Plugin and host repo config

Currently configuration values set in the plugin repo's .plzconfig are not respected when included from the host repo. This can be desirable, however it's usually not. For example, it doesn't make sense for the plugin to use the same go import path as the host repo, however we would want to use the same caching configuration.

I propose that we should start off with the host repos configuration as a base as we do now. We should then parse the subrepo's .plzconfig, allowing it to override values like the go import path in their .plzconfig. This will allow the plugin repo to override config values like the go import path while still using the other config from the host.

Plugins might also have version requirements on Please. We should parse the plugins .plzconfig to check the Please.Version field. A miss matched Please version might work. For this reason, if the versions don't match up, we should log a warning but still try our best to complete the build.

Test configuration

The last piece of the puzzle is when the plugin repo wants to set configuration values that are only useful when testing the plugin repo. These values shouldn't take effect when it's included as a subrepo. For example, a plugin may want to set up some SDK or toolchain so that it can run its tests without having to install the SDK on the CI server. We don't want to download this SDK again if the host repo already has one configured, so we should use the host repo's configuration over the plugins for the SDK.

Developers could work around this issue by having a .plzconfig.ci profile or similar. They could put the SDK or toolchain configuration in there, allowing them to separate this config from the main .plzconfig. This isn't very user friendly as working in the plugin repo would require passing --proflie ci to every plz invocation. To mediate this, we can codify this pattern in a .plzconfig_private file. This file will essentially be the config that is private to the subrepo, as such, it will only be loaded by plz when we're working directly in the plugin repo.

A language agnostic build system

Going forward, I plan to move the languages in the Please repo out into plugins. If the plugin API is truly first class, then this is a good way to prove it. It will also allow us to make breaking changes to the rule definitions without waiting for a major release of Please. At this point, we can say that Please is a truly language agnostic build system.

Some consideration into a testing plan should be made to make sure that changes to Please don't break plugins (as is the case with pleasings quite often). I think most of the e2e tests should stay in the Please repo, to make sure that Please integrates with the tips of these plugins.

[namtzigla]

my 2 questions here:

1. what happens when the plugin repo has a please version requirement different then the importing repo? (IMO a solution should be to remove from please (almost) all builtin rules and then ignore the subrepo please version requirement and hope for the best)
2. is there a please version target when the subrepo rules will be revisited (implementing @self)? I like your @self idea, that would solve so many issues w/ subrepos!

Thanks

[Tatskaari]

Thanks for the feedback.

1. We thought about that but somehow didn't make it into this RFC. We planned to log a warning as, like you said, it might work so it's best to try. If something goes wrong, the warning could be a useful hint to the solution.
2. I plan to get these features in place before we start on v17. The plan is to get all the bundled rules out into plugins. The bundled rules will then be thin wrappers around the plugins or something similar to maintain backwards compatibility. In v17 the bundled rules will be removed, and users will need to specify the language plugins they use and what version.