[experiment] First approach to extensible shell with plugins

[marc-gr]

For some time there have been discussions about how to extend elastic-package capabilities in a way that does not require changes in elastic-package codebase (ref #479).

I have explored this approach of an interactive shell that can be extended with Go plugins, for example to extend it with adhoc commands to do bulk operations, or to add other experimental capabilities to be tested.

This first experimental approach:

1- Describes a Command interface that the plugins need to follow.
2- Adds a shell command that starts an interactive shell with autocomplete and history capabilities that will load dynamically any commands found in ~./elastic-package/shell_plugins
3- Adds some example commands under pkg/shell/plugins.

The idea behind that would be to, if moved forward, have a new ie plugins repository that could be built independently and which artifacts would be placed in the before mentioned directory.

A requirement for this would be to move from internal to ie pkg some of the most used packages that can interact with packages to allow importing them from the plugins repository.

This setup would allow for much faster experimentation of new functionalities and to have it more isolated. If at some point we want to move some of this plugins to be builtin it should be a straightforward process since they will be using the elastic-package dependencies.

How to test it:

1- Fetch this branch
2- Install elastic-package $ make install or create the shell_plugins directory if already installed mkdir ~/.elastic-package/shell_plugins
3- Build the test plugins cd pkg/shell/plugins; go build -buildmode=plugin ./...; mv plugins.so ~/.elastic-package/shell_plugins
4- In integrations, run elastic-package shell
5- Play with the provided commands

github.com/elastic/integrations  main ✔                                                                                                                                                                                                                                  21h21m  
▶ elastic-package shell
Loaded 4 plugins
> shell initdb
Initializing database...
Loaded 236 packages
> shell where "json_extract(manifest, '$.owner.github') = 'elastic/security-external-integrations'"
Found 146 packages
> shell write-file --path "kibana/tags.yml" --contents '- text: Security Solution\n  asset_types:\n    - dashboard\n    - search\n'
> shell changelog --next minor --type enhancement --link https://github.com/elastic/integrations/pull/9999999 --description "Add tags.yml file."
> shell 

[marc-gr]

I also experimented with https://github.com/hashicorp/go-plugin to avoid some of the brittleness issues native plugins have.

Pros from RPC plugins:

1- less brittle (no need to be compiled with same versions of toolchain and dependencies)
2- more portable, since native plugins are not available in windows
3- safer, since they are safe to crash on their own without affecting the original application

Cons:

1- The usable interface for the commands is much more limited. No possibility to send any types over RCP, meaning no context.Context, pflags.Flags, etc can be used since they return or expect any parameters. We would need to define a more limited scope for what a plugin can do.

Native plugins are more convenient if we find reliable automation to solve their issues and we acknowledge the lack of WIndows support, and RPC ones are easier to maintain if we get to a good middle ground for the commands interfaces at use.

[marc-gr]

Added one more commit with an RPC implementation.

Currently it has several limitations compared with the original one:

1- Shared context.Context is limited only to the commands belonging to the same Plugin. I do not see this as an issue in the future but if so, we can come with some RPC implementation for context.Context or similar.
2- Flags are parsed on the plugin, meaning no autocomplete for flags in the shell. Again, no big deal.

After building the boilerplate creating new commands is straightforward, and overall I think this is a much more robust approach.

How to try it:

1- Fetch this branch
2- Install elastic-package $ make install or create the shell_plugins directory if already installed mkdir ~/.elastic-package/shell_plugins
3- Build the test plugins cd pkg/shell/plugins; go build ./...; mv plugins ~/.elastic-package/shell_plugins
4- In integrations, run elastic-package shell
5- Play with the provided commands

[marc-gr]

Extracted the required internal packages to pkg and created https://github.com/elastic/ep-shell-plugins to prove the final setup.

[P1llus]

This addition is great! really looking forward to see something like this getting merged at some point :)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ce00f4b

History

• 💔 Build #1632 failed 1cdb985
• 💔 Build #1631 failed c29bf72
• 💚 Build #1545 succeeded b291256
• 💚 Build #1535 succeeded 80f344d
• 💚 Build #1453 succeeded 5b17239

cc @marc-gr