Skip to content

Commit e8df06b

Browse files
smarchonesmarchone
authored
docs : add yaml plugin (#564)
* docs : add yaml plugin docs * fix docs for backward compatiblity support
1 parent f7ef0bd commit e8df06b

File tree

1 file changed

+151
-13
lines changed

1 file changed

+151
-13
lines changed

‎docs/docs/development/building-plugin.md‎

Lines changed: 151 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,16 @@ Optimus's responsibilities are currently divided in two parts, scheduling a tran
77

88
Whereas tasks used in jobs that define how the transformation will execute, what configuration does it need as input from user, how does this task resolves dependencies between each other, what kind of assets it might need. These questions are very open and answers to them could be different in different organisation and users. To allow flexibility of answering these questions by developers themselves, we have chosen to make it easy to contribute a new kind of task or even a hook. This modularity in Optimus is achieved using plugins.
99

10-
> Plugins are self-contained binaries which implements predefined protobuf interfaces to extend Optimus functionalities.
11-
1210
Optimus can be divided in two logical parts when we are thinking of a pluggable model, one is the **core** where everything happens which is common for all job/datastore, and the other part which could be variable and needs user specific definitions of how things should work which is a **plugin**.
1311

12+
Currently Optimus plugin can be implemented as binary executable.
13+
And `support for yaml` implementation of plugins is also introduced with limited
14+
scope (discussed below).
15+
16+
Optimus Plugin is just an adapter between optimus and what actually needs to be executed. Actual transformation will be packed in a docker image and Optimus will execute these arbitrary docker images as long as it has access to reach container registry. Plugin provides the optimus server with the info about the docker image.
17+
18+
> Plugin itself is not executed for transformation but only used for adapting conditions which Optimus requires to be defined for each task.
19+
1420
## Types of Plugins in Optimus
1521
At the moment mainly there are two types of plugins which optimus supports. These are : ***Hook*** and ***Task***
1622
Before getting into the difference between two plugins ,we need to get familiar with [Jobs](../concepts/overview.md#Job).
@@ -22,17 +28,108 @@ Before getting into the difference between two plugins ,we need to get familiar
2228
| Configuration | It has its own set of configs and asset directory. | It has its own set of configs and share the same asset directory across all hooks as the base job. |
2329

2430

25-
## Creating a plugin
31+
## Supported Use-Cases of Plugins in Optimus
2632

27-
At the moment Optimus supports task as well as hook plugins. In this section we will be explaining how to write a new task although both are very similar. Plugins are implemented using [go-plugin](https://github.com/hashicorp/go-plugin) developed by Hashicorp used in terraform and other similar products.
33+
* Plugin Info Usecases:
34+
* Discover all plugins and list their info.
35+
* Refer - `optimus version` (lists all the plugins available),
36+
* Project Side Usecases :
37+
* Survey to populate job specifications and assets.
38+
* Plugins provide the questionare and default values (default assets for job) to the survey implemtnation in optimus.
39+
* Refer - `optimus job create`
40+
* Server Side Usecases :
41+
* CompileAssets & DependencyResolver
42+
* Theses are currently supported server side behaviour that is delegated to plugins implementations.
43+
* Refer - [transformers](https://github.com/odpf/transformers/blob/main/task/bq2bq/main.go#L274)
44+
45+
## Binary Implementation of Plugin
46+
47+
Binary implementation of Plugins are self-contained binaries which implements predefined protobuf interfaces to extend Optimus functionalities.
48+
Binary Plugins are implemented using [go-plugin](https://github.com/hashicorp/go-plugin) developed by Hashicorp used in terraform and other similar products.
2849

2950
> Plugins can be implemented in any language as long as they can be exported as a single self-contained executable binary and implements a GRPC server.
3051
31-
It is recommended to use Go currently for writing plugins because of its cross platform build functionality and to reuse protobuf sdk provided within Optimus core. Although the plugin is written in Go, it will be just an adapter between what actually needs to be executed. Actual transformation will be packed in a docker image and Optimus will execute these arbitrary docker images as long as it has access to reach container registry.
52+
It is recommended to use Go currently for writing plugins because of its cross platform build functionality and to reuse protobuf sdk provided within Optimus core.
53+
54+
> Binary Plugins can potentially modify the behavior of Optimus in undesired ways. Exercise caution when adding new plugins developed by unrecognized developers.
55+
56+
57+
## Yaml Implementation of Plugin
58+
Most plugins are expected to implement just the info and project side use-cases (mentioned above) and thease are data-driven i.e., plugin just provide data to optimus.
59+
To simplify the development process of plugins, support for yaml mode of defining plugins is added.
60+
61+
```go
62+
// representation of a yaml plugin schema in golang
63+
64+
// below struct definition in golang can be marshalled
65+
// to generate yaml plugins
66+
67+
type YamlPlugin struct {
68+
// info use-case
69+
Name string `yaml:"name"`
70+
Description string `yaml:"description"`
71+
Plugintype string `yaml:"plugintype"`
72+
Pluginversion string `yaml:"pluginversion"`
73+
Image string `yaml:"image"`
74+
Secretpath string `yaml:"secretpath"`
75+
76+
// survey use-case
77+
Questions []struct {
78+
Name string `yaml:"name"`
79+
Prompt string `yaml:"prompt"`
80+
Help string `yaml:"help"`
81+
Regexp string `yaml:"regexp"`
82+
Validationerror string `yaml:"validationerror"`
83+
Minlength int `yaml:"minlength"`
84+
Required bool `yaml:"required,omitempty"`
85+
Maxlength int `yaml:"maxlength,omitempty"`
86+
Subquestions []struct {
87+
Ifvalue string `yaml:"ifvalue"`
88+
Questions []struct {
89+
Name string `yaml:"name"`
90+
Prompt string `yaml:"prompt"`
91+
Help string `yaml:"help"`
92+
Multiselect []string `yaml:"multiselect"`
93+
Regexp string `yaml:"regexp"`
94+
Validationerror string `yaml:"validationerror"`
95+
Minlength int `yaml:"minlength"`
96+
Required bool `yaml:"required,omitempty"`
97+
Maxlength int `yaml:"maxlength,omitempty"`
98+
} `yaml:"questions"`
99+
} `yaml:"subquestions,omitempty"`
100+
} `yaml:"questions"`
101+
102+
// default-static-values use-case
103+
Defaultassets []struct {
104+
Name string `yaml:"name"`
105+
Value string `yaml:"value"`
106+
} `yaml:"defaultassets"`
107+
Defaultconfig []struct {
108+
Name string `yaml:"name"`
109+
Value string `yaml:"value"`
110+
} `yaml:"defaultconfig"`
111+
}
112+
```
113+
refer to sample implemenation [here](#2a-creating-a-yaml-plugin)
114+
### Limitations of Yaml plugins:
115+
116+
Here the scope of yaml plugins is limited to drive survey, provide default values for job config and assets and provide plugin info. As majoiry of the plugins are expected to implement subset these use-cases, the support for yaml definitions for plugins is added which simplifies development, packaging and distribution of plugins.
117+
118+
For plugins that require to enrich optimus server side behaviour, yaml definitions falls short as this would require some code.
119+
120+
### Validating Yaml plugins:
121+
Also support for validating yaml plugin is added into optimus.
122+
After creating yaml definitions of plugin, one can validate them as below:
123+
```bash
124+
optimus plugin validate --path {{directory of yaml plugins}}
125+
```
32126

33-
> Plugin binary itself is not executed for transformation but only used for adapting conditions which Optimus requires to be defined for each task.
127+
** Note: The yaml plugin is expected to have file name as `optimus-plugin-{{name}}.yaml`
34128

35-
To demonstrate this wrapping functionality, let's create a plugin in Go and use python for actual transformation logic. You can choose to fork this [example](https://github.com/kushsharma/optimus-plugins) template and modify it as per your needs or start fresh. To demonstrate how to start from scratch, we will be starting from an empty git repository and build a plugin which will find potential hazardous **Near Earth Orbit** objects every day, let's call it **neo** for short.
129+
** Note: If Both yaml and binary plugin with same name are installed, Yaml implementation is prioritised over the corresponding counterparts in binary implemenation.
130+
## Creating Plugin : Tutorial
131+
132+
To demonstrate the above mentioned wrapping functionality, let's create a plugin in Go as well as a yaml definition and use python for actual transformation logic. You can choose to fork this [example](https://github.com/kushsharma/optimus-plugins) template and modify it as per your needs or start fresh. To demonstrate how to start from scratch, we will be starting from an empty git repository and build a plugin which will find potential hazardous **Near Earth Orbit** objects every day, let's call it **neo** for short.
36133

37134
Brief description of Neo is as follows
38135

@@ -43,7 +140,7 @@ Brief description of Neo is as follows
43140
- Output of this object count can be printed in logs for now but in a real use case can be pushed to Kafka topic or written to a database.
44141
- Plugin will be written in **Go** and **Neo** in **python**.
45142

46-
### Preparing task executor
143+
### 1. Preparing task executor
47144

48145
Start by initialising an empty git repository with the following folder structure
49146

@@ -199,12 +296,35 @@ RUN pip install -r requirements.txt
199296

200297
CMD ["python3", "main.py"]
201298
```
299+
### 2a. Creating a yaml plugin
300+
The Yaml implementation is as simple as providing the plugin details as below.
301+
```yaml
302+
name: Neo
303+
description: Near earth object tracker
304+
plugintype: task
305+
pluginversion: latest
306+
image: ghcr.io/kushsharma/optimus-task-neo-executor
307+
secretpath: /tmp/.secrets
308+
309+
questions:
310+
- name: RANGE_START
311+
prompt: Date range start
312+
help: YYYY-MM-DD format
313+
required: true
314+
- name: RANGE_END
315+
prompt: Date range end
316+
help: YYYY-MM-DD format
317+
required: true
318+
```
319+
Based on the usecase, additional validation can be added to the questions section. Refer [above](#yaml-implementation-of-plugin) for more usecases and features provided.
202320
321+
### 2b. Creating a binary plugin
203322
323+
At the moment Optimus supports task as well as hook plugins. In this section we will be explaining how to write a new task although both are very similar.
204324
205325
Now that base image is ready for execution, this needs to be scheduled at a fixed interval using `jobs` but for optimus to understand **Neo** task, we need to write an adapter for it.
206326

207-
### Implementing plugin interface
327+
#### Implementing plugin interface
208328

209329
Because we are using Go, start by initialising Go module in `neo` directory as follows
210330

@@ -591,19 +711,37 @@ Use this [repository](https://github.com/kushsharma/optimus-plugins) as an examp
591711
### Installing a plugin
592712

593713
Plugins need to be installed in Optimus server before it can be used. Optimus uses following directories for discovering plugin binaries
594-
595-
```shell
714+
```
715+
./.plugins
596716
./
597717
<exec>/
598718
<exec>/.optimus/plugins
599719
$HOME/.optimus/plugins
600720
/usr/bin
601721
/usr/local/bin
602722
```
723+
Even though the above list of directories are involved in plugin discovery, it is advised to use `.plugins` in the current working directory of the project or optimus binary.
603724
604-
If Optimus cli is used to generate specifications or deployment, plugin should be installed in a client's machine as well.
725+
### Server-side installation
726+
On Server side, plugins (both binary and yaml) can be installed declaratively at runtime by listing the plugin artifacts in plugins section of server config.
727+
```bash
728+
optimus plugin install -c config.yaml # This will install plugins in `.plugins` folder.
729+
```
730+
Plugin section to be added in server config:
731+
```yaml
732+
plugin:
733+
artifacts:
734+
- https://...path/to/optimus-plugin-neo.yaml # http
735+
- http://.../plugins.zip # zip
736+
- ../transformers/optimus-bq2bq_darwin_arm64 # relative paths
737+
- ../transformers/optimus-plugin-neo.yaml
738+
```
739+
### Project-side installation
740+
On the project side, where optimus cli is used to generate specifications or deployment, optimus cli can sync the yaml plugins supported-and-served by optimus server (with host as declared in project config).
605741
606-
> Plugins can potentially modify the behavior of Optimus in undesired ways. Exercise caution when adding new plugins developed by unrecognized developers.
742+
```bash
743+
optimus plugin sync -c optimus.yaml # This will install plugins in `.plugins` folder.
744+
```
607745

608746
### Using in job specification
609747

0 commit comments

Comments
 (0)