config: envoy gateway config generation

[nacx]

Related to #25

We should also generate the Envoy Gateway CRDs to load the desired extensions, instead of just raw Envoy config.

In order to implement this, we may need to add support for more extensions to the EnvoyExtensionPolicy because right now it only supports lua, wasm and extProc.

---

We should also debate if this should be in this project or we could even add such a feature to the Envoy Gateway project. Its CLI could actually embed the built-on-envoy CLI to provide a low-friction experience to current EG users.

Another option would be to have our custom CRDs in this project and the corresponding controller that could be installed on top of EG (like we do in EAIGW). The CRD could be something like:

apiVersion: extension.tetratelabs.io/v1alpha1
kind: RemoteExtension
metadata:
  name: my-custom-extension
  namespace: default
spec:
  name: my-custom-extension
  version: 1.0.0
  priority: 1
  config: |
    (...)
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: default

The controller could read this CRD, download the extension as needed, and configure the gateways to execute it. It could take care of mutating the pods if files need to be mounted, etc, etc. Ideally, we should not be using the Extension Server, as EAIGW already uses it, and EG can have only 1 defined, so we need to figure out if it is feasible for the controller to, for example, generate EnvoyPatchPolicies based on these CRDS (I believe it should be feasible?).
We can have this CRD and controller live on this repository, and contribute to EG upstream later, as it is not feasible to have this merged in EG for the launch date.

[nacx]

cc @zirain @zhaohuabing sharing early for visibility and ideas. This is just a proposal, but we need to figure out what would be a low-friction experience for users to very easily start using this on their existing setups.

[zirain]

can we use EPP instead of EEP?

[zhaohuabing]

We should also debate if this should be in this project or we could even add such a feature to the Envoy Gateway project. Its CLI could actually embed the built-on-envoy CLI to provide a low-friction experience to current EG users

This project could be a hub, a subcommand within egctl could be useful, like:

egctl plugin list --- list available plugins from the plugin hub
egctl plugin enable xxxx on Gateway/Route. --- enable a plugin on a Gateway/Route (generated EEP, EnvoyProxy sidecar config for extproc, etc.)

[zhaohuabing]

EG issue for DM support: envoyproxy/gateway#5668
Is there an expected time for Composer to be part of the official envoy Github project? cc @nacx

[nacx]

can we use EPP instead of EEP?

Yes, that would be the trivial way to make it work, but EPPs are hard to maintain and understand by the regular user, and error-prone. Since the whole idea of this project is to bring simplicity, I think we should aim for a high-level CRD. Since it is unrealistic to have that on EG initially, I proposed having it here.

[zirain]

EEP is not the best choice as it's disabled by default. There's some security complain when using in multi-tenaut, see envoyproxy/gateway#8102. A new high-level CRD may have the same.

[nacx]

Idea may be to have the custom CRD initially on TEG, and start with an EnvoyPatchPolicy in BOE to avoid depending on EG initially. We can start with basic support to generate an EPP, then work in parallel on TEG on the controller and proper high-level experience, but at least we should have something here, even if it is as basic as an EPP.

@zhaohuabing @zirain would you be able to work on this EPP thing (feel free to self-assign!)?

Ideally, we should be able to generate the config and generate the EPP for the configured filters, and additionally print some instructions to help the user edit it if needed to adjust names, etc (although the command to generate the config could accept more parameters if that facilitates generating a correct EPP).