Create 'Service' CRD to register and reference k8s and non-k8s compute resources

[delqn]

The intent of this PR is to begin a discussion and propose the addition of a new CRD Service.

The purpose of the new service.specs.smi-spec.io is to register and uniquely identify a component (service) within a service mesh, which spans multiple Kubernetes clusters, and also includes non-Kubernetes compute resources. The component could be anything resolvable to a list of IP addresses and a port:

• Kubernetes service
• DNS record
• Virtual Machines within a cloud provider

Once a service is registered, foo.mesh for instance, it will be referenced by TrafficTarget (example below), TrafficSplit, etc.

For each type of Service, the service mesh needs to implement a resolver. The new CRD's duty is to provide sufficient and unambiguous information to the service mesh controller, so that it can obtain:

• list of IP addresses for the given service
• TLS certificate

The proposal is to augment the existing spec and add the optional ability to reference a Service, not replace references to ServiceAccount. For service meshes where a single Kubernetes cluster is involved - kind: ServiceAccount (illustrated in TrafficTarget) is sufficient. For multiple Kubernetes clusters forming a mesh however, ServiceAccount is not specific enough - we need to identify the Kubernetes cluster and ideally also the Kubernetes service within that service account.

In the proposal below we deliberately skip the explicit indication of a ServiceAccount. Identity discovery for each registered service would be an implementation detail of the particular service mesh. In the case of Service with type: Kubernetes, there could be a 1:1 relationship between Kubernetes Service and ServiceAccount - the service mesh would expect the Service to belong to a ServiceAccount with the same name.

Example

In the example below we register 3 unique services:

• foo.mesh, which is resolvable via DNS
• bar.mesh, which is a unique name alias of a Kubernetes Service on a uniquely identified Kubernetes cluster and the given name space
• baz.mesh - referencing a virtual machine scale set within a specific Azure subscription, along with a port

The TrafficTarget in this case declares that services foo.mesh, bar.mesh, and baz.mesh (as well as prometheus) are allowed to access service-a on port 8080 and at the given URL endpoints.

To show a simpler example:

kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha1
metadata:
  name: bar-to-baz
destination:
  kind: Service
  name: baz.mesh
specs:
- kind: HTTPRouteGroup
  name: the-routes
  matches:
  - metrics
sources:
- kind: Service
  name: bar.mesh

The policy above declares that bar.mesh can access baz.mesh.

To be more specific - the policy declares that bar.mesh, which is the pods forming Kubernetes Service bar, in the default namespace on Kubernetes cluster aks-4d61b17c.hcp.westus2.azmk8s.io, can access service baz.mesh, which is a virtual machine scale set on Azure uniquely identified by /resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz.

Existing work

I see that @nicholasjackson is already working on something very similar (#60) and would love to combine the 2 efforts.

Note on the code changes

I don't intend on actually merging the specific proposed changes below. Once we have achieved consensus, I will augment this PR (or create a new one) with the actual CRD + documentation and examples. For the time being I am going to temporarily leverage the existing example of TrafficTarget for better illustration.

Note on naming

Even though it is name-spaced under specs.smi-spec.io, the name Service may be confusing since it conflicts with the existing core Kubernetes resource. Looking forward to suggestions on better naming this CRD.

[grampelberg]

Could you describe the problem you're solving a little bit more? It sounds like you're looking to do virtual host style stuff?

Couple other comments:

• As of right now, TrafficTarget is managed on the server side. This would suggest that it can also be applied on the client side. While that makes sense, I think we'll want to make sure we're super explicit about that.
• I'm not a fan of Service, will make it tough to talk about. I think we can get more specific on what it is called one we're a little closer to the problem being solved.
• I love the examples, adding locators is an interesting idea. I'm hesitant to add a single resource that is so overloaded by types. As TrafficTarget has an object reference, why can't we just use the built in k8s objects for this?

[delqn]

@grampelberg Thanks for taking the time to read through this. Responses inline:

Could you describe the problem you're solving a little bit more? It sounds like you're looking to do virtual host style stuff?

At the moment SMI spec is focused on Kubernetes. I'd like to extend the spec so I can declare a Service Mesh, which spans:

• multiple Kubernetes clusters
• Kubernetes clusters and Virtual Machines
• Virtual Machines (service mesh with no Kubernetes clusters)

The proposal is to uniquely identify a service running on a given k8s cluster and reference that in the existing SMI CRDs. I think of this as a declarative way to create a new service registry entry.

• As of right now, TrafficTarget is managed on the server side. This would suggest that it can also be applied on the client side. While that makes sense, I think we'll want to make sure we're super explicit about that.

I picked TrafficTarget to illustrate how the new Service CRD would be referenced. I am not suggesting we change how or where TrafficTarget is enforced. I picked TrafficTarget because it currently references ServiceAccount in the examples. To define a service mesh across multiple Kubernetes clusters, containing service accounts of the same name, I would need more context to identify the pods being referenced.

• I'm not a fan of Service, will make it tough to talk about. I think we can get more specific on what it is called one we're a little closer to the problem being solved.

Would love to come up with a better name for this CRD, once we have established that this declarative service registry is good for the community. Another idea was MeshService. ServiceEntry already exists and serves similar purpose.

• I love the examples, adding locators is an interesting idea. I'm hesitant to add a single resource that is so overloaded by types. As TrafficTarget has an object reference, why can't we just use the built in k8s objects for this?

It would have been great to reuse the existing Kubernetes Service. It has certain limitations (selector could be no more than 63 characters, labels can't contain slashes etc). It seems awkward to try to retrofit it to be able to address a VM, which is not part of a k8s cluster.

[grampelberg]

Ahhha! This is great. Let's separate these all into separate pieces:

multiple Kubernetes clusters

I've been spending a bunch of time thinking about this one recently. There are two pieces to this one:

• service discovery - for this, I think k8s Service is actually the correct resource here. It allows for fallback on applications not in the mesh and allows folks to build operators that are k8s generic.
• identity - this is gonna require a new resource, something that configures the remote clusters and identity to use for those. I'd love to get into this conversation around egress as it feels like there's the opportunity to solve this for third party services as well, imagine github.com.

Kubernetes clusters and Virtual Machines

This is the one I'm probably the weakest on. What problem really needs solving here? If it is service discovery, I'm thinking the classic k8s Service works. I'm reasonably sure I'm missing a bunch of the subtle problems though!

Virtual Machines (service mesh with no Kubernetes clusters)

As SMI is currently defined as CRDs for k8s, should this one be out of scope?

[delqn]

@grampelberg I'm so glad to hear that you are thinking about multi-Kubernetes cluster service mesh. I know a bunch of smart people are thinking about it and we'll be able to extend SMI to accommodate for that.

I'd like to focus on augmenting SMI to bridge a Kubernetes cluster and a VM in this case - I'd like to flatten them into one service mesh. (I love how @mitchellh calls it a "flat world")

Here is another way to look at what I'm trying to solve:

The issue filed here is similar: linkerd/linkerd2#2048
Essentially - how do we register a non-Kubernetes service in the Service Mesh?
The proposal in linkerd/linkerd2#2048 is to use the CLI tool to add, join, or register (out-of-cluster) nodes to an existing mesh. I think SMI should be able to accommodate for that scenario. Let's declare the out-of-cluster nodes with SMI in lieu of issuing vendor-specific commands.

Here is an example:

# Register a service consisting of a specific cloud vendor's component
# Members of the service (IP addresses) are resolvable by querying the
# particular cloud provider (type) for the given resource (locator)
apiVersion: specs.smi-spec.io/v1alpha1
kind: Service
metadata:
  name: baz.mesh
spec:
  type: Azure
  locator: /resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz
  port: 7654

@grampelberg I tried using the classic k8s Service. I tried to shoehorn the idea of a VM locator in it:

kind: Service
apiVersion: v1
metadata:
  name: out-of-cluster-service

spec:
  type: ClusterIP
  selector:
    locator: /resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz

  ports:
  - port: 7654

But kubectl apply -f the YAML above results in:

delyan@Azure:~$ kubectl apply -f Service.yaml
The Service "out-of-cluster-service" is invalid:
* spec.selector: Invalid value: "/resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz": must be no more than 63 characters
* spec.selector: Invalid value: "/resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

So like I said in my first message - it would have been good to reuse k8s Service, but it has limitations. For that reason I propose we create a new CRD to register these out of cluster services.

As SMI is currently defined as CRDs for k8s, should this one be out of scope?

Agreed - non-Kubernetes service meshes can stay out of scope for now.

@nicholasjackson Would love to hear your opinion on the proposal here to provide the consul join functionality with a declarative SMI CRD.

[grampelberg]

@grampelberg I tried using the classic k8s Service. I tried to shoehorn the idea of a VM locator in it:

What's the benefit to having /resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz instead of an IP address?

I was thinking something like this:

apiVersion: v1
kind: Service
metadata:
  name: linkerd-web
  namespace: linkerd
spec:
  ports:
  - name: http
    port: 8084
    protocol: TCP
    targetPort: 8084
  - name: admin-http
    port: 9994
    protocol: TCP
    targetPort: 9994
---
apiVersion: v1
kind: Endpoints
metadata:
  name: linkerd-web
  namespace: linkerd
subsets:
- addresses:
  - ip: 192.168.10.10
    nodeName: baz
    targetRef:
      kind: VM
      name: baz
      namespace: linkerd
  ports:
  - name: http
    port: 8084
    protocol: TCP
  - name: admin-http
    port: 9994
    protocol: TCP

If we use a pattern like this, apps that aren't part of the mesh will be able to use the service and talk to the VMs.

[lachie83]

Could you reference this as TrafficSplit as I feel it is a more compelling example for this proposal?

[grampelberg]

Here's how I would do this:

kind: AzureResource
metadata:
    name: my-resource-group
    namespace: dev
    labels:
        app: foobar
selector: /resource/subscriptions/e3f0/resourceGroups/mesh-rg/providers/Microsoft.Compute/virtualMachineScaleSets/baz
---
kind: Service
metadata:
    name: baz
    namespace: dev
spec:
    ports:
    - name: http
      port: 8084
      targetPort: 8000
      protocol: TCP
    selector:
        app: foobar

There would then be an operator installed on cluster that watches AzureResource and updates the Endpoints for the baz service. This:

• Uses k8s primitives, so it works for clients that are not part of the mesh.
• Provides service mesh implementation for free.
• Allows mixing and matching, imagine pods and VMs all as parts of the same service as part of a transition.
• Fits with the existing SMI specifications - no updates of TrafficSplit would be required to support this.
• Keeps a ton of flexibility on the cloud provider side. As the only thing required is an IP address, you could imagine the selector pointing to load balancers, individual vms, node pools, pretty much anything that is addressable.

Open Questions

• This does not address routing in any way and assumes that the cloud provider has figured that part out.
• I would need to boot back up on ARM to provide a better AzureResource definition, this feels a little sparse right now.

[aanandr]

@grampelberg - can the Azure example be merged with the earlier proposal just above it? Right now they look quite different. For e.g. can the Azure resource string be specified as an endpoint?
Your first proposal is similar to the Istio service entry.

[grampelberg]

@aanandr I'm not sure what your question is. My proposal completely replaces the original.

[aanandr]

@delqn - this has the facility to use different source types (e.g. a Kubernetes Service, DNS name, Azure resource name). But does it allow users to also use these types for the destination? This will allow creation of access policies for applications running in VMs, for e.g.

[aanandr]

In the main write-up you mentioned about supporting both ServiceAccount and service. Users can use what they want. Can users use only service name even for single K8S clusters?

[lachie83]

@michelleN to follow up

[michelleN]

@draychev - should we close this out for now?

[bridgetkromhout]

Closing based on @michelleN & @draychev agreement.