NRI on Koordinator

[saintube]

In this talk, we discuss why and how the Node Resource Interface (NRI) could be used in Koordinator.

NRI Introduction

Overview

ref: https://mp.weixin.qq.com/s/gwww7Is2_lG7m20wdAeWSg

There are several approaches to extending the Kubernetes CRI (Container Runtime Interface) to manage container resources (e.g. adjust the cpuset of a container).

• Default
  • kubelet -- CRI Request -> CRI Runtime (containerd/cri-o) -- OCI Spec -> OCI-compatible runtime (runc/kata) -> containers
• CRI Proxy (e.g. koordlet + runtime-proxy)
  • kubelet -- CRI Request -> CRI Proxy -- CRI Request (hooked) -> CRI Runtime -- OCI Spec -> OCI compatible runtime -> containers
• Standalone (e.g. koordlet)
  • kubelet -- CRI Request -> CRI Runtime -- OCI Spec -> OCI compatible runtime -> containers
  • kubelet ---> Node Agent -> CRI Runtime / containers

Instead of modifying the CRI request, NRI offers us a new way to manager container resources by adjusting OCI-spec.

NRI API

The Runtime service is the public interface runtimes expose for NRI plugins. All requests on this interface are initiated by the plugin. The interface provides functions for

• initiating plugin registration
• requesting unsolicited updates to containers
  The Plugin service is the public interface NRI uses to interact with plugins. All requests on this interface are initiated by NRI/the runtime. The interface provides functions for
• configuring the plugin
• getting initial list of already existing pods and containers
• hooking the plugin into pod/container lifecycle events
• shutting down the plugin

Metadata

Managed Resources

LinuxResources

type LinuxResources struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Memory         *LinuxMemory         `protobuf:"bytes,1,opt,name=memory,proto3" json:"memory,omitempty"`
	Cpu            *LinuxCPU            `protobuf:"bytes,2,opt,name=cpu,proto3" json:"cpu,omitempty"`
	HugepageLimits []*HugepageLimit     `protobuf:"bytes,3,rep,name=hugepage_limits,json=hugepageLimits,proto3" json:"hugepage_limits,omitempty"`
	BlockioClass   *OptionalString      `protobuf:"bytes,4,opt,name=blockio_class,json=blockioClass,proto3" json:"blockio_class,omitempty"`
	RdtClass       *OptionalString      `protobuf:"bytes,5,opt,name=rdt_class,json=rdtClass,proto3" json:"rdt_class,omitempty"`
	Unified        map[string]string    `protobuf:"bytes,6,rep,name=unified,proto3" json:"unified,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	Devices        []*LinuxDeviceCgroup `protobuf:"bytes,7,rep,name=devices,proto3" json:"devices,omitempty"` // for NRI v1 emulation
}

LinuxDevices

type LinuxDevice struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Path     string            `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
	Type     string            `protobuf:"bytes,2,opt,name=type,proto3" json:"type,omitempty"`
	Major    int64             `protobuf:"varint,3,opt,name=major,proto3" json:"major,omitempty"`
	Minor    int64             `protobuf:"varint,4,opt,name=minor,proto3" json:"minor,omitempty"`
	FileMode *OptionalFileMode `protobuf:"bytes,5,opt,name=file_mode,json=fileMode,proto3" json:"file_mode,omitempty"`
	Uid      *OptionalUInt32   `protobuf:"bytes,6,opt,name=uid,proto3" json:"uid,omitempty"`
	Gid      *OptionalUInt32   `protobuf:"bytes,7,opt,name=gid,proto3" json:"gid,omitempty"`
}

Subscriptable Events

How-To Access

There are two ways to access NRI:

1. Pre-Registered Plugins

• Configs prepared in the directory /opt/nri/plugins.

2. External Plugins

• Plugins communicate and register with NRI Adaption via /var/run/nri/nri.sock.
• Runtime can configure to disable External Plugins.

NRI on Koordinator

Access NRI

Add a Koord NRI Plugin for

1. Get OCI specs from the NRI Adaption.
2. Transform the OCI spec into internal protocols (i.e. PodContext, ContainerContext) to re-use existing runtime hook plugins.

Workflow

... --> CRI Runtime - OCI Spec -> NRI Adaption --> Koord NRI Plugin - OCI Spec (adjusted) -> ...

vs. Koord Runtime Proxy

• Advantages
  • Adopt the upstream standard of the container runtime.
  • Pluggable deployed than the Runtime Proxy.
• Challenges
  • Need to upgrade container >= 1.7.0.
  • Fault tolerance: koordlet expects some plugin failures to be tolerated.
    • Koordlet: (PluginFailPolicy) https://github.com/koordinator-sh/koordinator/blob/main/pkg/koordlet/runtimehooks/hooks/hooks.go#L79
    • NRI: (if FatalError(err) { plugin.close() } else abort) https://github.com/containerd/nri/blob/v0.3.0/pkg/adaptation/adaptation.go#L290

[saintube]

#1277