Add comprehensive GitHub Copilot instructions for Retina development

Copilot · matmerr · Copilot · commit 22c6730290d1 · 2025-09-19T22:39:26.000Z
Co-authored-by: matmerr &lt;6521405+matmerr@users.noreply.github.com&gt;
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
@@ -0,0 +1,125 @@
+# Retina - eBPF Network Observability Platform
+
+Retina is a cloud-agnostic, open-source Kubernetes network observability platform that provides centralized monitoring for application health, network health, and security. Built with Go, eBPF, and container technologies for Linux and Windows environments.
+
+Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.
+
+## Working Effectively
+
+### Environment Setup
+- Install required dependencies:
+  - `sudo apt update && sudo apt install -y clang llvm-strip-18 jq`
+  - `sudo ln -sf /usr/bin/llvm-strip-18 /usr/bin/llvm-strip` (if needed)
+- Go 1.24.6+ required (check with `go version`)
+- Docker and Helm required for container operations
+- `clang` and `llvm-strip` are CRITICAL for eBPF compilation
+
+### Core Build Commands
+- `make retina` -- builds retina binary -- takes ~1 minute (includes eBPF generation). NEVER CANCEL. Set timeout to 10+ minutes.
+- `make retina-capture-workload` -- builds capture workload binary -- takes ~2 seconds
+- CLI: `cd cli && go build -o ../output/linux_amd64/retina/kubectl-retina .` -- takes ~6 seconds
+- `make clean` -- clean build artifacts -- takes < 1 second
+
+### Testing and Validation
+- `make test` -- runs full test suite -- takes 10+ minutes. NEVER CANCEL. Set timeout to 20+ minutes.
+- Basic package tests: `go test -timeout 10m -tags=unit ./pkg/...` -- takes ~6 minutes with some expected failures in cross-platform code
+- `make fmt` -- format code -- takes ~1 second
+- `make lint` -- runs linting -- takes ~2-3 minutes. May show some issues in generated mock files that can be ignored.
+
+### BPF Generation and Plugins
+- BPF generation for current architecture works: generates `.o` and `.go` files for eBPF programs
+- Cross-compilation (ARM64) may fail in development environment - this is expected
+- Plugin test example: `cd test/plugin && make test-packetforward` -- builds and runs plugin tests with sudo
+- Individual plugins can be tested but require network traffic to show meaningful data
+
+## Validation Scenarios
+
+### Basic Functionality Validation
+Always test these core workflows after making changes:
+
+1. **Build and run agent**:
+   ```bash
+   make retina
+   ./output/linux_amd64/retina/retina --help
+   ```
+
+2. **Build and test CLI**:
+   ```bash
+   cd cli && go build -o ../output/linux_amd64/retina/kubectl-retina .
+   ./output/linux_amd64/retina/kubectl-retina --help
+   ./output/linux_amd64/retina/kubectl-retina version
+   ```
+
+3. **Test plugin functionality**:
+   ```bash
+   cd test/plugin && make test-packetforward
+   # This will run until Ctrl+C - expect to see "Start collecting packet forward metrics"
+   ```
+
+## Critical Build Timing and Warnings
+
+### NEVER CANCEL - Build Time Expectations
+- **Main binary build (`make retina`)**: Takes 1-2 minutes including eBPF generation. NEVER CANCEL. Use timeout 10+ minutes.
+- **Full test suite (`make test`)**: Takes 10-15 minutes. NEVER CANCEL. Use timeout 20+ minutes.
+- **Linting (`make lint`)**: Takes 2-3 minutes. NEVER CANCEL. Use timeout 5+ minutes.
+- **Plugin tests**: Can run indefinitely waiting for network traffic - this is expected behavior.
+
+### Expected Build Issues
+- Cross-compilation for ARM64 may fail with "exec format error" - this is expected in development environment
+- Some unit tests fail due to missing kubebuilder/etcd dependencies - this is expected
+- Lint warnings about generated mock files are expected and can be ignored
+- BPF compilation warnings about operator precedence are expected and harmless
+
+## Project Structure
+
+### Key Directories
+- `/pkg/plugin/` - eBPF plugins for network observability (conntrack, dropreason, packetforward, etc.)
+- `/controller/` - Main retina agent controller code
+- `/cli/` - kubectl-retina CLI implementation  
+- `/captureworkload/` - Network capture workload implementation
+- `/operator/` - Kubernetes operator code
+- `/test/plugin/` - Individual plugin test utilities
+- `/docs/08-Contributing/02-development.md` - Detailed development guide
+
+### Important Files
+- `Makefile` - Primary build system with comprehensive targets
+- `go.mod` - Go 1.24.6, extensive Kubernetes and eBPF dependencies
+- `.devcontainer/` - GitHub Codespaces configuration with required tools
+- `.github/workflows/` - CI/CD pipelines for testing and building
+
+## Common Development Tasks
+
+### Before Committing Changes
+Always run these commands before pushing:
+1. `make fmt` - Format code (required for CI)
+2. `make lint` - Check code quality (may show expected warnings)
+3. `make retina` - Ensure main binary builds correctly
+4. Test CLI: `cd cli && go build .` - Ensure CLI builds
+
+### Working with eBPF Code
+- eBPF source files are in `pkg/plugin/*/\_cprog/` directories
+- Generated files: `*_bpfel_x86.o` and `*_bpfel_x86.go` 
+- BPF generation happens automatically during build
+- Some compiler warnings about operator precedence are expected and harmless
+
+### Container and Helm Operations
+- `make retina-image` - Build container image (requires registry access)
+- Images are published to GHCR (GitHub Container Registry)
+- Helm charts are available in the repository for deployment
+- Full deployment requires Kubernetes cluster with appropriate permissions
+
+## Debugging and Troubleshooting
+
+### Common Issues
+- **"exec format error"**: Usually cross-compilation issue, rebuild for current arch
+- **Missing kubebuilder**: Some tests require Kubernetes test environment
+- **Plugin tests hanging**: Normal behavior - plugins wait for network traffic
+- **Lint failures**: Check if they're in generated mock files (can be ignored)
+- **BPF compilation warnings**: Expected and generally harmless
+
+### Log Analysis
+- Agent logs provide detailed eBPF and networking information
+- CLI operations are verbose and show capture job creation details
+- Plugin tests show real-time network metrics when traffic is available
+
+This is a complex, enterprise-grade networking platform. Take time to understand the eBPF integration and Kubernetes-native architecture before making changes.
diff --git a/cli/cmd/root.go b/cli/cmd/root.go
@@ -24,9 +24,9 @@ type Config struct {
 }
 
 var Retina = &cobra.Command{
-	Use: "kubectl-retina",
+	Use:   "kubectl-retina",
 	Short: "A kubectl plugin for Retina",
-	Long: "A kubectl plugin for Retina\nRetina is an eBPF distributed networking observability tool for Kubernetes.",
+	Long:  "A kubectl plugin for Retina\nRetina is an eBPF distributed networking observability tool for Kubernetes.",
 	PersistentPreRun: func(*cobra.Command, []string) {
 		var config Config
 		file, _ := os.ReadFile(ClientConfigPath)
diff --git a/deploy/testutils/grafana/dashboards/simplify-grafana-overwrite_test.go b/deploy/testutils/grafana/dashboards/simplify-grafana-overwrite_test.go
@@ -11,7 +11,6 @@ import (
 func TestOverwriteDashboards(t *testing.T) {
 	// get all json's in various generation deploly folders
 	files, err := filepath.Glob("../../../grafana-dashboards/*.json")
-
 	if err != nil {
 		t.Fatal(err)
 	}
diff --git a/deploy/testutils/grafana/dashboards/simplify-grafana_test.go b/deploy/testutils/grafana/dashboards/simplify-grafana_test.go
@@ -12,7 +12,6 @@ import (
 func TestDashboardsAreSimplified(t *testing.T) {
 	// get all json's in this folder
 	files, err := filepath.Glob("../../../grafana-dashboards/*.json")
-
 	if err != nil {
 		t.Fatal(err)
 	}
diff --git a/init/retina/retina b/init/retina/retina
diff --git a/operator/cilium-crds/k8s/fakeresource.go b/operator/cilium-crds/k8s/fakeresource.go
@@ -8,8 +8,7 @@ import (
 	"github.com/cilium/cilium/pkg/k8s/resource"
 )
 
-type fakeresource[T k8sRuntime.Object] struct {
-}
+type fakeresource[T k8sRuntime.Object] struct{}
 
 func (f *fakeresource[T]) Events(ctx context.Context, opts ...resource.EventsOpt) <-chan resource.Event[T] {
 	return make(<-chan resource.Event[T])
diff --git a/pkg/k8s/watcher_linux.go b/pkg/k8s/watcher_linux.go
@@ -23,9 +23,7 @@ func init() {
 	}
 }
 
-var (
-	logger = logging.DefaultLogger.WithField(logfields.LogSubsys, "k8s-watcher")
-)
+var logger = logging.DefaultLogger.WithField(logfields.LogSubsys, "k8s-watcher")
 
 func Start(ctx context.Context, k *watchers.K8sWatcher) {
 	logger.Info("Starting Kubernetes watcher")
diff --git a/pkg/plugin/ciliumeventobserver/parser_linux.go b/pkg/plugin/ciliumeventobserver/parser_linux.go
@@ -16,9 +16,7 @@ import (
 	"go.uber.org/zap"
 )
 
-var (
-	ErrEmptyData = errors.New("empty data")
-)
+var ErrEmptyData = errors.New("empty data")
 
 func (p *parser) Init() error {
 	parser, err := hp.New(slog.Default().With("cilium", "parser"),
diff --git a/pkg/plugin/hnsstats/vfp_counters_windows.go b/pkg/plugin/hnsstats/vfp_counters_windows.go
@@ -152,7 +152,6 @@ func getVfpPortCountersRaw(portGUID string) (string, error) {
 
 	cmd := exec.Command("cmd", "/c", vfpCmd)
 	out, err := cmd.Output()
-
 	if err != nil {
 		return "", errors.Wrap(err, "errored while running vfpctrl /get-port-counter")
 	}
diff --git a/pkg/plugin/linuxutil/types_linux.go b/pkg/plugin/linuxutil/types_linux.go
@@ -9,8 +9,10 @@ import (
 	"github.com/safchain/ethtool"
 )
 
-const name = "linuxutil"
-const defaultLimit = 2000
+const (
+	name         = "linuxutil"
+	defaultLimit = 2000
+)
 
 //go:generate go run go.uber.org/mock/mockgen@v0.4.0 -source=types_linux.go -destination=linuxutil_mock_generated_linux.go -package=linuxutil
 type linuxUtil struct {
diff --git a/pkg/telemetry/telemetry.go b/pkg/telemetry/telemetry.go
@@ -200,6 +200,7 @@ func (t *TelemetryClient) heartbeat(ctx context.Context, funcs ...func() map[str
 	maps.Copy(props, t.profile.GetMemoryUsage())
 	t.TrackEvent("heartbeat", props)
 }
+
 func metricsCardinality(gatherer prometheus.Gatherer) (int, error) {
 	if gatherer == nil {
 		return 0, fmt.Errorf("failed to get metrics Gatherer: %w", ErrorNilCombinedGatherer)
diff --git a/test/e2e/framework/kubernetes/check-pod-status.go b/test/e2e/framework/kubernetes/check-pod-status.go
@@ -37,7 +37,6 @@ func (w *WaitPodsReady) Prevalidate() error {
 // Primary step where test logic is executed
 // Returning an error will cause the test to fail
 func (w *WaitPodsReady) Run() error {
-
 	config, err := clientcmd.BuildConfigFromFlags("", w.KubeConfigFilePath)
 	if err != nil {
 		return fmt.Errorf("error building kubeconfig: %w", err)
@@ -59,7 +58,6 @@ func (w *WaitPodsReady) Stop() error {
 }
 
 func WaitForPodReady(ctx context.Context, clientset *kubernetes.Clientset, namespace, labelSelector string) error {
-
 	printIterator := 0
 	conditionFunc := wait.ConditionWithContextFunc(func(context.Context) (bool, error) {
 		defer func() {
diff --git a/test/e2e/framework/kubernetes/get-external-crd.go b/test/e2e/framework/kubernetes/get-external-crd.go
@@ -55,7 +55,7 @@ func extractFileName(rawURL string) (string, error) {
 }
 
 func saveToFile(filename string, data []byte) error {
-	err := os.WriteFile(filename, data, 0644)
+	err := os.WriteFile(filename, data, 0o644)
 	if err != nil {
 		return fmt.Errorf("failed to write crd.yaml to /crds dir : %w", err)
 	}
diff --git a/test/e2e/framework/kubernetes/install-retina-helm.go b/test/e2e/framework/kubernetes/install-retina-helm.go
@@ -68,7 +68,7 @@ func (i *InstallHelmChart) Run() error {
 		return fmt.Errorf("image namespace is not set: %w", errEmpty)
 	}
 
-	//Download necessary CRD's
+	// Download necessary CRD's
 	err = downloadExternalCRDs(i.ChartPath)
 	if err != nil {
 		return fmt.Errorf("failed to load external crd's: %w", err)
diff --git a/test/e2e/framework/scaletest/add-shared-labels.go b/test/e2e/framework/scaletest/add-shared-labels.go
@@ -35,7 +35,6 @@ func (a *AddSharedLabelsToAllPods) Prevalidate() error {
 // Primary step where test logic is executed
 // Returning an error will cause the test to fail
 func (a *AddSharedLabelsToAllPods) Run() error {
-
 	if a.NumSharedLabelsPerPod < 1 {
 		return nil
 	}
diff --git a/test/e2e/framework/scaletest/add-unique-labels.go b/test/e2e/framework/scaletest/add-unique-labels.go
@@ -27,7 +27,6 @@ func (a *AddUniqueLabelsToAllPods) Prevalidate() error {
 // Primary step where test logic is executed
 // Returning an error will cause the test to fail
 func (a *AddUniqueLabelsToAllPods) Run() error {
-
 	if a.NumUniqueLabelsPerPod < 1 {
 		return nil
 	}
diff --git a/test/e2e/framework/scaletest/create-network-policies.go b/test/e2e/framework/scaletest/create-network-policies.go
@@ -33,7 +33,6 @@ func (c *CreateNetworkPolicies) Prevalidate() error {
 // Primary step where test logic is executed
 // Returning an error will cause the test to fail
 func (c *CreateNetworkPolicies) Run() error {
-
 	config, err := clientcmd.BuildConfigFromFlags("", c.KubeConfigFilePath)
 	if err != nil {
 		return fmt.Errorf("error building kubeconfig: %w", err)
diff --git a/test/e2e/framework/scaletest/delete-and-re-add-labels.go b/test/e2e/framework/scaletest/delete-and-re-add-labels.go
@@ -118,7 +118,6 @@ func (d *DeleteAndReAddLabels) addLabels(ctx context.Context, clientset *kuberne
 }
 
 func (d *DeleteAndReAddLabels) deleteLabels(ctx context.Context, clientset *kubernetes.Clientset, pods *corev1.PodList, patch string) error {
-
 	for _, pod := range pods.Items {
 		log.Println("Deleting label from Pod", pod.Name)
 
diff --git a/test/e2e/framework/scaletest/get-publish-metrics.go b/test/e2e/framework/scaletest/get-publish-metrics.go
@@ -76,7 +76,6 @@ func (g *GetAndPublishMetrics) Run() error {
 	g.errs = new(errgroup.Group)
 
 	g.errs.Go(func() error {
-
 		t := time.NewTicker(defaultInterval)
 		defer t.Stop()
 
@@ -100,7 +99,6 @@ func (g *GetAndPublishMetrics) Run() error {
 
 			}
 		}
-
 	})
 
 	return nil
@@ -135,7 +133,6 @@ func (g *GetAndPublishMetrics) Prevalidate() error {
 }
 
 func (g *GetAndPublishMetrics) getAndPublishMetrics() error {
-
 	ctx := context.TODO()
 
 	labelSelector := labels.Set(g.Labels).String()
@@ -161,7 +158,6 @@ func (g *GetAndPublishMetrics) getAndPublishMetrics() error {
 		log.Println("Publishing metrics to AppInsights")
 		for _, metric := range allMetrics {
 			g.telemetryClient.TrackEvent("scale-test", metric)
-
 		}
 	}
 
@@ -192,7 +188,6 @@ func (g *GetAndPublishMetrics) getAndPublishMetrics() error {
 type metric map[string]string
 
 func (g *GetAndPublishMetrics) getPodsMetrics(ctx context.Context, labelSelector string) ([]metric, error) {
-
 	var pods *v1.PodList
 
 	retrier := retry.Retrier{Attempts: defaultRetryAttempts, Delay: defaultRetryDelay}
diff --git a/test/e2e/framework/scaletest/templates/networkpolicy.go b/test/e2e/framework/scaletest/templates/networkpolicy.go
@@ -5,23 +5,21 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-var (
-	NetworkPolicy = netv1.NetworkPolicy{
-		TypeMeta: metav1.TypeMeta{
-			Kind:       "NetworkPolicy",
-			APIVersion: "networking.k8s.io/v1",
+var NetworkPolicy = netv1.NetworkPolicy{
+	TypeMeta: metav1.TypeMeta{
+		Kind:       "NetworkPolicy",
+		APIVersion: "networking.k8s.io/v1",
+	},
+	ObjectMeta: metav1.ObjectMeta{
+		Name: "template-network-policy",
+	},
+	Spec: netv1.NetworkPolicySpec{
+		PolicyTypes: []netv1.PolicyType{
+			"Ingress",
+			"Egress",
 		},
-		ObjectMeta: metav1.ObjectMeta{
-			Name: "template-network-policy",
+		PodSelector: metav1.LabelSelector{
+			MatchLabels: map[string]string{},
 		},
-		Spec: netv1.NetworkPolicySpec{
-			PolicyTypes: []netv1.PolicyType{
-				"Ingress",
-				"Egress",
-			},
-			PodSelector: metav1.LabelSelector{
-				MatchLabels: map[string]string{},
-			},
-		},
-	}
-)
+	},
+}
diff --git a/test/e2e/framework/scaletest/validate-options.go b/test/e2e/framework/scaletest/validate-options.go
@@ -37,7 +37,6 @@ func (po *ValidateAndPrintOptions) Prevalidate() error {
 
 // Returning an error will cause the test to fail
 func (po *ValidateAndPrintOptions) Run() error {
-
 	log.Printf("Starting to scale with folowing options: %+v", po.Options)
 
 	return nil
diff --git a/test/e2e/retina_perf_test.go b/test/e2e/retina_perf_test.go
@@ -17,11 +17,9 @@ import (
 	"github.com/stretchr/testify/require"
 )
 
-var (
-	// Add flags for the test
-	// retina-mode: basic, advanced, hubble
-	retinaMode = flag.String("retina-mode", "basic", "One of basic or advanced")
-)
+// Add flags for the test
+// retina-mode: basic, advanced, hubble
+var retinaMode = flag.String("retina-mode", "basic", "One of basic or advanced")
 
 func validateRetinaMode(t *testing.T) {
 	switch *retinaMode {
diff --git a/test/e2e/scenarios/perf/publish-perf-results.go b/test/e2e/scenarios/perf/publish-perf-results.go
@@ -29,7 +29,7 @@ func (v *PublishPerfResults) Run() error {
 		return nil
 	}
 
-	resultsFile, err := os.OpenFile(v.ResultsFile, os.O_RDONLY, 0644)
+	resultsFile, err := os.OpenFile(v.ResultsFile, os.O_RDONLY, 0o644)
 	if err != nil {
 		return errors.Wrap(err, "failed to open results file")
 	}
diff --git a/test/multicloud/test/utils/utils_test.go b/test/multicloud/test/utils/utils_test.go

Original file line number	Diff line number	Diff line change
`@@ -11,7 +11,6 @@ import (`
`11`	`11`	`func TestOverwriteDashboards(t *testing.T) {`
`12`	`12`	`// get all json's in various generation deploly folders`
`13`	`13`	`files, err := filepath.Glob("../../../grafana-dashboards/*.json")`
`14`		`-`
`15`	`14`	`if err != nil {`
`16`	`15`	`t.Fatal(err)`
`17`	`16`	`}`
Original file line number	Diff line number	Diff line change
`@@ -12,7 +12,6 @@ import (`
`12`	`12`	`func TestDashboardsAreSimplified(t *testing.T) {`
`13`	`13`	`// get all json's in this folder`
`14`	`14`	`files, err := filepath.Glob("../../../grafana-dashboards/*.json")`
`15`		`-`
`16`	`15`	`if err != nil {`
`17`	`16`	`t.Fatal(err)`
`18`	`17`	`}`
Original file line number	Diff line number	Diff line change
`@@ -8,8 +8,7 @@ import (`
`8`	`8`	`"github.com/cilium/cilium/pkg/k8s/resource"`
`9`	`9`	`)`
`10`	`10`
`11`		`-type fakeresource[T k8sRuntime.Object] struct {`
`12`		`-}`
	`11`	`+type fakeresource[T k8sRuntime.Object] struct{}`
`13`	`12`
`14`	`13`	`func (f *fakeresource[T]) Events(ctx context.Context, opts ...resource.EventsOpt) <-chan resource.Event[T] {`
`15`	`14`	`return make(<-chan resource.Event[T])`
Original file line number	Diff line number	Diff line change
`@@ -23,9 +23,7 @@ func init() {`
`23`	`23`	`}`
`24`	`24`	`}`
`25`	`25`
`26`		`-var (`
`27`		`- logger = logging.DefaultLogger.WithField(logfields.LogSubsys, "k8s-watcher")`
`28`		`-)`
	`26`	`+var logger = logging.DefaultLogger.WithField(logfields.LogSubsys, "k8s-watcher")`
`29`	`27`
`30`	`28`	`func Start(ctx context.Context, k *watchers.K8sWatcher) {`
`31`	`29`	`logger.Info("Starting Kubernetes watcher")`
Original file line number	Diff line number	Diff line change
`@@ -16,9 +16,7 @@ import (`
`16`	`16`	`"go.uber.org/zap"`
`17`	`17`	`)`
`18`	`18`
`19`		`-var (`
`20`		`- ErrEmptyData = errors.New("empty data")`
`21`		`-)`
	`19`	`+var ErrEmptyData = errors.New("empty data")`
`22`	`20`
`23`	`21`	`func (p *parser) Init() error {`
`24`	`22`	`parser, err := hp.New(slog.Default().With("cilium", "parser"),`
Original file line number	Diff line number	Diff line change
`@@ -152,7 +152,6 @@ func getVfpPortCountersRaw(portGUID string) (string, error) {`
`152`	`152`
`153`	`153`	`cmd := exec.Command("cmd", "/c", vfpCmd)`
`154`	`154`	`out, err := cmd.Output()`
`155`		`-`
`156`	`155`	`if err != nil {`
`157`	`156`	`return "", errors.Wrap(err, "errored while running vfpctrl /get-port-counter")`
`158`	`157`	`}`