Fix: Move away from kube-rbac-proxy sidecar for metric server auth.

[codeknight03]

Description

This PR fixes ##573

Notes for Reviewers

Impact

• Meshery and any other clients of the metric server of the operator have to move to port port:8443
• Providing valid certs for Production deployments is needed.
• Created a new bundle because this is a breaking change.

Tests Done

** Local Run ( Outside Cluster)**
Out side cluster run with

make run

• Without Token : Returned Unauthorized as expected
  

• With secureMetrics=false: Returned Metrics as expected
  

• With Token: Can not be tested for Local as without a k8s cluster can not generate token.

Within Cluster run
After running the following commands

kind create cluster --name operator-test
sudo docker build -t layer5/meshery-operator:stable-latest .
kind load docker-image layer5/meshery-operator:stable-latest --name operator-test
make install
kubectl create ns meshery
make deploy

Also had to set ImagePullPolicy:IfNotPresent in config/manager/manager.yaml

Performed the following tests

• Without Token : Returned Unauth as expected
  

• With SA Token:

1. Create sa bound to the metrics-reader-cluster role

kubectl create sa metrics-reader-sa -n meshery
kubectl create clusterrolebinding metrics-reader-sa-binding \                                              
  --clusterrole=meshery-metrics-reader \
  --serviceaccount=meshery:metrics-reader-sa

2. Create token for this sa

TOKEN=$(kubectl create token metrics-reader-sa -n meshery)

3. Got Metrics as expected
   

Credits

• A lot of work on this was already done by @fyzanshaik , and I have borrowed the key ideas. We can close this MR have him cherry pick these commits if he has any other things to add.

Signed commits

• Yes, I signed my commits.

[welcome]

Yay, your first pull request! 👍 A contributor will be by to give feedback soon. In the meantime, please review the Layer5 Community Welcome Guide and sure to join the community Slack.
Be sure to double-check that you have signed your commits. Here are instructions for making signing an implicit activity while peforming a commit.

[codeknight03]

Blocker before merging:

make test

Is failing because I am on darwin/arm64 and it expects the binaries to be in linux/amd64. Need help @aabidsofi19.

[aabidsofi19]

@codeknight03 you might try running on codespaces . I am having some trouble with kubernetes on my linux

[fyzanshaik]

@codeknight03 you can custom build the binaries for this using go. GOOS=linux GOARCH=amd64 go build -o bin/linux_amd64/mybinary main.go

[lekaf974]

not sure this is a valid url

[lekaf974]

meshery-operator is more than a multi-service mesh

[lekaf974]

Suggested change

	name: meshsyncs.meshery.io
	name: meshsync.meshery.io

[lekaf974]

Suggested change

	if secureMetrics {
	// Add the auth filter *only* if secure serving is enabled
	metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
	setupLog.Info(
	"Metrics endpoint protection enabled using controller-runtime filters",
	"address",
	metricsAddr,
	)
	} else {
	// Log if metrics are insecure (should be rare if default is true)
	setupLog.Info(
	"Metrics endpoint is serving insecurely",
	"address",
	metricsAddr,
	)
	}
	// Log if metrics are insecure (should be rare if default is true)
	metricsEndpointLog := "Metrics endpoint is serving insecurely"
	if secureMetrics {
	// Add the auth filter *only* if secure serving is enabled
	metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
	metricsEndpointLog = "Metrics endpoint is serving with protection enabled using controller-runtime filters" // perhaps reword this one to make it a little more similar to the insecure message
	}
	setupLog.Info(
	metricsEndpointLog,
	"address",
	metricsAddr,
	)

[lekaf974]

this changes looks more than a patch. it should be a major upgrade or at least minor one ?

[codeknight03]

@codeknight03 you can custom build the binaries for this using go. GOOS=linux GOARCH=amd64 go build -o bin/linux_amd64/mybinary main.go

Yeah this helps, but actually the source of my troubles is

make test-env

is downloading all the binaries that are required for the test env, based on my platform i.e. darwin/arm64. But

make test 

needs binaries place at the path ./bin/linux/amd64 so I am thinking of running this either in codespaces or in one of my hetzner boxes I have.

[leecalcote]

@codeknight03, are you still chewing on this one?

[codeknight03]

This one is complete as it is I am working on the test cases. But haven't had a lot of progress to be honest.