Create dev mode instructions for operator deployment

[Tomcli]

Currently, the operator always watching the Kubeflow resources to reconcile when something is missing. This is good for production environment, but not very friendly when we need to remove and test resources in our development and testing setup. It would be nice to have a dev_mode flag to disable the operator watcher for development.

/cc @moficodes

[vpavlin]

How would the reconcilation be triggered then? (I.e. what would the operator do if not watch:) )

[moficodes]

I think the goal is to make the operator more like kfctl. With the dev mode operator is just wrapping kfctl running the command once and thats it.

its useful for quickly iterating and testing the operator deployment.

[moficodes]

I can take a look at it.

[moficodes]

/assign

[vpavlin]

Why not just use kfctl then?

Or even better, use the operator-sdk tooling for development - https://github.com/operator-framework/getting-started#2-run-locally-outside-the-cluster

[Tomcli]

This is coming from one of our users who doesn't have much experience as a devops. We probably don't have to disable all the watchers, we only want to disable the watcher for monitoring the k8s resources https://github.com/kubeflow/kfctl/blob/master/pkg/controller/kfdef/kfdef_controller.go#L119

[Tomcli]

also, this is an opt-out feature, so it shouldn't change the behavior of the current operator deployment.

[Tomcli]

Why not just use kfctl then?

Or even better, use the operator-sdk tooling for development - https://github.com/operator-framework/getting-started#2-run-locally-outside-the-cluster

For most of our users, kfctl is sufficient in this case. However, we have some users that are using window or have very little experience with terminal. So able to use operator for development would be nice for them.

[vpavlin]

Can you help me to understand the use case again - maybe with more details? It sounds like there is a very specific case which would get treatment in the operator where it should rather be treated by educating the user(s).

[Tomcli]

Since the default behavior for operator now is to reapply the kfdef if there a delete event from any kfctl resource, users that made changes to the Kubeflow deployment with kubectl edit instead of updating kubeflow/manifests will lose their configuration. I do agree educating the users is the right approach, but I'm seeing some users are afraid to use operator when they see a big learning curve for deployment.

I suggest only use this flag for users that are deploying Kubeflow by themselves in a dev setup. So those who are interested in the Kubeflow project will be more committed to learn about kustomize and kfdef to deploy Kubeflow with the operator in the right way.

[nakfour]

@Tomcli I don't think it is a good idea to override a normal operator workflow to satisfy a small set of users. Another option they can do as @tumido pointed out is to install the operator, install Kubeflow and then pull down the operator pod instance to 0. This will remove the operator pod watching and doing the reconcile function. I am absolutely not a fan of adding code that breaks the fundamental function of an operator.

[Tomcli]

Thanks @nakfour, pull down the operator pod instance to 0 can be a good option. Then we probably want to add some instructions for:

1. How to stop watching kubeflow deployment (using kubectl, k8s/ocp ui to cover different audiences)
2. When to resume watching (e.g. deleting kubeflow, update kfdef)

Hopefully this way we should able to help out our users without changing the operator behaviors.