Merge remote-tracking branch 'origin/main' into move-scalingAction

Author: rjeberhard

documentation/site/content/managing-domains/domain-lifecycle/scaling.md

@@ -23,8 +23,8 @@ The operator provides several ways to initiate scaling of WebLogic clusters, inc
 * [Using Domain lifecycle sample scripts](#using-domain-lifecycle-sample-scripts)
 * [Calling the operator's REST scale API, for example, from `curl`](#calling-the-operators-rest-scale-api)
 * [Kubernetes Horizontal Pod Autoscaler (HPA)](#kubernetes-horizontal-pod-autoscaler-hpa)
-* [Using a WLDF policy rule and script action to call the operator's REST scale API](#using-a-wldf-policy-rule-and-script-action-to-call-the-operators-rest-scale-api)
-* [Using a Prometheus alert action to call the operator's REST scale API](#using-a-prometheus-alert-action-to-call-the-operators-rest-scale-api)
+* [Using a WLDF policy rule and script action to call the scaling script](#using-a-wldf-policy-rule-and-script-action-to-call-the-scaling-script)
+* [Using a Prometheus alert action to call the scaling script](#using-a-prometheus-alert-action-to-call-the-scaling-script)
 
 ### `kubectl` CLI commands
 Use the Kubernetes command-line tool, `kubectl`, to manually scale WebLogic clusters with the following commands:
@@ -221,19 +221,15 @@ $ kubectl exec --stdin --tty sample-domain1-managed-server1 -- /bin/bash
 
 For more in-depth information on the Kubernetes Horizontal Pod Autoscaler, see [Horizontal Pod Autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
 
-#### Using a WLDF policy rule and script action to call the operator's REST scale API
+#### Using a WLDF policy rule and script action to call the scaling script
 The WebLogic Diagnostics Framework (WLDF) is a suite of services and APIs that collect and surface metrics that provide visibility into server and application performance.
 To support automatic scaling of WebLogic clusters in Kubernetes, WLDF provides the Policies and Actions component, which lets you write policy expressions for automatically executing scaling
 operations on a cluster. These policies monitor one or more types of WebLogic Server metrics, such as memory, idle threads, and CPU load.  When the configured threshold
 in a policy is met, the policy is triggered, and the corresponding scaling action is executed.  The WebLogic Kubernetes Operator project provides a shell script, [`scalingAction.sh`](https://github.com/oracle/weblogic-kubernetes-operator/blob/{{< latestMinorVersion >}}/kubernetes/samples/scripts/scaling/scalingAction.sh),
-for use as a Script Action, which illustrates how to issue a request to the operator’s REST endpoint.
-
-{{% notice note %}}
-Beginning with operator version 4.0.5, the operator's REST endpoint is disabled by default. Install the operator with the Helm install option `--set "enableRest=true"` to enable the REST endpoint.
-{{% /notice %}}
+for use as a Script Action, which initiates a scaling of the selected cluster.
 
 ##### Configure automatic scaling of WebLogic clusters in Kubernetes with WLDF
-The following steps are provided as a guideline on how to configure a WLDF Policy and Script Action component for issuing scaling requests to the operator's REST endpoint:
+The following steps are provided as a guideline on how to configure a WLDF Policy and Script Action component for issuing scaling requests:
 
 1. Copy the [`scalingAction.sh`](https://github.com/oracle/weblogic-kubernetes-operator/blob/{{< latestMinorVersion >}}/kubernetes/samples/scripts/scaling/scalingAction.sh) script to `$DOMAIN_HOME/bin/scripts` so that it's accessible within the Administration Server pod. For more information, see [Configuring Script Actions](https://docs.oracle.com/en/middleware/standalone/weblogic-server/14.1.1.0/wldfc/config_notifications.html#GUID-5CC52534-13CD-40D9-915D-3380C86580F1) in _Configuring and Using the Diagnostics Framework for Oracle WebLogic Server_.
 
@@ -244,23 +240,6 @@ see [Configuring Policies and Actions](https://docs.oracle.com/en/middleware/sta
 
      b. Configure a WLDF script action and associate the [`scalingAction.sh`](https://github.com/oracle/weblogic-kubernetes-operator/blob/{{< latestMinorVersion >}}/kubernetes/samples/scripts/scaling/scalingAction.sh) script.
 
-Important notes about the configuration properties for the Script Action:
-
-The `scalingAction.sh` script requires access to the SSL certificate of the operator’s endpoint and this is provided through the environment variable `INTERNAL_OPERATOR_CERT`.  
-The operator’s SSL certificate can be found in the `internalOperatorCert` entry of the operator’s ConfigMap `weblogic-operator-cm`:
-
-For example:
-```none
-#> kubectl describe configmap weblogic-operator-cm -n weblogic-operator
-...
-Data
-====
-internalOperatorCert:
-----
-LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR3akNDQXFxZ0F3SUJBZ0lFRzhYT1N6QU...
-...
-```
-
 The scalingAction.sh script accepts a number of customizable parameters:
 
 * `action` - scaleUp or scaleDown (Required)
@@ -279,12 +258,6 @@ Set this to `https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}` when
 
 * `wls_domain_namespace` - Kubernetes Namespace in which the WebLogic domain is defined, default=`default`
 
-* `operator_service_name` - WebLogic Kubernetes Operator Service name of the REST endpoint, default=`internal-weblogic-operator-service`
-
-* `operator_service_account` - Kubernetes Service Account name for the operator, default=`weblogic-operator`
-
-* `operator_namespace` – Namespace in which the operator is deployed, default=`weblogic-operator`
-
 * `scaling_size` – Incremental number of Managed Server instances by which to scale up or down, default=`1`
 
 You can use any of the following tools to configure policies for diagnostic system modules:
@@ -294,7 +267,7 @@ You can use any of the following tools to configure policies for diagnostic syst
 * REST
 * JMX application    
 
-A more in-depth description and example on using WLDF's Policies and Actions component for initiating scaling requests through the operator's REST endpoint can be found in the blogs:
+A more in-depth description and example on using WLDF's Policies and Actions component for initiating scaling requests can be found in the blogs:
 
 * [Automatic Scaling of WebLogic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/automatic-scaling-of-weblogic-clusters-on-kubernetes-v2)
 * [WebLogic Dynamic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/weblogic-dynamic-clusters-on-kubernetes)
@@ -313,11 +286,8 @@ apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: weblogic-domain-cluster-role
 rules:
-- apiGroups: [""]
-  resources: ["services/status"]
-  verbs: ["get", "list", "watch"]
 - apiGroups: ["weblogic.oracle"]
-  resources: ["domains"]
+  resources: ["domains", "clusters", "clusters/scale"]
   verbs: ["get", "list", "patch", "update"]
 ---
 #
@@ -360,7 +330,7 @@ roleRef:
 #### Horizontal Pod Autoscaler (HPA) using WebLogic Exporter Metrics
 Please read this blog post to learn how to scale a WebLogic cluster, based on WebLogic metrics provided by the Monitoring Exporter, using the Kubernetes Horizontal Pod Autoscaler (HPA). We will use the Prometheus Adapter to gather the names of the available metrics from Prometheus at regular intervals. A custom configuration of the adapter will expose only metrics that follow specific formats. [Horizontal Pod Autoscaler (HPA) using WebLogic Exporter Metrics](https://blogs.oracle.com/weblogicserver/post/horizontal-pod-autoscaler-hpausing-weblogic-exporter-metrics). See this corresponding video for a demonstration of the blog post in action. [WebLogic Kubernetes Operator support for Kubernetes Horizontal Pod Autoscaling](https://www.youtube.com/watch?v=aKBG6yJ3sMg).
 
-#### Using a Prometheus alert action to call the operator's REST scale API
+#### Using a Prometheus alert action to call the scaling script
 In addition to using the WebLogic Diagnostic Framework for automatic scaling of a dynamic cluster,
 you can use a third-party monitoring application like Prometheus.  Please read the following blog for
 details about [Using Prometheus to Automatically Scale WebLogic Clusters on Kubernetes](https://blogs.oracle.com/weblogicserver/using-prometheus-to-automatically-scale-weblogic-clusters-on-kubernetes-v5).

integration-tests/src/test/java/oracle/weblogic/kubernetes/ItReadOnlyRootFS.java

@@ -554,36 +554,16 @@ private static void validateContainerSpec(V1Container container, String podName,
       logger.info("PASS: Container " + containerName + " in pod " + podName + " has readOnlyRootFilesystem=true");
     }
 
-    if (!isInitContainer) {
-      // 2. For regular containers, exec to check /tmp mount
-      try {
-
-        ExecResult result = execCommand(namespace, podName, containerName, true, "df", "-h", "/tmp");
-        String stdout = result.stdout();
-        if (stdout == null || !stdout.contains("tmpfs")) {
-          String msg = "FAIL: /tmp is not mounted as tmpfs in container " + containerName + " in pod " + podName;
-          logger.severe(msg);
-          failures.add(msg);
-        } else {
-          logger.info("PASS: /tmp is mounted as tmpfs in container " + containerName + " in pod " + podName);
-        }
-      } catch (Exception e) {
-        String msg = "FAIL: Exec failed for container " + containerName + " in pod " + podName + ": " + e.getMessage();
-        logger.severe(msg);
-        failures.add(msg);
-      }
+    // 2. Check volumeMounts
+    List<V1VolumeMount> volumeMounts = container.getVolumeMounts();
+    boolean hasTmpMount = volumeMounts != null && volumeMounts.stream()
+        .anyMatch(mount -> mount.getMountPath() != null && mount.getMountPath().startsWith("/tmp"));
+    if (!hasTmpMount) {
+      String msg = "FAIL: container " + containerName + " in pod " + podName + " does not have /tmp mounted";
+      logger.severe(msg);
+      failures.add(msg);
     } else {
-      // 3. For init container, check volumeMounts
-      List<V1VolumeMount> volumeMounts = container.getVolumeMounts();
-      boolean hasTmpMount = volumeMounts != null && volumeMounts.stream()
-          .anyMatch(mount -> mount.getMountPath() != null && mount.getMountPath().startsWith("/tmp"));
-      if (!hasTmpMount) {
-        String msg = "FAIL: Init container " + containerName + " in pod " + podName + " does not have /tmp mounted";
-        logger.severe(msg);
-        failures.add(msg);
-      } else {
-        logger.info("PASS: Init container " + containerName + " in pod " + podName + " has /tmp mounted");
-      }
+      logger.info("PASS: container " + containerName + " in pod " + podName + " has /tmp mounted");
     }
   }
 

integration-tests/src/test/java/oracle/weblogic/kubernetes/actions/impl/Domain.java

@@ -4,6 +4,7 @@
 package oracle.weblogic.kubernetes.actions.impl;
 
 import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.ArrayList;
@@ -1068,12 +1069,39 @@ private static boolean copyFileFromPod(String namespace,
     return true;
   }
 
-  private static void scaleViaScript(String opNamespace, String domainNamespace,
+  private static boolean scaleViaScript(String opNamespace, String domainNamespace,
                                      String domainUid, String scalingAction, String clusterName,
                                      String opServiceAccount, int scalingSize,
                                      String domainHomeLocation,
                                      V1Pod adminPod) {
     LoggingFacade logger = getLogger();
+
+    String secretName = Secret.getSecretOfServiceAccount(opNamespace, opServiceAccount);
+    if (secretName.isEmpty()) {
+      logger.info("Did not find secret of service account {0} in namespace {1}", opServiceAccount, opNamespace);
+      return false;
+    }
+    logger.info("Got secret {0} of service account {1} in namespace {2}",
+        secretName, opServiceAccount, opNamespace);
+
+    logger.info("Getting service account token stored in secret {0} to authenticate as service account {1}"
+        + " in namespace {2}", secretName, opServiceAccount, opNamespace);
+    String secretToken = Secret.getSecretEncodedToken(opNamespace, secretName);
+    if (secretToken == null || secretToken.isEmpty()) {
+      logger.info("Did not get encoded token for secret {0} associated with service account {1} in namespace {2}",
+          secretName, opServiceAccount, opNamespace);
+      return false;
+    }
+    logger.info("Got encoded token for secret {0} associated with service account {1} in namespace {2}: {3}",
+        secretName, opServiceAccount, opNamespace, secretToken);
+
+    // decode the secret encoded token
+    String decodedToken = OKD ? secretToken : new String(Base64.getDecoder().decode(secretToken));
+    logger.info("Got decoded token for secret {0} associated with service account {1} in namespace {2}: {3}",
+        secretName, opServiceAccount, opNamespace, decodedToken);
+
+    assertNotNull(decodedToken, "Couldn't get secret, token is null");
+
     StringBuffer scalingCommand = new StringBuffer()
         //.append(Paths.get(domainHomeLocation + "/bin/scripts/scalingAction.sh"))
         .append(Paths.get("cd /u01; /u01/scalingAction.sh"))
@@ -1085,12 +1113,8 @@ private static void scaleViaScript(String opNamespace, String domainNamespace,
         .append(domainNamespace)
         .append(" --cluster_name=")
         .append(clusterName)
-        .append(" --operator_namespace=")
-        .append(opNamespace)
-        .append(" --operator_service_account=")
-        .append(opServiceAccount)
-        .append(" --operator_service_name=")
-        .append("internal-weblogic-operator-svc")
+        .append(" --access_token=")
+        .append(decodedToken)
         .append(" --scaling_size=")
         .append(scalingSize)
         .append(" --kubernetes_master=")
@@ -1103,6 +1127,8 @@ private static void scaleViaScript(String opNamespace, String domainNamespace,
     ExecResult result = null;
     assertNotNull(adminPod, "admin pod is null");
     assertNotNull(adminPod.getMetadata(), "admin pod metadata is null");
+    Path scalingActionLogPath = Paths.get(RESULTS_ROOT + "/" + domainUid + "-scalingAction.log");
+    Path scalingActionOutPath = Paths.get(RESULTS_ROOT + "/" + domainUid + "-scalingAction.out");
     try {
       result = assertDoesNotThrow(() -> Kubernetes.exec(adminPod, null, true,
           "/bin/sh", "-c", commandToExecuteInsidePod),
@@ -1119,42 +1145,58 @@ private static void scaleViaScript(String opNamespace, String domainNamespace,
       testUntil(
               () -> copyFileFromPod(domainNamespace, adminPod.getMetadata().getName(), null,
                       "/u01/scalingAction.log",
-                      Paths.get(RESULTS_ROOT + "/" + domainUid + "-scalingAction.log")),
+                      scalingActionLogPath),
               logger,
               "Copying scalingAction.log from admin server pod");
       // copy scalingAction.out to local
       testUntil(
               () -> copyFileFromPod(domainNamespace, adminPod.getMetadata().getName(), null,
                       "/u01/scalingAction.out",
-                      Paths.get(RESULTS_ROOT + "/" + domainUid + "-scalingAction.out")),
+                      scalingActionOutPath),
               logger,
               "Copying scalingAction.out from admin server pod");
+
+      try {
+        logger.info("Contents of scalingAction.log:\n" + Files.readString(scalingActionLogPath));
+        logger.info("Contents of scalingAction.out:\n" + Files.readString(scalingActionOutPath));
+      } catch (IOException ioex) {
+        // no-op
+      }
+
       throw err;
 
     }
     // copy scalingAction.log to local
     testUntil(
         () -> copyFileFromPod(domainNamespace, adminPod.getMetadata().getName(), null,
           "/u01/scalingAction.log",
-          Paths.get(RESULTS_ROOT + "/" + domainUid + "-scalingAction.log")),
+          scalingActionLogPath),
         logger,
         "Copying scalingAction.log from admin server pod");
     // copy scalingAction.out to local
     testUntil(
         () -> copyFileFromPod(domainNamespace, adminPod.getMetadata().getName(), null,
             "/u01/scalingAction.out",
-            Paths.get(RESULTS_ROOT + "/" + domainUid + "-scalingAction.out")),
+            scalingActionOutPath),
         logger,
         "Copying scalingAction.out from admin server pod");
     //      domainHomeLocation + "/bin/scripts/scalingAction.log",
 
+    try {
+      logger.info("Contents of scalingAction.log:\n" + Files.readString(scalingActionLogPath));
+      logger.info("Contents of scalingAction.out:\n" + Files.readString(scalingActionOutPath));
+    } catch (IOException ioex) {
+      // no-op
+    }
+
     // checking for exitValue 0 for success fails sometimes as k8s exec api returns non-zero exit value even on success,
     // so checking for exitValue non-zero and stderr not empty for failure, otherwise its success
 
     assertFalse(result.exitValue() != 0 && result.stderr() != null && !result.stderr().isEmpty(),
         String.format("Command %s failed with exit value %s, stderr %s, stdout %s",
             commandToExecuteInsidePod, result.exitValue(), result.stderr(), result.stdout()));
 
+    return true;
   }
 
 }

kubernetes/charts/weblogic-operator/templates/_operator-clusterrole-domain-admin.tpl

@@ -40,4 +40,7 @@ rules:
 - apiGroups: ["weblogic.oracle"]
   resources: ["domains/status", "clusters/status"]
   verbs: ["get", "watch"]
+- apiGroups: ["weblogic.oracle"]
+  resources: ["clusters/scale"]
+  verbs: ["update", "patch"]
 {{- end }}

kubernetes/charts/weblogic-operator/templates/_operator-clusterrole-general.tpl

@@ -33,6 +33,9 @@ rules:
 - apiGroups: ["weblogic.oracle"]
   resources: ["domains", "clusters", "domains/status", "clusters/status"]
   verbs: ["get", "create", "list", "watch", "update", "patch"]
+- apiGroups: ["weblogic.oracle"]
+  resources: ["clusters/scale"]
+  verbs: ["update", "patch"]
 - apiGroups: ["authentication.k8s.io"]
   resources: ["tokenreviews"]
   verbs: ["create"]

kubernetes/charts/weblogic-operator/templates/_operator-role.tpl

@@ -23,4 +23,10 @@ rules:
 - apiGroups: ["weblogic.oracle"]
   resources: ["domains", "clusters"]
   verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
+- apiGroups: ["weblogic.oracle"]
+  resources: ["domains/status", "clusters/status"]
+  verbs: ["get", "watch"]
+- apiGroups: ["weblogic.oracle"]
+  resources: ["clusters/scale"]
+  verbs: ["update", "patch"]
 {{- end }}