[K8S][HELM] Add Hadoop configuration files support

[dnskr]

Why are the changes needed?

The PR adds support for Hadoop configuration files to be used by Apache Kyuubi, Apache Spark etc.

The PR is continuation of PR #6521 and relates to the issue #6123.

How was this patch tested?

1. Create hadoop-configs.yaml file (ConfigMap with core-site.xml and hive-site.xml entries):

apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-configs
data:
  'core-site.xml': |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
        <property>
            <name>hadoop.pr.test</name>
            <value>configmap</value>
        </property>
    </configuration>
  'hive-site.xml': |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
        <property>
            <name>hive.pr.test</name>
            <value>configmap</value>
        </property>
    </configuration>

2. Create ConfigMap from hadoop-configs.yaml file:

kubectl create -f hadoop-configs.yaml

3. Create custom values-hadoop.yaml (overwrites core-site.xml):

image:
  repository: apache/kyuubi
  tag: 1.10.0-spark

rbac:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods", "configmaps", "services"]
      verbs: ["create", "list", "delete", "watch", "deletecollection", "get"]

hadoopConf:
  files:
   'core-site.xml': |
     <?xml version="1.0"?>
     <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
     <configuration>
       <property>
         <name>hadoop.pr.test</name>
         <value>values</value>
       </property>
     </configuration>
   'hdfs-site.xml': |
     <?xml version="1.0"?>
     <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
     <configuration>
       <property>
         <name>hdfs.pr.test</name>
         <value>values</value>
       </property>
     </configuration>
  filesFrom:
  - configMap:
      name: hadoop-configs

sparkConf:
  files:
    'spark-defaults.conf': |
      spark.submit.deployMode=client
      spark.kubernetes.container.image=apache/spark:3.5.2
      spark.kubernetes.authenticate.driver.serviceAccountName=kyuubi

4. Install the chart

helm install kyuubi charts/kyuubi -f values-hadoop.yaml

5. Check there are 3 files in the Hadoop configuration directory:

kubectl exec kyuubi-0 -- ls /opt/hadoop/conf

core-site.xml
hdfs-site.xml
hive-site.xml

6. Check /opt/hadoop/conf/core-site.xml has content from ConfigMap:

kubectl exec kyuubi-0 -- cat /opt/hadoop/conf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hadoop.pr.test</name>
        <value>configmap</value>
    </property>
</configuration>

7. Check /opt/hadoop/conf/hdfs-site.xml has content from values:

kubectl exec kyuubi-0 -- cat /opt/hadoop/conf/hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hdfs.pr.test</name>
    <value>values</value>
  </property>
</configuration>

8. Check /opt/hadoop/conf/hive-site.xml has content from ConfigMap:

kubectl exec kyuubi-0 -- cat /opt/hadoop/conf/hive-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.pr.test</name>
        <value>configmap</value>
    </property>
</configuration>

9. Check configuration values from Spark session:

kubectl exec kyuubi-0 -- ./bin/beeline -u 'jdbc:hive2://kyuubi-thrift-binary:10009' -e 'set hadoop.pr.test;'
+-----------------+------------+
|       key       |   value    |
+-----------------+------------+
| hadoop.pr.test  | configmap  |
+-----------------+------------

kubectl exec kyuubi-0 -- ./bin/beeline -u 'jdbc:hive2://kyuubi-thrift-binary:10009' -e 'set hdfs.pr.test;'
+---------------+---------+
|      key      |  value  |
+---------------+---------+
| hdfs.pr.test  | values  |
+---------------+---------

kubectl exec kyuubi-0 -- ./bin/beeline -u 'jdbc:hive2://kyuubi-thrift-binary:10009' -e 'set hive.pr.test;'
+---------------+------------+
|      key      |   value    |
+---------------+------------+
| hive.pr.test  | configmap  |
+---------------+------------

Was this patch authored or co-authored using generative AI tooling?

No

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (a051253) to head (8c8665f).
Report is 4 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6875   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         688     688           
  Lines       42551   42551           
  Branches     5801    5801           
======================================
  Misses      42551   42551           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dnskr]

Thanks for the review, merged to master.