fix: generation of web.conf for endpoints mentioned in endpointUrl (#1884)

**Issue number:** ADDON-82845

### PR Type

**What kind of change does this PR introduce?**
* [x] Feature
* [x] Bug Fix
* [ ] Refactoring (no functional or API changes)
* [ ] Documentation Update
* [ ] Maintenance (dependency updates, CI, etc.)

## Summary

### Changes

UCC will now able to generate stanzas for endpoint mentioned in
`endpointUrl`. Also, earlier we were statically passing methods for
non-ui rest handlers which is now being dynamically set as users wants
them.

Added an attribute name `capabilities` through which users can set
required roles for their endpoints.


### User experience

Users no longer need to create stanzas for the endpoints specified in
`endpointUrl`, as these will now be generated automatically by UCC.
Additionally, users can define `capabilities` for their endpoints, which
will be reflected in `restmap.conf`.

## Checklist

If an item doesn't apply to your changes, leave it unchecked.

### Review

* [x] self-review - I have performed a self-review of this change
according to the [development
guidelines](https://splunk.github.io/addonfactory-ucc-generator/contributing/#development-guidelines)
* [x] Changes are documented. The documentation is understandable,
examples work [(more
info)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#documentation-guidelines)
* [x] PR title and description follows the [contributing
principles](https://splunk.github.io/addonfactory-ucc-generator/contributing/#pull-requests)
* [ ] meeting - I have scheduled a meeting or recorded a demo to explain
these changes (if there is a video, put a link below and in the ticket)

### Tests

See [the testing
doc](https://splunk.github.io/addonfactory-ucc-generator/contributing/#build-and-test).

* [x] Unit - tests have been added/modified to cover the changes
* [x] Smoke - tests have been added/modified to cover the changes
* [ ] UI - tests have been added/modified to cover the changes
* [x] coverage - I have checked the code coverage of my changes [(see
more)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#checking-the-code-coverage)

**Demo/meeting:**

*Reviewers are encouraged to request meetings or demos if any part of
the change is unclear*

Author: hetangmodi-crest

docs/configurations/index.md

@@ -12,6 +12,7 @@ proxy configuration, and logging level configuration.
 | description                                                       | string | To provide a brief summary of an configuration page.    |
 | [subDescription](../advanced/sub_description.md)                  | object | To provide broader description of an configuration page |
 | [tabs](#tabs-properties)<span class="required-asterisk">\*</span> | array  | To specify a list of tab.                               |
+| capabilities | object | Specifies the capabilities required for the given methods. The provided capabilities will be generated in restmap.conf. For more information, refer to the Splunk documentation on [restmap.conf](https://docs.splunk.com/Documentation/Splunk/latest/Admin/restmapconf#restmap.conf.spec).|
 
 ### Tabs properties
 
@@ -72,7 +73,11 @@ Currently available tab components:
         {
             "type": "loggingTab"
         }
-    ]
+    ],
+    "capabilities": {
+        "put": "admin_all_objects",
+        "post": "list_storage_passwords"
+    }
 }
 ```
 

docs/inputs/index.md

@@ -20,6 +20,7 @@ provided, a dropdown field will appear on the Inputs page. In contrast, a button
 | [table](../table.md)                                                      | object  | It displays input stanzas in a tabular format.                                                                                                                                                                                                                                                                                                       |
 | groupsMenu                                                                | array   | This property allows you to enable the [multi-level menu](./multilevel_menu.md) feature.                                                                                                                                                                                                                                                             |
 | [services](#services-properties)<span class="required-asterisk">\*</span> | array   | It specifies a list of modular inputs.                                                                                                                                                                                                                                                                                                               |
+| capabilities | object | Specifies the capabilities required for the given methods. The provided capabilities will be generated in restmap.conf. For more information, refer to the Splunk documentation on [restmap.conf](https://docs.splunk.com/Documentation/Splunk/latest/Admin/restmapconf#restmap.conf.spec).|
 | readonlyFieldId                                                           | string  | A field of the boolean entity that UCC checks for each input. If the field's value is [truthful](https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/ListOfDataTypes), the corresponding input cannot be edited from the UI. There is no way to change this from the UI; it is supposed to be changed via REST.                      |
 | hideFieldId                                                               | string  | A field of the boolean entity that UCC checks for each input. If the field's value is [truthful](https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/ListOfDataTypes), the corresponding input is hidden from the UI. There is no way to change this from the UI; it is supposed to be changed via REST. Check out an example below. |
 | useInputToggleConfirmation                                                | boolean | When true, displays a confirmation modal before toggling an input's status between active and inactive.                                                                                                                                                                                                                                                                 |

docs/options/rest_handlers.md

@@ -13,10 +13,16 @@ custom handlers developed by the user.
         "name": "StorageBuckets",
         "endpoint": "Splunk_TA_Example_buckets",
         "handlerType": "EAI",
+        "resourcePresent": true,
         "registerHandler": {
           "file": "storage_buckets.py",
           "actions": ["list", "create", "remove", "edit"]
         },
+        "capabilities": {
+          "put": "admin_all_objects",
+          "post": "list_storage_passwords"
+        },
+        "parentResource": "splunk_ta_uccexample",
         "requestParameters": {
           "create": {
             "param_name": {
@@ -57,6 +63,9 @@ OpenApi entries will be generated for the handler with the specified request and
 |--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | registerHandler    | If it is specified, UCC will add proper lines to `web.conf` and `restmap.conf`. You need to specify handler file name and EAI actions. Example: `{"file": "handler.py", "actions" ["create", "list"]}` |
 | requestParameters  | Request parameters for each action. The parameters are used in the OpenApi specification. See below.                                                                                                   |
+| resourcePresent | Specifies whether the collection should expose its resources as well via web.conf or not. Default value is False.|
+| capabilities |Specifies the capabilities required for the given methods. The provided capabilities will be generated in restmap.conf. For more information, refer to the Splunk documentation on [restmap.conf](https://docs.splunk.com/Documentation/Splunk/latest/Admin/restmapconf#restmap.conf.spec).|
+| parentResource | Specifies parent Resource for the given endpoint, it will get added in the `match` attribute of restmap.conf. Default value is "/.|
 | responseParameters | Response parameters for each action. The parameters are used in the OpenApi specification. See below.                                                                                                  |
 
 `registerHandler` is needed to register the handler in the `web.conf` and `restmap.conf` files.

splunk_add_on_ucc_framework/commands/rest_builder/global_config_builder_schema.py

@@ -57,8 +57,11 @@ def __init__(self, global_config: global_config_lib.GlobalConfig):
         self._settings_conf_file_names: List[str] = list()
         self._configs_conf_file_names: List[str] = list()
         self._oauth_conf_file_names: List[str] = list()
-        self._endpoints: Dict[str, RestEndpointBuilder] = {}
+        self._configuration_endpoints: Dict[str, RestEndpointBuilder] = {}
+        self._inputs_endpoints: Dict[str, RestEndpointBuilder] = {}
         self._parse_builder_schema()
+        self._endpoints: Dict[str, RestEndpointBuilder] = {}
+        self._preserve_endpoint_dict()
 
     @property
     def product(self) -> str:
@@ -84,10 +87,23 @@ def oauth_conf_file_names(self) -> List[str]:
     def endpoints(self) -> List[RestEndpointBuilder]:
         return list(self._endpoints.values())
 
+    @property
+    def configuration_endpoints(self) -> List[RestEndpointBuilder]:
+        return list(self._configuration_endpoints.values())
+
+    @property
+    def inputs_endpoints(self) -> List[RestEndpointBuilder]:
+        return list(self._inputs_endpoints.values())
+
     @property
     def need_reload(self) -> bool:
         return False
 
+    def _preserve_endpoint_dict(self) -> Dict[str, RestEndpointBuilder]:
+        self._endpoints.update(self._configuration_endpoints)
+        self._endpoints.update(self._inputs_endpoints)
+        return self._endpoints
+
     def _parse_builder_schema(self) -> None:
         self._builder_configs()
         self._builder_settings()
@@ -110,7 +126,7 @@ def _builder_configs(self) -> None:
                         log_stanza=log_details.get("name"),
                         log_level_field=log_details.get("entity", [{}])[0].get("field"),
                     )
-                    self._endpoints["oauth"] = oauth_endpoint
+                    self._configuration_endpoints["oauth"] = oauth_endpoint
                     if oauth_endpoint.conf_name not in self._oauth_conf_file_names:
                         self._oauth_conf_file_names.append(oauth_endpoint.conf_name)
 
@@ -152,7 +168,7 @@ def _builder_configs(self) -> None:
             else:
                 endpoint = SingleModelEndpointBuilder(**endpoint_params)
 
-            self._endpoints[name] = endpoint
+            self._configuration_endpoints[name] = endpoint
             content = self._get_oauth_enitities(config["entity"])
             fields, special_fields = self._parse_fields(content)
             entity = SingleModelEntityBuilder(
@@ -177,7 +193,7 @@ def _builder_settings(self) -> None:
             rest_handler_class=REST_HANDLER_DEFAULT_CLASS,
             need_reload=self.need_reload,
         )
-        self._endpoints["settings"] = endpoint
+        self._configuration_endpoints["settings"] = endpoint
         for setting in self.global_config.settings:
             if setting.get("entity") is not None:
                 content = self._get_oauth_enitities(setting["entity"])
@@ -212,7 +228,7 @@ def _builder_inputs(self) -> None:
                     rest_handler_class=rest_handler_class,
                     need_reload=self.need_reload,
                 )
-                self._endpoints[name] = single_model_endpoint
+                self._inputs_endpoints[name] = single_model_endpoint
                 content = self._get_oauth_enitities(input_item["entity"])
                 fields, special_fields = self._parse_fields(content)
                 single_model_entity = SingleModelEntityBuilder(
@@ -232,7 +248,7 @@ def _builder_inputs(self) -> None:
                     rest_handler_module=rest_handler_module,
                     rest_handler_class=rest_handler_class,
                 )
-                self._endpoints[name] = data_input_endpoint
+                self._inputs_endpoints[name] = data_input_endpoint
                 content = self._get_oauth_enitities(input_item["entity"])
                 fields, special_fields = self._parse_fields(content)
                 data_input_entity = DataInputEntityBuilder(

splunk_add_on_ucc_framework/commands/rest_builder/user_defined_rest_handlers.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 #
 from copy import deepcopy
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from typing import Dict, Any, Iterable, Optional, Set, List
 
 from splunk_add_on_ucc_framework.commands.openapi_generator import oas
@@ -73,6 +73,9 @@ class EndpointRegistrationEntry:
     name: str
     rh_name: str
     actions_list: List[str]
+    capabilities: Dict[str, str] = field(default_factory=dict)
+    resourcePresent: bool = False
+    parentResource: str = "/"
 
     def actions(self) -> List[str]:
         """
@@ -90,6 +93,9 @@ class RestHandlerConfig:
     name: str
     endpoint: str
     handlerType: str
+    capabilities: Dict[str, str] = field(default_factory=dict)
+    resourcePresent: bool = False
+    parentResource: str = "/"
     registerHandler: Optional[Dict[str, Any]] = None
     requestParameters: Optional[Dict[str, Dict[str, Any]]] = None
     responseParameters: Optional[Dict[str, Dict[str, Any]]] = None
@@ -305,6 +311,9 @@ def endpoint_registration_entry(self) -> Optional[EndpointRegistrationEntry]:
             name=self.endpoint,
             rh_name=file,
             actions_list=self.registerHandler["actions"],
+            resourcePresent=self.resourcePresent,
+            parentResource=self.parentResource,
+            capabilities=self.capabilities,
         )
 
 

splunk_add_on_ucc_framework/generators/conf_files/create_restmap_conf.py

@@ -37,15 +37,33 @@ def __init__(
         super().__init__(global_config, input_dir, output_dir)
         self.conf_file = "restmap.conf"
         self.endpoints: List[Union[RestEndpointBuilder, EndpointRegistrationEntry]] = []
+        self.configuration_endpoints: List[
+            Union[RestEndpointBuilder, EndpointRegistrationEntry]
+        ] = []
+        self.inputs_endpoints: List[
+            Union[RestEndpointBuilder, EndpointRegistrationEntry]
+        ] = []
+        self.custom_endpoints: List[
+            Union[RestEndpointBuilder, EndpointRegistrationEntry]
+        ] = []
 
         if global_config.has_pages():
-            self.endpoints.extend(self._gc_schema.endpoints)
+            self.configuration_endpoints.extend(self._gc_schema.configuration_endpoints)
+            self.inputs_endpoints.extend(self._gc_schema.inputs_endpoints)
             self.namespace = self._gc_schema.namespace
-            self.endpoints.extend(
-                global_config.user_defined_handlers.endpoint_registration_entries
-            )
+        self.custom_endpoints.extend(
+            global_config.user_defined_handlers.endpoint_registration_entries
+        )
 
-        self.endpoint_names = ", ".join(sorted([ep.name for ep in self.endpoints]))
+        self.configuration_endpoint_names = ", ".join(
+            sorted([ep.name for ep in self.configuration_endpoints])
+        )
+        self.inputs_endpoint_names = ", ".join(
+            sorted([ep.name for ep in self.inputs_endpoints])
+        )
+        self.endpoints.extend(self.configuration_endpoints)
+        self.endpoints.extend(self.inputs_endpoints)
+        self.endpoints.extend(self.custom_endpoints)
 
     def generate(self) -> Optional[List[Dict[str, str]]]:
         if not self.endpoints:
@@ -57,8 +75,14 @@ def generate(self) -> Optional[List[Dict[str, str]]]:
         )
         rendered_content = self._template.render(
             endpoints=self.endpoints,
-            endpoint_names=self.endpoint_names,
+            configuration_endpoints=self.configuration_endpoints,
+            inputs_endpoints=self.inputs_endpoints,
+            configuration_endpoint_names=self.configuration_endpoint_names,
+            inputs_endpoint_names=self.inputs_endpoint_names,
             namespace=self.namespace,
+            configuration_capability=self._global_config.capabilities(config=True),
+            input_capability=self._global_config.capabilities(inputs=True),
+            custom_endpoints=self.custom_endpoints,
         )
         return [
             {

splunk_add_on_ucc_framework/generators/conf_files/create_web_conf.py

@@ -24,6 +24,13 @@
 from splunk_add_on_ucc_framework.generators.file_generator import FileGenerator
 from splunk_add_on_ucc_framework.global_config import GlobalConfig
 
+actions_to_methods = {
+    "edit": ["POST", "GET"],
+    "remove": ["DELETE"],
+    "list": ["GET"],
+    "create": ["POST"],
+}
+
 
 class WebConf(FileGenerator):
     __description__ = (
@@ -36,21 +43,38 @@ def __init__(
     ) -> None:
         super().__init__(global_config, input_dir, output_dir)
 
+    def get_methods_for_actions(self, actions: List[str]) -> List[str]:
+        methods = []
+        for action in actions:
+            methods.extend(actions_to_methods.get(action, []))
+
+        return sorted(list(set(methods)))
+
     def generate(self) -> Optional[List[Dict[str, str]]]:
         if not self._global_config.has_pages():
             return None
 
         endpoints: List[Union[RestEndpointBuilder, EndpointRegistrationEntry]] = []
+        custom_endpoints: List[
+            Union[RestEndpointBuilder, EndpointRegistrationEntry]
+        ] = []
         endpoints.extend(self._gc_schema.endpoints)
-        endpoints.extend(
+        methods = {}
+        custom_endpoints.extend(
             self._global_config.user_defined_handlers.endpoint_registration_entries
         )
+
+        for endpoint in custom_endpoints:
+            methods[endpoint.name] = self.get_methods_for_actions(endpoint.actions())
         conf_file = "web.conf"
 
         file_path = self.get_file_output_path(["default", conf_file])
         rendered_content = self._render(
             "web_conf.template",
             endpoints=endpoints,
+            custom_endpoints=custom_endpoints,
+            methods=methods,
+            endpointUrl=self._global_config.endpointUrl,
         )
         return [
             {

splunk_add_on_ucc_framework/global_config.py

@@ -15,8 +15,9 @@
 #
 import functools
 import json
-from typing import Optional, Any, List, Dict
+from typing import Optional, Any, List, Dict, Set
 from dataclasses import dataclass, field, fields
+from urllib.parse import urlparse
 
 import yaml
 
@@ -166,6 +167,30 @@ def configuration(self) -> List[Any]:
             return self._content["pages"]["configuration"]["tabs"]
         return []
 
+    @property
+    def endpointUrl(self) -> List[str]:
+        def extract_urls(entities_list: List[Any]) -> Set[str]:
+            urls = set()
+            for content in entities_list:
+                for entity in content.get("entity", []):
+                    if entity.get("type") in ("multipleSelect", "singleSelect"):
+                        endpoint = entity.get("options", {}).get("endpointUrl")
+                        # only include those endpoint which starts with data
+                        # the other ones are covered if they are defined in options->
+                        # TODO; If add-ons uses any other endpoints apart from data then add it here.
+                        if endpoint and endpoint.startswith("data"):
+                            urls.add(urlparse(endpoint).path)
+            return urls
+
+        urls = set()
+        if self.has_configuration():
+            urls.update(extract_urls(self.configuration))
+
+        if self.has_inputs():
+            urls.update(extract_urls(self.inputs))
+
+        return list(urls)
+
     @property
     def pages(self) -> List[Any]:
         if "pages" in self._content:
@@ -233,6 +258,13 @@ def display_name(self) -> str:
     def version(self) -> str:
         return self.meta["version"]
 
+    def capabilities(self, **kwargs: bool) -> Dict[str, str]:
+        if kwargs.get("config") and self.has_configuration():
+            return self._content["pages"]["configuration"].get("capabilities", {})
+        elif kwargs.get("inputs") and self.has_inputs():
+            return self._content["pages"]["inputs"].get("capabilities", {})
+        return {}
+
     @property
     def ucc_version(self) -> str:
         return self.meta["_uccVersion"]