Closes #689 - Support semantic model binding per environment #692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

Copilot wants to merge 23 commits into main from copilot/support-semantic-model-binding

.prettierignore

Original file line number	Diff line number	Diff line change
		@@ -1 +1,2 @@
		sample/workspace/
		docs/how_to/parameterization.md

docs/how_to/parameterization.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -50,8 +50,15 @@ spark_pool: @@
                   name: "PROD-Pool-name"
     semantic_model_binding:
-        - connection_id: "connection_id"
-          semantic_model_name: "semantic_model_name"
+        default:
+            connection_id:
+                PPE: "PPE-connection_id"
+                PROD: "PROD-connection_id"
+        models:
+            - semantic_model_name: "semantic_model_name"
+              connection_id:
+                  PPE: "PPE-connection_id"
+                  PROD: "PROD-connection_id"
     ```
     Raise a [feature request](https://github.com/microsoft/fabric-cicd/issues/new?template=2-feature.yml) for additional parameterization capabilities.
@@ Expand Down Expand Up / @@ -138,19 +145,51 @@ spark_pool: @@
     ### `semantic_model_binding`
-    Semantic model binding is used to connect semantic models that require cloud or on-premises data sources to the appropriate connection after deployment. The `semantic_model_binding` parameter automatically configures these connections during the deployment process, ensuring your semantic models can refresh data from cloud and on-premises sources in the target environment.
+    Semantic model binding automatically connects semantic models to the appropriate data source connection (e.g., cloud or on-premises) after deployment, ensuring your models can refresh data in the target environment.
+    **Important:** The legacy format is on a deprecation path. Please migrate to the recommended format.
+    **Recommended format:**
     ```yaml
     semantic_model_binding:
-        # Required field: value must be a string (GUID)
-        # Connection Ids can be found from the Fabric UI under Settings -> Manage Connections and gateways -> Settings pane of the connection
-        - connection_id: <connection_id>
-        # Required field: value must be a string or a list of strings
-          semantic_model_name: <semantic_model_name>
-        # OR
-          semantic_model_name: [<semantic_model_name1>,<semantic_model_name2>,...]
+        # Default connection for all models not explicitly listed
+        default:
+            connection_id:
+                PPE: <PPE-connection_guid>
+                PROD: <PROD-connection_guid>
+                # OR use _ALL_ for same connection across environments
+                # _ALL_: <connection_guid>
+        # Explicit bindings override default
+        models:
+            - semantic_model_name: "<semantic_model_name>"
+              connection_id:
+                  PPE: <PPE-connection_guid>
+                  PROD: <PROD-connection_guid>
+            - semantic_model_name: ["<semantic_model_name1>", "<semantic_model_name2>", ...]
+              connection_id:
+                  _ALL_: <connection_guid>
     ```
+    **Legacy format:**
+    ```yaml
+    # Legacy format:
+    semantic_model_binding:
+        - connection_id: <connection_guid>
+          # Required field: value must be a string or a list of strings
+          semantic_model_name: "<semantic_model_name>"
+          # OR
+          semantic_model_name: ["<semantic_model_name1>","<semantic_model_name2>", ...]
+    ```
+    **Notes:**
+    - The `_ALL_` environment key (case-insensitive) can be used in the `connection_id` dictionary to apply the same connection to any target environment.
+    - Connection ID values must be valid GUIDs.
     ## Advanced Find and Replace
     ### `find_value` Regex
@@ Expand Down @@

sample/workspace/parameter.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -126,10 +126,24 @@ spark_pool: @@
           # Optional field:
           item_name:
+    # Deprecation notice: 'gateway_binding' will not be supported in future releases.
     gateway_binding:
         - gateway_id: "f96870d5-5f86-49ad-bf41-5967fd7c1c6d"
           dataset_name: "gateway_sm2"
+    # Deprecation notice: Legacy format will not be supported in future releases.
     semantic_model_binding:
         - connection_id: "76e05dfe-9855-4e3d-a410-1dda048dbe99"
           semantic_model_name: ["cloudconnections", "MySemanticModel_ADLS_Gen2"]
+    # New format (recommended): Supports default connections and per-item overrides
+    semantic_model_binding:
+        default:
+            connection_id:
+                PPE: "76e05dfe-9855-4e3d-a410-1dda048dbe99"
+                PROD: "c4f8e2b1-3d2a-4f5b-9c6e-7a8b9c0d1e2f"
+        models:
+            - semantic_model_name: ["cloudconnections", "MySemanticModel_ADLS_Gen2"]
+              connection_id:
+                PPE: "f96870d5-5f86-49ad-bf41-5967fd7c1c6d"
+                PROD: "a1b2c3d4-5678-90ab-cdef-1234567890ab"

src/fabric_cicd/_items/_environment.py

-Original file line number
+Diff line change
@@ Expand Up @@
                 parameter_dict = fabric_workspace_obj.environment_parameter["spark_pool"]
                 for key in parameter_dict:
                     instance_pool_id = key["instance_pool_id"]
-                    replace_value = process_environment_key(fabric_workspace_obj, key["replace_value"])
+                    replace_value = process_environment_key(fabric_workspace_obj.environment, key["replace_value"])
                     input_name = key.get("item_name")
                     if instance_pool_id == pool_id and (input_name == item_name or not input_name):
                         # replace any found references with specified environment value
@@ Expand Down @@

src/fabric_cicd/_items/_semanticmodel.py

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -8,11 +8,129 @@
  
    from fabric_cicd import FabricWorkspace, constants

    from fabric_cicd._common._item import Item

    from fabric_cicd._items._base_publisher import ItemPublisher

    from fabric_cicd._parameter._utils import process_environment_key

    from fabric_cicd.constants import EXCLUDE_PATH_REGEX_MAPPING, ItemType

    logger = logging.getLogger(__name__)

    def build_binding_mapping_legacy(fabric_workspace_obj: FabricWorkspace, semantic_model_binding: list) -> dict:

        """

        Build the connection mapping from legacy list-based semantic_model_binding parameter.

        Args:

            fabric_workspace_obj: The FabricWorkspace object

            semantic_model_binding: The semantic_model_binding parameter as a list

        Returns:

            Dictionary mapping semantic model names to connection IDs

        """

        logger.warning(

            "The legacy 'semantic_model_binding' list format is deprecated and will be removed in a future release. "

            "Please migrate to the new dictionary format with 'default' and 'models' keys. "

            "See: https://microsoft.github.io/fabric-cicd/how_to/parameterization/"

        )

        item_type = "SemanticModel"

        binding_mapping = {}

        repository_models = set(fabric_workspace_obj.repository_items.get(item_type, {}).keys())

        for entry in semantic_model_binding:

            connection_id = entry.get("connection_id")

            model_names = entry.get("semantic_model_name", [])

            if not connection_id:

                logger.debug("No connection_id found in semantic_model_binding entry, skipping")

                continue

            # Legacy format only supports string connection_id

            if isinstance(connection_id, dict):

                logger.warning(

                    "Environment-specific connection_id dictionaries are not supported in the legacy format. "

                    "Please migrate to the new dictionary format to use environment-specific values."

                )

                continue

            if isinstance(model_names, str):

                model_names = [model_names]

            for name in model_names:

                if name not in repository_models:

                    logger.warning(f"Semantic model '{name}' specified in parameter.yml not found in repository")

                    continue

                binding_mapping[name] = connection_id

        return binding_mapping

    def build_binding_mapping(

        fabric_workspace_obj: FabricWorkspace, semantic_model_binding: dict, environment: str

    ) -> dict:

        """

        Build the connection mapping from semantic_model_binding parameter. The new format requires

        environment-specific connection_id values (use '_ALL_' for all environments).

        Supports:

        - default.connection_id: Applied to all models in the repository that are not explicitly listed

        - models: List of explicit model-to-connection mappings

        Args:

            fabric_workspace_obj: The FabricWorkspace object

            semantic_model_binding: The semantic_model_binding parameter dictionary

            environment: The target environment name (_ALL_ key can be used)

        Returns:

            Dictionary mapping semantic model names to connection IDs

        """

        item_type = "SemanticModel"

        binding_mapping = {}

        repository_models = set(fabric_workspace_obj.repository_items.get(item_type, {}).keys())

        # Get default connection_id for this environment

        default_connection_id = None

        default_config = semantic_model_binding.get("default", {})

        if default_config:

            connection_id_config = default_config.get("connection_id", {})

            connection_id_config = process_environment_key(fabric_workspace_obj, connection_id_config)

            default_connection_id = connection_id_config.get(environment)

            if not default_connection_id:

                logger.debug(f"Environment '{environment}' not found in default.connection_id")

        # Process explicit model bindings

        explicit_models = set()

        models_config = semantic_model_binding.get("models", [])

        for model in models_config:

            model_names = model.get("semantic_model_name", [])

            connection_id_config = model.get("connection_id", {})

            if isinstance(model_names, str):

                model_names = [model_names]

            connection_id_config = process_environment_key(fabric_workspace_obj, connection_id_config)

            connection_id = connection_id_config.get(environment)

            if not connection_id:

                logger.debug(f"Environment '{environment}' not found in connection_id for semantic model(s): {model_names}")

                continue

            # Track models with explicit bindings to exclude from default connection assignment

            explicit_models.update(model_names)

            for name in model_names:

                if name not in repository_models:

                    logger.warning(f"Semantic model '{name}' specified in parameter.yml not found in repository")

                    continue

                binding_mapping[name] = connection_id

        # Apply default connection to non-explicit models

        if default_connection_id:

            default_models = repository_models - explicit_models

            for model_name in default_models:

                binding_mapping[model_name] = default_connection_id

                logger.debug(f"Applying default connection to semantic model '{model_name}'")

        return binding_mapping

    def get_connections(fabric_workspace_obj: FabricWorkspace) -> dict:

        """

        Get all connections from the workspace.

    @@ -23,6 +141,7 @@ def get_connections(fabric_workspace_obj: FabricWorkspace) -> dict:
  
        Returns:

            Dictionary with connection ID as key and connection details as value

        """

        # https://learn.microsoft.com/en-us/rest/api/fabric/core/connections/list-connections

        connections_url = f"{constants.FABRIC_API_ROOT_URL}/v1/connections"

        try:

    @@ -54,36 +173,31 @@ def bind_semanticmodel_to_connection(
  
        Args:

            fabric_workspace_obj: The FabricWorkspace object containing the items to be published.

            connections: Dictionary of connection objects with connection ID as key.

            connection_details: Dictionary mapping dataset names to connection IDs from parameter.yml.

            connection_details: Dictionary mapping semantic model names to connection IDs from parameter.yml.

        """

        item_type = ItemType.SEMANTIC_MODEL.value

        # Loop through each semantic model in the semantic_model_binding section

        for dataset_name, connection_id in connection_details.items():

        for model_name, connection_id in connection_details.items():

            # Check if the connection ID exists in the connections dict

            if connection_id not in connections:

                logger.warning(f"Connection ID '{connection_id}' not found for semantic model '{dataset_name}'")

                continue

            # Check if this semantic model exists in the repository

            if dataset_name not in fabric_workspace_obj.repository_items.get(item_type, {}):

                logger.warning(f"Semantic model '{dataset_name}' not found in repository")

                logger.warning(f"Connection ID '{connection_id}' not found for semantic model '{model_name}'")

                continue

            # Get the semantic model object

            item_obj = fabric_workspace_obj.repository_items[item_type][dataset_name]

            # Get the semantic model object (validated during binding mapping creation)

            item_obj = fabric_workspace_obj.repository_items[item_type][model_name]

            model_id = item_obj.guid

            logger.info(f"Binding semantic model '{dataset_name}' (ID: {model_id}) to connection '{connection_id}'")

            logger.info(f"Binding semantic model '{model_name}' (ID: {model_id}) to connection '{connection_id}'")

            try:

                # Get the connection details for this semantic model from Fabric API

                # https://learn.microsoft.com/en-us/rest/api/fabric/core/items/list-item-connections

                item_connections_url = f"{constants.FABRIC_API_ROOT_URL}/v1/workspaces/{fabric_workspace_obj.workspace_id}/items/{model_id}/connections"

                connections_response = fabric_workspace_obj.endpoint.invoke(method="GET", url=item_connections_url)

                connections_data = connections_response.get("body", {}).get("value", [])

                if not connections_data:

                    logger.warning(f"No connections found for semantic model '{dataset_name}'")

                    logger.debug(f"No existing connections found for semantic model '{model_name}', skipping binding")

                    continue

                # Use the first connection as the template

    @@ -98,22 +212,23 @@ def bind_semanticmodel_to_connection(
  
                request_body = build_request_body({"connectionBinding": connection_binding})

                # Make the bind connection API call

                powerbi_url = f"{constants.FABRIC_API_ROOT_URL}/v1/workspaces/{fabric_workspace_obj.workspace_id}/semanticModels/{model_id}/bindConnection"

                # https://learn.microsoft.com/en-us/rest/api/fabric/semanticmodel/items/bind-semantic-model-connection

                binding_url = f"{constants.FABRIC_API_ROOT_URL}/v1/workspaces/{fabric_workspace_obj.workspace_id}/semanticModels/{model_id}/bindConnection"

                bind_response = fabric_workspace_obj.endpoint.invoke(

                    method="POST",

                    url=powerbi_url,

                    url=binding_url,

                    body=request_body,

                )

                status_code = bind_response.get("status_code")

                if status_code == 200:

                    logger.info(f"Successfully bound semantic model '{dataset_name}' to connection '{connection_id}'")

                    logger.info(f"Successfully bound semantic model '{model_name}' to connection '{connection_id}'")

                else:

                    logger.warning(f"Failed to bind semantic model '{dataset_name}'. Status code: {status_code}")

                    logger.warning(f"Failed to bind semantic model '{model_name}'. Status code: {status_code}")

            except Exception as e:

                logger.error(f"Failed to bind semantic model '{dataset_name}' to connection: {e!s}")

                logger.error(f"Failed to bind semantic model '{model_name}' to connection: {e!s}")

                continue

    @@ -155,27 +270,26 @@ def publish_one(self, item_name: str, _item: Item) -> None:
  
        def post_publish_all(self) -> None:

            """Bind semantic models to connections after all models are published."""

            model_with_binding_dict = self.fabric_workspace_obj.environment_parameter.get("semantic_model_binding", [])

            if not model_with_binding_dict:

            semantic_model_binding = self.fabric_workspace_obj.environment_parameter.get("semantic_model_binding", {})

            if not semantic_model_binding:

                return

            # Build connection mapping from semantic_model_binding parameter

            binding_mapping = {}

            for model in model_with_binding_dict:

                model_name = model.get("semantic_model_name", [])

                connection_id = model.get("connection_id")

                if isinstance(model_name, str):

                    model_name = [model_name]

                for name in model_name:

                    binding_mapping[name] = connection_id

            connections = get_connections(self.fabric_workspace_obj)

            # Build connection mapping from semantic_model_binding parameter (support legacy or new formats)

            environment = self.fabric_workspace_obj.environment

            if isinstance(semantic_model_binding, list):

                binding_mapping = build_binding_mapping_legacy(self.fabric_workspace_obj, semantic_model_binding)

            elif isinstance(semantic_model_binding, dict):

                binding_mapping = build_binding_mapping(self.fabric_workspace_obj, semantic_model_binding, environment)

            else:

                logger.warning(

                    f"Invalid 'semantic_model_binding' type: {type(semantic_model_binding).__name__}. "

                    "Expected list or dict. Skipping semantic model binding."

                )

                return

            if binding_mapping:

                connections = get_connections(self.fabric_workspace_obj)

                bind_semanticmodel_to_connection(

                    fabric_workspace_obj=self.fabric_workspace_obj,

                    connections=connections,

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Closes #689 - Support semantic model binding per environment #692

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Closes #689 - Support semantic model binding per environment #692

Are you sure you want to change the base?

Uh oh!

Closes #689 - Support semantic model binding per environment #692

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!