Az Monitor Extension: **az monitor data-collection rule create** is buggy

[AfstoneBharfon]

Describe the bug

The CLI command az monitor data-collection rule create can take a Data Collection Rule (DCR) payload json file as input.

This command seems to ignore fields like streamDeclarations and logFiles which are part of the DCR model.

This is our latest swagger: https://github.com/Azure/azure-rest-api-specs/blob/fce14bf881523f16b14d8ecf325e004465cdcf56/specification/monitor/resource-manager/Microsoft.Insights/stable/2022-06-01/dataCollectionRules_API.json

Using the command above in debug mode with the below input file, I noticed that streamDeclarations and logFiles were missing from the request payload.

The goal is to align the model supported by the CLI and the one specified in our swagger specs.

custom-logs-dcr-payload.json

Related command

az monitor data-collection rule create --resource-group rg-komla --location centralus --name custom-logs-dcr-cli-1 --rule-file custom-logs-dcr-payload.json --debug

Errors

We are getting a bad request from the RP because of those missing fields

Issue script & Debug output

cli.knack.cli: Command arguments: ['monitor', 'data-collection', 'rule', 'create', '--resource-group', 'rg-komla', '--location', 'centralus', '--name', 'custom-logs-dcr-cli-1', '--rule-file', 'custom-logs-dcr-payload.json', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
Init colorama.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x01C0BD60>, <function OutputProducer.on_global_arguments at 0x01CCC6A0>, <function CLIQuery.on_global_arguments at 0x01CDF2F8>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'monitor': ['azure.cli.command_modules.monitor', 'azext_amcs']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: monitor 0.738 32 123
cli.azure.cli.core: Total (1) 0.738 32 123
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: monitor-control-service 0.034 10 37 C:\Users\komlaafoutou.azure\cliextensions\monitor-control-service
cli.azure.cli.core: Total (1) 0.034 10 37
cli.azure.cli.core: Loaded 41 groups, 160 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : monitor data-collection rule create
cli.azure.cli.core: Command table: monitor data-collection rule create
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x03FEE2F8>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\komlaafoutou.azure\commands\2023-11-27.14-31-48.monitor_data-collection_rule_create.30712.log'.
az_command_data_logger: command args: monitor data-collection rule create --resource-group {} --location {} --name {} --rule-file {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x040414A8>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x04041580>, <function register_cache_arguments..add_cache_arguments at 0x040559B8>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x01CCC6E8>, <function CLIQuery.handle_query_parameter at 0x01CDF340>, <function register_ids_argument..parse_ids_arguments at 0x04055970>]
az_command_data_logger: extension name: monitor-control-service
az_command_data_logger: extension version: 0.4.1
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=MonitorClient
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\Users\komlaafoutou\.azure\msal_token_cache.bin', encrypt=True
cli.azure.cli.core.auth.binary_cache: load: C:\Users\komlaafoutou.azure\msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
cli.azure.cli.core.auth.binary_cache: save: C:\Users\komlaafoutou.azure\msal_http_cache.bin
cli.azure.cli.core.auth.binary_cache: save: C:\Users\komlaafoutou.azure\msal_http_cache.bin
urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443
urllib3.connectionpool: https://login.microsoftonline.com:443 "GET /72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0/.well-known/openid-configuration HTTP/1.1" 200 1753
cli.azure.cli.core.auth.binary_cache: save: C:\Users\komlaafoutou.azure\msal_http_cache.bin
cli.azure.cli.core.auth.binary_cache: save: C:\Users\komlaafoutou.azure\msal_http_cache.bin
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
cli.azure.cli.core.util: attempting to read file custom-logs-dcr-payload.json as utf-8-sig
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.application: Found 1 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': '0e3e66e4-476e-4f44-9e74-2ac5cd23aca3.72f988bf-86f1-41af-91ab-2d7cd011db47', 'family_id': '1'}
msal.telemetry: Generate or reuse correlation_id: 6e9368e7-3b76-4d07-8d5a-fb5a76eb8961
msal.application: Cache attempts an RT
urllib3.connectionpool: https://login.microsoftonline.com:443 "POST /72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token HTTP/1.1" 200 5519
msal.token_cache: event={
"client_id": "04b07795-8ddb-461a-bbee-02f9e1bf7b46",
"data": {
"claims": null,
"refresh_token": "",
"scope": [
"https://management.core.windows.net//.default",
"offline_access",
"profile",
"openid"
]
},
"environment": "login.microsoftonline.com",
"grant_type": "refresh_token",
"params": null,
"response": {
"access_token": "",
"client_info": "eyJ1aWQiOiIwZTNlNjZlNC00NzZlLTRmNDQtOWU3NC0yYWM1Y2QyM2FjYTMiLCJ1dGlkIjoiNzJmOTg4YmYtODZmMS00MWFmLTkxYWItMmQ3Y2QwMTFkYjQ3In0",
"expires_in": 5208,
"ext_expires_in": 5208,
"foci": "1",
"id_token": "***",
"scope": "https://management.core.windows.net//user_impersonation https://management.core.windows.net//.default",
"token_type": "Bearer"
},
"scope": [
"https://management.core.windows.net//user_impersonation",
"https://management.core.windows.net//.default"
],
"skip_account_creation": true,
"token_endpoint": "https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token"
}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/6f1b91a9-5242-4017-bc01-e262558ddab7/resourceGroups/rg-komla/providers/Microsoft.Insights/dataCollectionRules/custom-logs-dcr-cli-1?api-version=2022-06-01'
cli.azure.cli.core.sdk.policies: Request method: 'PUT'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies: 'Content-Length': '904'
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'c03290e2-8d74-11ee-b3d5-c03c598a8bf7'
cli.azure.cli.core.sdk.policies: 'CommandName': 'monitor data-collection rule create'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--resource-group --location --name --rule-file --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.39.0 (MSI) azsdk-python-mgmt-monitor/1.0.0b1 Python/3.10.5 (Windows-10-10.0.22621-SP0)'
cli.azure.cli.core.sdk.policies: 'Authorization': ''
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: {"location": "centralus", "properties": {"dataSources": {}, "destinations": {"logAnalytics": [{"workspaceResourceId": "/subscriptions/6f1b91a9-5242-4017-bc01-e262558ddab7/resourcegroups/rg-komla/providers/microsoft.operationalinsights/workspaces/m-la-01", "name": "myloganalyticsworkspace"}]}, "dataFlows": [{"streams": ["Custom-stream-1"], "destinations": ["myloganalyticsworkspace"], "transformKql": "source | project TimeGenerated = now(), time_current_utc = dt, time_sunrise_utc = sys.sunrise, time_sunset_utc = sys.sunset, city = name, longitude = coord.lon, latitude = coord.lat, weather_current = weather.main, weather_description = weather.description, wind_speed = wind.speed, wind_degree = wind.deg, wind_gust = wind.gust, temp_current = main.temp, temp_min = main.temp_min, temp_max = main.temp_max, pressure = main.pressure, humidity = main.humidity", "outputStream": "Custom-AMA_LOGS_CL"}]}}
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/6f1b91a9-5242-4017-bc01-e262558ddab7/resourceGroups/rg-komla/providers/Microsoft.Insights/dataCollectionRules/custom-logs-dcr-cli-1?api-version=2022-06-01 HTTP/1.1" 400 270
cli.azure.cli.core.sdk.policies: Response status: 400
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Length': '270'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-resource-requests': '149'
cli.azure.cli.core.sdk.policies: 'Request-Context': 'appId=cid-v1:2bbfbac8-e1b0-44af-b9c6-3a40669d37e3'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '4b7d0a00-1652-4a2c-b754-2c0b85bd8401'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'c03290e2-8d74-11ee-b3d5-c03c598a8bf7'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTUS2:20231127T223150Z:4b7d0a00-1652-4a2c-b754-2c0b85bd8401'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '4ef10a55-f1fc-4b51-8e92-a54cbd0ad7b9'
cli.azure.cli.core.sdk.policies: 'api-supported-versions': '2019-11-01-preview, 2021-04-01, 2021-09-01-preview, 2022-06-01, 2023-03-11'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: 083BBBDD5A604D8AB5398725471ABBFE Ref B: CO6AA3150220047 Ref C: 2023-11-27T22:31:49Z'
cli.azure.cli.core.sdk.policies: 'Date': 'Mon, 27 Nov 2023 22:31:51 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"InvalidPayload","message":"Data collection rule is invalid","details":[{"code":"InvalidInputSchema","message":"Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.","target":"properties.dataFlows[0]"}]}}
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "C:\Users\komlaafoutou.azure\cliextensions\monitor-control-service\azext_amcs\manual\custom.py", line 148, in data_collection_rules_create
return _data_collection_rules_create(client,
File "C:\Users\komlaafoutou.azure\cliextensions\monitor-control-service\azext_amcs\manual\custom.py", line 94, in _data_collection_rules_create
return client.create(resource_group_name=resource_group_name,
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer
File "C:\Users\komlaafoutou.azure\cliextensions\monitor-control-service\azext_amcs\vendored_sdks\amcs\operations_data_collection_rules_operations.py", line 647, in create
raise HttpResponseError(response=response, model=error, error_format=ARMErrorFormat)
azure.core.exceptions.HttpResponseError: (InvalidPayload) Data collection rule is invalid
Code: InvalidPayload
Message: Data collection rule is invalid
Exception Details: (InvalidInputSchema) Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.
Code: InvalidInputSchema
Message: Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.
Target: properties.dataFlows[0]

cli.azure.cli.core.azclierror: (InvalidPayload) Data collection rule is invalid
Code: InvalidPayload
Message: Data collection rule is invalid
Exception Details: (InvalidInputSchema) Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.
Code: InvalidInputSchema
Message: Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.
Target: properties.dataFlows[0]
az_command_data_logger: (InvalidPayload) Data collection rule is invalid
Code: InvalidPayload
Message: Data collection rule is invalid
Exception Details: (InvalidInputSchema) Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.
Code: InvalidInputSchema
Message: Schema for input stream 'Custom-stream-1' for destination 'myloganalyticsworkspace' is not available.
Target: properties.dataFlows[0]
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x03FEE418>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 7.156 seconds (init: 2.292, invoke: 4.864)
telemetry.save: Save telemetry record of length 3562 in cache
telemetry.check: Returns Positive.
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry_init_.pyc C:\Users\komlaafoutou.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The request payload is expected to contain everyhting in the json input file including streamDeclarations and logFiles

Environment Summary

azure monitor cli extension

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.