Can not create data-collection rule with a custom stream #7399
Assignees
Labels
Auto-Assign
Auto assign by bot
Azure CLI Team
The command of the issue is owned by Azure CLI team
customer-reported
Issues that are reported by GitHub users external to the Azure organization.
Monitor
needs-team-attention
This issue needs attention from Azure service team or SDK team
question
The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Milestone
Comments
Joris-van-der-Wel
added
the
bug
microsoft-github-policy-service
bot
added
question
|
Thank you for opening this issue, we will look into it. |
microsoft-github-policy-service
bot
added
the
Azure CLI Team
yonzhan
removed
the
bug
|
@Joris-van-der-Wel thanks Joris for this difference. Not sure how azure portal deal with this but azure cli generates code based on service api doc, which can be referred here: https://github.com/Azure/azure-rest-api-specs/blob/42869ad83518d47ab2a24c07d888cb29fdeb43fd/specification/monitor/resource-manager/Microsoft.Insights/stable/2022-06-01/dataCollectionRules_API.json#L1135-L1148 . If that's the common user case, we'll let service team know the gap and whether they need to remove this enum validations. |
AllyW
added
the
needs-team-attention
|
I think that jsonschema definition is indeed wrong. I think the proper schema would be something like: {
"streams": {
"description": "List of streams for this data flow.",
"type": "array",
"items": {
"pattern": "^(Microsoft-Event|Microsoft-InsightsMetrics|Microsoft-Perf|Microsoft-Syslog|Microsoft-WindowsEvent|Custom-[\\w-]+)$",
"type": "string"
}
}
}Also, it seems to me that the actual API server uses a different schema to validate with? Would it not be possible to copy that schema? For example if I pass a wrong value to the "streams" property, the server will respond with: {
"error": {
"code": "InvalidPayload",
"message": "Data collection rule is invalid",
"details": [
{
"code": "InvalidStream",
"message": "'Streams' stream 'Foo' must be a custom stream or one of the allowed streams.",
"target": "Properties.DataFlows[0].Streams[0]"
}
]
}
}Similarly, if I use an invalid property name for the stream declaration, I get this error: {
"error": {
"code": "InvalidPayload",
"message": "Data collection rule is invalid",
"details": [
{
"code": "InvalidStreamDeclaration",
"message": "'' stream 'Foo' must be a custom stream.",
"target": "Properties.StreamDeclarations[0].Key"
},
{
"code": "InvalidStreamDeclaration",
"message": "'' is not a valid custom stream name. It should start with 'Custom-' followed by a letter and then any combination of alphanumeric characters and '-', '_'",
"target": "Properties.StreamDeclarations[0].Key"
}
]
}
}Custom streams are supported by Azure in all of the other ways that I have tried: Azure Portal
This will look something like this: {
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
"location": "westeurope",
"type": "Microsoft.Insights/dataCollectionRules",
"name": "example-dcr",
"properties": {
"dataSources": {},
"destinations": {
"logAnalytics": [{
"workspaceResourceId": "/subscriptions/XXX/resourcegroups/joris/providers/microsoft.operationalinsights/workspaces/XXX",
"workspaceId": "XXX",
"name": "123456789abcdef123456789abcdef12"
}]
},
"dataFlows": [{
"streams": ["Custom-Example_CL"],
"transformKql": "source",
"destinations": ["123456789abcdef123456789abcdef12"],
"outputStream": "Custom-Example_CL"
}],
"streamDeclarations": {
"Custom-Example_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "hello",
"type": "string"
}
]
}
},
"dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX"
}
}Azure CLI (list)Next, try to use: [
{
"dataCollectionEndpointId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionEndpoints/XXX",
"dataFlows": [{
"destinations": ["123456789abcdef123456789abcdef12"],
"outputStream": "Custom-Example_CL",
"streams": ["Custom-Example_CL"],
"transformKql": "source"
}],
"dataSources": {},
"destinations": {
"logAnalytics": [{
"name": "123456789abcdef123456789abcdef12",
"workspaceId": "XXX",
"workspaceResourceId": "/subscriptions/XXX/resourcegroups/XXX/providers/microsoft.operationalinsights/workspaces/XXX"
}]
},
"etag": "\"XXX\"",
"id": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Insights/dataCollectionRules/example-dcr",
"immutableId": "dcr-3456789abcdef123456789abcdef1245",
"location": "westeurope",
"name": "example-dcr",
"provisioningState": "Succeeded",
"resourceGroup": "XXX",
"streamDeclarations": {
"Custom-Example_CL": {
"columns": [
{
"name": "TimeGenerated",
"type": "datetime"
},
{
"name": "hello",
"type": "string"
}
]
}
},
"systemData": {
"createdAt": "2024-03-28T11:00:00Z",
"createdBy": "jvanderwel@XXX",
"createdByType": "User",
"lastModifiedAt": "2024-03-28T11:00:00Z",
"lastModifiedBy": "jvanderwel@XXX",
"lastModifiedByType": "User"
},
"type": "Microsoft.Insights/dataCollectionRules"
}
]Azure Resource ManagerI can deploy a bicep template that uses a custom stream without issue: resource workspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
name: 'myworkspace'
location: 'westeurope'
}
resource dce 'Microsoft.Insights/dataCollectionEndpoints@2022-06-01' = {
name: 'mydce'
location: 'westeurope'
}
resource table 'Microsoft.OperationalInsights/workspaces/tables@2022-10-01' = {
parent: workspace
name: 'Example_CL'
properties: {
schema: {
name: 'Example_CL'
columns: [
{name: 'TimeGenerated', type: 'datetime'}
{name: 'hello', type: 'string'}
]
}
}
}
resource dcr 'Microsoft.Insights/dataCollectionRules@2022-06-01' = {
name: 'mydcr'
location: 'westeurope'
properties: {
dataCollectionEndpointId: dce.id
dataFlows: [
{
destinations: ['log-analytics-dest']
outputStream: 'Custom-Example_CL'
streams: ['Custom-Example_CL']
transformKql: 'source'
}
]
destinations: {
logAnalytics: [
{
name: 'log-analytics-dest'
workspaceResourceId: workspace.id
}
]
}
streamDeclarations: {
'Custom-Example_CL': {
columns: [
{name: 'TimeGenerated', type: 'datetime'}
{name: 'hello', type: 'string'}
]
}
}
}
} |
4 tasks
|
@Joris-van-der-Wel we will add support for enumeration arguments which skips validation recently. Please follow the linked pr for future development and releases. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I am attempting to create a data collection rule in azure monitor, that is intended to be used with a custom application that calls the Logs Ingestion API. I have created a logs-analytics workspace containing a custom table. My data collection rule should be configured without any data sources, instead a custom stream for my table is declared.
This all works fine when I configure this via the Azure Portal GUI. However the CLI refuses to create the data-collection rule because the validation for the
--data-flowsoption is too strict. Specifically, the validation for the"streams"property: https://github.com/Azure/azure-cli-extensions/blob/e2dad8c97f70c6b002749a7864f9dda2485498dd/src/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py#L157C9-L159C10If I comment out line 158 (containing
enum=...):nano ~/.azure/cliextensions/monitor-control-service/azext_amcs/aaz/latest/monitor/data_collection/rule/_create.py.Then my rule is created without issue.
Related command
Errors
unrecognized value 'Custom-MyTable_CL' from choices '['Microsoft-Event', 'Microsoft-InsightsMetrics', 'Microsoft-Perf', 'Microsoft-Syslog', 'Microsoft-WindowsEvent']'
Issue script & Debug output
Expected behavior
The rule should be created without error
Environment Summary
Additional context
No response
The text was updated successfully, but these errors were encountered: