Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

500 Internal server error when creating grafana dashboard through az grafana dashboard create #6965

Closed
veerun14 opened this issue Nov 8, 2023 · 6 comments
Closed

500 Internal server error when creating grafana dashboard through az grafana dashboard create #6965

veerun14 opened this issue Nov 8, 2023 · 6 comments
Labels
Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization. extension/grafana az grafana question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.

Comments

@veerun14
Copy link

veerun14 commented Nov 8, 2023

Describe the bug

The command I ran:

az grafana dashboard create --resource-group $CLUSTER_RESOURCEGROUP --name $AZURE_MANAGED_GRAFANA --definition ../monitoring/grafana/windows-server-containers-reliability.json

The output I got:

cli.azext_amg.custom: b'<body style="background-color: rgb(38, 52, 72); ">\n <br />\n <h1 style="font-family: verdana; text-align: center; color: white; align-content:baseline">\n\n <svg style="vertical-align: text-bottom; margin-right:10px" width="40" height="40" viewBox="0 0 40 40" fill="none" xmlns="http://www.w3.org/2000/svg">\n <path d="M12.3913 4.13597C12.6205 3.45707 13.2571 3 13.9736 3H24.6745L13.5831 35.8633C13.354 36.5422 12.7173 36.9992 12.0008 36.9992H3.62616C2.48366 36.9992 1.6785 35.8777 2.04385 34.7952L12.3913 4.13597Z" fill="url(#paint0_linear)" />\n <path d="M29.4453 25.0039H12.619C11.9113 25.0039 11.5701 25.8713 12.0882 26.3534L23.0468 36.5521C23.3559 36.8397 23.7624 36.9996 24.1845 36.9996H33.5992L29.4453 25.0039Z" fill="#0078D4" />\n <path d="M13.9737 3C13.2571 3 12.6205 3.45707 12.3913 4.13597L2.04386 34.7952C1.67851 35.8777 2.48367 36.9992 3.62617 36.9992H12.0008C12.7173 36.9992 13.354 36.5422 13.5831 35.8633L15.6682 29.6852L23.0469 36.5521C23.3559 36.8397 23.7624 36.9996 24.1846 36.9996H33.5993L29.4453 25.0039H17.2481L24.6745 3H13.9737Z" fill="url(#paint1_linear)" />\n <path d="M27.6071 4.13743C27.3779 3.45854 26.7413 3.00146 26.0248 3.00146H14.049H14.1035C14.8329 3.00146 15.481 3.46674 15.7142 4.15781L26.0415 34.7571C26.4134 35.8591 25.5938 37.0007 24.4308 37.0007H24.2487H36.3722C37.5148 37.0007 38.3199 35.8792 37.9546 34.7967L27.6071 4.13743Z" fill="url(#paint2_linear)" />\n <defs>\n <linearGradient id="paint0_linear" x1="13.9224" y1="4.17684" x2="2.83367" y2="36.9356" gradientUnits="userSpaceOnUse">\n <stop stop-color="#114A8B" />\n <stop offset="1" stop-color="#0669BC" />\n </linearGradient>\n <linearGradient id="paint1_linear" x1="22.7802" y1="25.1033" x2="20.2172" y2="25.9703" gradientUnits="userSpaceOnUse">\n <stop stop-opacity="0.3" />\n <stop offset="0.0711768" stop-opacity="0.2" />\n <stop offset="0.321031" stop-opacity="0.1" />\n <stop offset="0.623053" stop-opacity="0.05" />\n <stop offset="1" stop-opacity="0" />\n </linearGradient>\n <linearGradient id="paint2_linear" x1="20.9518" y1="4.17835" x2="33.1237" y2="36.6074" gradientUnits="userSpaceOnUse">\n <stop stop-color="#3CCBF4" />\n <stop offset="1" stop-color="#2892DF" />\n </linearGradient>\n </defs>\n</svg>\n\n Internal Error\n </h1>\n\n <br />\n\n <p style="font-family: verdana; font-size: 120%; text-align: center; color: white;">\n\n The request has failed because of an internal error. Trace ID: 00-23b0d2453be1ab707039dc436f2ae336-ac2bb27bb2d65a44-00\n\n <br />\n\n <br />\n\n <a style="font-family: verdana; text-align: center; color: white;" href="https://aka.ms/managed-grafana-docs">\n Please check the documentation for further details.\n </a>\n </p>\n</body>'

The output I expect:
Successfully create a grafana dashboard

Related command

az grafana dashboard create --resource-group $CLUSTER_RESOURCEGROUP --name $AZURE_MANAGED_GRAFANA --definition ../monitoring/grafana/windows-server-containers-reliability.json --debug

Errors

`cli.azext_amg.custom: b'\n
\n

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n Internal Error\n

\n\n
\n\n

\n\n The request has failed because of an internal error. Trace ID: 00-23b0d2453be1ab707039dc436f2ae336-ac2bb27bb2d65a44-00\n\n
\n\n
\n\n \n Please check the documentation for further details.\n \n

\n'

Issue script & Debug output

`
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=DashboardManagementClient
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\Users\\.azure\msal_token_cache.bin', encrypt=True
cli.azure.cli.core.auth.binary_cache: load: C:\Users*****************************.azure\msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/*****************************/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/*****************************/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/*****************************/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 6e1e2bae-b753-4904-9902-38dfe94f05e8
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/*****************************/resourceGroups/*****************************/providers/Microsoft.Dashboard/grafana/**********?api-version=2022-10-01-preview'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '74d704a7-7e5d-11ee-b05b-002248b71f91'
cli.azure.cli.core.sdk.policies: 'CommandName': 'grafana dashboard create'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--resource-group --name --definition --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.49.0 azsdk-python-mgmt-dashboard/1.0.0b1 Python/3.10.10 (Windows-10-10.0.22631-SP0)'
cli.azure.cli.core.sdk.policies: 'Authorization': ''
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups//providers/Microsoft.Dashboard/grafana/**********?api-version=2022-10-01-preview HTTP/1.1" 200 None
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Transfer-Encoding': 'chunked'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Content-Encoding': 'gzip'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'ETag': '"dd0128b7-0000-0100-0000-654ad5450000"'
cli.azure.cli.core.sdk.policies: 'Vary': 'Accept-Encoding'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies: 'x-ms-providerhub-traffic': 'True'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '524394fb-19f6-4e91-8b35-2fec3a3cf512'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '15d99788-d576-4708-9be2-c54b6f70b23b'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTUS2:20231108T173716Z:15d99788-d576-4708-9be2-c54b6f70b23b'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Date': 'Wed, 08 Nov 2023 17:37:15 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"id":"/subscriptions//resourceGroups//providers/Microsoft.Dashboard/grafana/","name":"","type":"microsoft.dashboard/grafana","sku":{"name":"Standard"},"location":"EastUS2","tags":{},"systemData":{"createdBy":"@microsoft.com","createdByType":"User","createdAt":"2023-11-08T00:18:33.2749345Z","lastModifiedBy":"@microsoft.com","lastModifiedByType":"User","lastModifiedAt":"2023-11-08T00:24:12.9396562Z"},"identity":{"principalId":"324067cc-3d65-4cbf-b2bd-39d3a451768b","tenantId":"","type":"SystemAssigned"},"properties":{"provisioningState":"Succeeded","grafanaVersion":"9.5.13","endpoint":"https://**********-cmdyfsggfkf8hsf2.eus2.grafana.azure.com","zoneRedundancy":"Disabled","publicNetworkAccess":"Enabled","autoGeneratedDomainNameLabelScope":"TenantReuse","apiKey":"Disabled","deterministicOutboundIP":"Disabled","grafanaIntegrations":{"azureMonitorWorkspaceIntegrations":[{"azureMonitorWorkspaceResourceId":"/subscriptions//resourcegroups//providers/microsoft.monitor/accounts/cplatdi-large-************************-works"}]},"grafanaConfigurations":{"smtp":{"enabled":false}},"grafanaMajorVersion":"9"}}
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/*****************************/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/*****************************/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/*****************************/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/*****************************/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('ce34e7e5-485f-4d76-964f-b3d2b16d1e4f/.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: e301501f-7fe3-4d05-a303-8a667ab50765
urllib3.connectionpool: Starting new HTTPS connection (1): -cmdyfsggfkf8hsf2.eus2.grafana.azure.com:443
urllib3.connectionpool: https://-cmdyfsggfkf8hsf2.eus2.grafana.azure.com:443 "POST /api/dashboards/db HTTP/1.1" 500 None
cli.azext_amg.custom: b'\n
\n

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n Internal Error\n

\n\n
\n\n

\n\n The request has failed because of an internal error. Trace ID: 00-23b0d2453be1ab707039dc436f2ae336-ac2bb27bb2d65a44-00\n\n
\n\n
\n\n \n Please check the documentation for further details.\n \n

\n'
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "C:\Users*.azure\cliextensions\amg\azext_amg\custom.py", line 414, in create_dashboard
return _create_dashboard(cmd, grafana_name, definition=definition, title=title, folder_id=folder_id,
File "C:\Users*.azure\cliextensions\amg\azext_amg\custom.py", line 441, in _create_dashboard
response = _send_request(cmd, resource_group_name, grafana_name, "post", "/api/dashboards/db",
File "C:\Users*******************.azure\cliextensions\amg\azext_amg\custom.py", line 987, in _send_request
response.raise_for_status()
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\requests/models.py", line 953, in raise_for_status
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://-cmdyfsggfkf8hsf2.eus2.grafana.azure.com/api/dashboards/db

cli.azure.cli.core.azclierror: 500 Server Error: Internal Server Error for url: https://-cmdyfsggfkf8hsf2.eus2.grafana.azure.com/api/dashboards/db
az_command_data_logger: 500 Server Error: Internal Server Error for url: https://-cmdyfsggfkf8hsf2.eus2.grafana.azure.com/api/dashboards/db
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x0499C5C8>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 2.300 seconds (init: 0.589, invoke: 1.710)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3574 in cache
telemetry.check: Negative: The C:\Users*******************.azure\telemetry.txt was modified at 2023-11-08 09:27:17.763482, which in less than 600.000000 s
veeru@CPC-venuv-X3439:~/*****************************/CPlatStudio/azure/cluster$ az grafana dashboard create
--resource-group $CLUSTER_RESOURCEGROUP
--name $AZURE_MANAGED_GRAFANA
--definition ../monitoring/grafana/aks-app-deployment.json
b'\n
\n

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n\n Internal Error\n

\n\n
\n\n

\n\n The request has failed because of an internal error. Trace ID: 00-fa186f6284e9c2335d25bbc50751d30e-c290af2d22057dbf-00\n\n
\n\n
\n\n \n Please check the documentation for further details.\n \n

\n'
500 Server Error: Internal Server Error for url: https://-cmdyfsggfkf8hsf2.eus2.grafana.azure.com/api/dashboards/db
`

Expected behavior

successfully create a grafana dashboard.

Environment Summary

azure-cli 2.49.0 *

core 2.49.0 *
telemetry 1.0.8 *

Extensions:
aks-preview 0.5.157
amg 1.2.7
azure-devops 0.26.0

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\venuvvul.azure\cliextensions'

Python (Windows) 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:05:00) [MSC v.1929 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

You have 3 update(s) available. Consider updating your CLI installation with 'az upgrade'

Additional context

There seems to be no indication of the problem even in the debug output when this error happens. Some sort of location or indication of what might have gone wrong would be useful
@veerun14 veerun14 added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Nov 8, 2023
@microsoft-github-policy-service microsoft-github-policy-service bot added question The issue doesn't require a change to the product in order to be resolved. Most issues start as that customer-reported Issues that are reported by GitHub users external to the Azure organization. labels Nov 8, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Nov 8, 2023

Thank you for opening this issue, we will look into it.

@microsoft-github-policy-service microsoft-github-policy-service bot added Auto-Assign Auto assign by bot extension/grafana az grafana Service Attention This issue is responsible by Azure service team. labels Nov 8, 2023
@veerun14
Copy link
Author

It looks like even az grafana data-source create command fails with similar error. I am not sure if it is my setup or if there is an issue here. @yonzhan, were you able to repro it?

az grafana data-source create -n cplat-aks-grafana --definition '{ "access": "proxy", "jsonData": { "azureAuthType": "msi", "subscriptionId": "*****************", "tenantId": "**************", "logAnalyticsWorkspaceId": "/*******************" }, "name": "Azure Monitor-la-cplat-aks", "type": "grafana-azure-monitor-datasource" }'

@veerun14
Copy link
Author

Note that I have Grafana Admin role on the Azure Managed Grafana instance and I am owner of the sub in which Azure Managed Grafana was created.

@veerun14
Copy link
Author

any update on this one?

@ABZhang0
Copy link
Contributor

Hi @veerun14, sorry for the delayed response. I just checked our logs, and it appears to be a known issue with our token access. There has been a fix out that should be in this earlier this month, so can you try the same 'create dashboard' operation again and see if it works?

@yonzhan yonzhan removed the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Dec 21, 2023
@veerun14
Copy link
Author

veerun14 commented Mar 1, 2024

Yup. This can be closed now. It has been working without issues after the fix for a while now

@veerun14 veerun14 closed this as completed Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Auto-Assign Auto assign by bot customer-reported Issues that are reported by GitHub users external to the Azure organization. extension/grafana az grafana question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@veerun14 @ABZhang0 @yonzhan