Partially Failing Instrumentation for Python K8s Service with Dynatrace

[azacharov]

Hi,

I’m experiencing issues with Dynatrace instrumentation in my Python-based Kubernetes service

I have implemented tracing using the autodynatrace SDK in my FastAPI application, which runs on Uvicorn.

However, I see many “no response” errors for my Kubernetes service endpoints in Dynatrace.
As verified through application logs, these errors do not correspond to actual failures.

Additionally, I’m receiving the following diagnostic error in Dynatrace:

“Some data could not be collected or transmitted. This is most likely due to a resource congestion on network, host, or process level in your monitored environment

(Diagnostic codes: C1, A5)"

Here are some details of my Kubernetes Python service:

Framework: FastAPI.
Server: Uvicorn.
Autodynatrace Setup:
Installed autodynatrace in the Dockerfile.
Set the AUTOWRAPT_BOOTSTRAP=autodynatrace environment variable

1. What could be causing these “no response” errors, given that the endpoints function correctly?

2.Could the diagnostic error codes (C1, A5) be related to this behavior?

3. Are there additional configurations or adjustments recommended for Python FastAPI services in Kubernetes to optimize tracing and prevent these issues?

Here are OneAgent analyses:

[C/Python SDK] end() order wrong or unsupported async/gevent usage (736)

Nov 17 2024, 16:10:43 - Yesterday, 15:40:38
Classification: Warning
The end() calls on the SDK tracers are in a wrong order. Can result from usage of async/await or greenlets (gevent, green threads) e.g. with Python gunicorn. Please see https://github.com/Dynatrace/OneAgent-SDK-for-Python/blob/master/README.md#tracers Can also be simply forgetting to call start (tracers are not started at creation) or end or skipping them (e.g. by exception)

Recommendation: Check you are calling end in the reverse order of start and in the correct (OS) thread. In Python, prefer with blocks to explicit start/end calls.

Especially if using async/await, green threads or other threading abstractions, check https://github.com/Dynatrace/OneAgent-SDK-for-Python/blob/master/README.md#tracers.

If using a server like gunicorn, you might be able to configure it to not use green threads (normal threads are fine) but be aware that this will have a performance impact.