ECS container ID not being parsed/sent #823
Comments
|
Tangentially related, docker uses cgroup v2 (since docker 20.10) which does not expose the container ID in Instead, it looks like the container ID can be resolved from |
|
Looks like this has come up in various elastic apm repos (elastic/apm-agent-python#1691, elastic/apm#523). elastic/apm#523 (comment) is exactly the issue that we're running into. |
|
Proposed PR #966 |
|
Implemented by #966 |
|
@Nacoma Hi. We implemented capturing container ID in PRs #966 and #975 and it's already merged to main branch. Please try it out and let us know if it works on a AWS ECS container. Also if you could provide examples of |
That's wonderful! We have tickets in place to move to the new version. I appreciate the work you all are doing on this.
Absolutely! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Infrastructure correlation isn't working because the
container.idfield is not present.To Reproduce
Expected behavior
This is the ingested message received from the application/APM:
{ "@timestamp": [ "2022-11-22T22:33:34.985Z" ], "agent.name": [ "php" ], "agent.version": [ "1.6.1" ], "data_stream.dataset": [ "apm" ], "data_stream.namespace": [ "develop" ], "data_stream.type": [ "traces" ], "ecs.version": [ "1.12.0" ], "event.agent_id_status": [ "missing" ], "event.ingested": [ "2022-11-22T22:33:36.000Z" ], "event.outcome": [ "success" ], "host.hostname": [ "fcf701cd93a8" ], "host.ip": [ "172.17.0.2" ], "host.name": [ "fcf701cd93a8" ], "http.request.method": [ "POST" ], "observer.ephemeral_id": [ "6ddef0a9-c6d4-4398-a08c-1c525d41e645" ], "observer.hostname": [ "ip-blah.ec2.internal" ], "observer.id": [ "77dcda3d-0f0e-4721-88fb-bad936adaaa3" ], "observer.type": [ "apm-server" ], "observer.version": [ "8.5.1" ], "process.pid": [ 53 ], "processor.event": [ "transaction" ], "processor.name": [ "transaction" ], "service.environment": [ "develop" ], "service.language.name": [ "PHP" ], "service.language.version": [ "8.1.12" ], "service.name": [ "Api" ], "service.node.name": [ "fcf701cd93a8" ], "service.runtime.name": [ "PHP" ], "service.runtime.version": [ "8.1.12" ], "timestamp.us": [ 1669156414985718 ], "trace.id": [ "57cfce17b3fc98f6f7405f60bd2ae45d" ], "transaction.duration.us": [ 111750 ], "transaction.id": [ "f473bb3a724dc27c" ], "transaction.name": [ "POST /" ], "transaction.name.text": [ "POST /" ], "transaction.result": [ "HTTP 4xx" ], "transaction.sampled": [ true ], "transaction.span_count.started": [ 0 ], "transaction.type": [ "request" ], "url.domain": [ ], "url.full": [ ], "url.full.text": [ ], "url.original": [ ], "url.original.text": [ ], "url.path": [ "/" ], "url.scheme": [ "http" ], "_id": "Kwt5oYQBq_2iaxTGqjNm", "_index": ".ds-traces-apm-develop-2022.11.22-000001", "_score": null }Metricbeat (elastic-agent), installed on the host, is correctly identifying the containers:
{ "_index": ".ds-metrics-docker.container-develop-2022.11.22-000001", "_id": "_wuCoYQBq_2iaxTGSz52", "_version": 1, "_score": null, "fields": { "elastic_agent.version": [ "8.5.1" ], "host.os.name.text": [ "Amazon Linux" ], "host.hostname": [ "ip-blah.ec2.internal" ], "host.mac": [ "02-42-C2-AB-8F-BF", "02-6B-A7-10-33-CF", "42-7C-4E-97-2D-45", "CA-C4-80-19-42-B1" ], "docker.container.size.rw": [ 0 ], "container.id": [ "fcf701cd93a80cd5ae1ead16a80b871b0daea72d59868661b8bbef39d79dc1e6" ], "cloud.availability_zone": [ "us-east-1a" ], "service.type": [ "docker" ], "container.name": [ ], "docker.container.labels.com_amazonaws_ecs_container-name": [ "app" ], "container.image.name": [ ], "host.os.version": [ "2" ], "host.os.name": [ "Amazon Linux" ], "agent.name": [ "ip-blah.ec2.internal" ], "host.name": [ "ip-blah.ec2.internal" ], "event.agent_id_status": [ "verified" ], "docker.container.ip_addresses": [ "172.17.0.2" ], "cloud.region": [ "us-east-1" ], "host.os.type": [ "linux" ], "data_stream.type": [ "metrics" ], "docker.container.command": [ "docker-php-entrypoint entrypoint" ], "host.architecture": [ "x86_64" ], "cloud.machine.type": [ "t3a.2xlarge" ], "cloud.provider": [ "aws" ], "container.runtime": [ "docker" ], "cloud.service.name": [ "EC2" ], "agent.id": [ "0c019f2c-a7a8-4d00-ab78-0d3bd41e4b93" ], "host.containerized": [ false ], "ecs.version": [ "8.0.0" ], "service.address": [ "unix:///var/run/docker.sock" ], "docker.container.labels.com_amazonaws_ecs_task-arn": [ ], "agent.version": [ "8.5.1" ], "host.os.family": [ "redhat" ], "docker.container.status": [ "Up 34 minutes" ], "docker.container.labels.com_amazonaws_ecs_task-definition-version": [ "410" ], "docker.container.created": [ "2022-11-22T22:08:43.000Z" ], "docker.container.labels.com_amazonaws_ecs_task-definition-family": [ ], "cloud.instance.id": [ "i-0ca7c5a56b11067bf" ], "host.ip": [ "10.0.1.253", "fe80::6b:a7ff:fe10:33cf", "172.17.0.1", "fe80::42:c2ff:feab:8fbf", "fe80::c8c4:80ff:fe19:42b1", "fe80::407c:4eff:fe97:2d45" ], "agent.type": [ "metricbeat" ], "docker.container.labels.role": [ "php" ], "event.module": [ "docker" ], "host.os.kernel": [ "4.14.275-207.503.amzn2.x86_64" ], "elastic_agent.snapshot": [ false ], "host.id": [ "9055fc78a53c4749b759e32a794d060f" ], "elastic_agent.id": [ "0c019f2c-a7a8-4d00-ab78-0d3bd41e4b93" ], "data_stream.namespace": [ "develop" ], "metricset.period": [ 10000 ], "host.os.codename": [ "Karoo" ], "event.duration": [ 2777816 ], "metricset.name": [ "container" ], "cloud.image.id": [ "ami-03297dd2c45657fd8" ], "event.ingested": [ "2022-11-22T22:43:01.000Z" ], "docker.container.size.root_fs": [ 0 ], "@timestamp": [ "2022-11-22T22:43:00.619Z" ], "host.os.platform": [ "amzn" ], "cloud.account.id": [ "121952031047" ], "data_stream.dataset": [ "docker.container" ], "agent.ephemeral_id": [ "b7ab80e1-fbe5-43b6-9bcc-ed55093f9d4b" ], "event.dataset": [ "docker.container" ], "docker.container.labels.com_amazonaws_ecs_cluster": [ "awseb-development-api-uimmpxrb6e" ] }, "highlight": { "container.id": [ "@kibana-highlighted-field@fcf701cd93a80cd5ae1ead16a80b871b0daea72d59868661b8bbef39d79dc1e6@/kibana-highlighted-field@" ] }, "sort": [ 1669156980619 ] }May be a different container ID now, but
/proc/self/cgroupcontains:The text was updated successfully, but these errors were encountered: