metricbeats: update monitor mappings to include new TBS metrics #138131

[rubvs]

Proposed commit message

Update metricbeat beat module mapping to contain latest apm-server monitoring metrics fields to improve TBS.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Author's Checklist

• Verify mappings are present in the metricbeat-* datastream.

How to test this PR locally

Step 1: Ensure Elasticsearch & Kibana is running

> cd apm-server
> docker-compose up elasticsearch kibana

Step 2: Create APM Server config

Create an apm-server.yml config file:

apm-server:
  host: "127.0.0.1:8200"

output.elasticsearch:
  enabled: true
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

http:
  enabled: true
  host: localhost
  port: 5066

Step 3: Start APM Server

Run APM Server binary directly:

> cd apm-server
> ./apm-server -e -v -c apm-server.yml

Step 4: Configure Metricbeat WITHOUT xpack

Create metricbeat.yml config file:

metricbeat.modules:
- module: beat
  metricsets:
    - state
    - stats
  period: 10s
  hosts: ["http://localhost:5066"]

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "changeme"

Step 5: Start Metricbeat

Run Metricbeat binary directly:

> cd beats
> make python-env
> source build/python-env/bin/activate

> cd beats/metricbeat
> make mage
> mage build

> sudo chown root modules.d/system.yml
> sudo chown root metricbeat.yml
> sudo ./metricbeat -c metricbeat.yml -e

Step 6: Verify Datastream

Verify fields in datastream metricbeat-*

GET metricbeat-*/_search
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {"term": {"metricset.name": "stats"}},
        {"term": {"beat.type": "apm-server"}}
      ]
    }
  },
  "_source": ["beat.stats.apm_server.sampling.tail.storage"],
  "sort": [{"@timestamp": {"order": "desc"}}]
}

"hits": [
  {
    "_index": ".ds-metricbeat-9.3.0-2025.11.19-000001",
    "_id": "o1aSmZoBfa9YIotqCAfv",
    "_score": null,
    "_source": {
      "beat": {
        "stats": {
          "apm_server": {
            "sampling": {
              "tail": {
                "storage": {
                  "disk_total": 994662584320,
                  "disk_usage_threshold": 80,
                  "disk_used": 336527384576,
                  "lsm_size": 10553,
                  "storage_limit": 0,
                  "value_log_size": 0
                }
              }
            }
          }
        }
      }
    },
    "sort": [
      1763513067282
    ]
  }
]

Verify mappings for datastream metricbeat-*

GET metricbeat-*/_mapping?filter_path=**.apm_server.**.storage

{
  ".ds-metricbeat-9.3.0-2025.11.19-000001": {
    "mappings": {
      "properties": {
        "beat": {
          "properties": {
            "stats": {
              "properties": {
                "apm_server": {
                  "properties": {
                    "sampling": {
                      "properties": {
                        "tail": {
                          "properties": {
                            "storage": {
                              "properties": {
                                "disk_total": {
                                  "type": "float"
                                },
                                "disk_usage_threshold": {
                                  "type": "long"
                                },
                                "disk_used": {
                                  "type": "float"
                                },
                                "lsm_size": {
                                  "type": "long"
                                },
                                "storage_limit": {
                                  "type": "long"
                                },
                                "value_log_size": {
                                  "type": "long"
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Related issues

• Relates monitoring: expose storage limit and disk metrics for TBS monitoring apm-server#15533

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @rubvs? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/metricbeat/exported-fields-beat.md