mz-debug: Fix HTTP dumping for multi-process clusters - #37816
Merged
SangJunBak merged 5 commits intoJul 27, 2026
Merged
Conversation
Contributor
Author
|
orchestratord test run https://buildkite.com/materialize/nightly/builds/17495 |
SangJunBak
marked this pull request as ready for review
July 23, 2026 02:07
def-
approved these changes
Jul 23, 2026
| agents: | ||
| queue: hetzner-aarch64-16cpu-32gb | ||
|
|
||
| - id: orchestratord-mz-debug-scaled-replica |
Contributor
There was a problem hiding this comment.
This is maybe a bit too specific to add a full job in CI for.
Contributor
Author
There was a problem hiding this comment.
@def- I do feel like this test is valuable and something we don't already have in any pipeline: an e2e test of cpu/memory profiling in a cloud environment with multiple pods. It does feel weird to lump this into the orchestratord tests however. Would it make sense to port this as a cloudtest in cloudtest/test_mz_debug_tool.py?
Contributor
Author
There was a problem hiding this comment.
Test ported over to cloudtest https://buildkite.com/materialize/nightly/builds/17509#019f8f9c-f82c-4bc0-bf85-4c70417f8823
SangJunBak
marked this pull request as draft
July 23, 2026 13:01
SangJunBak
force-pushed
the
jun/fix-http-multi-process-dumping
branch
from
July 23, 2026 13:18
910e34e to
f46e28b
Compare
Because multi-process replicas can have multiple pods per service, we account for that when dumping resources.
Creates a test that asserts each respective profile exists after an mz-debug run. Also tests for multi-process replicas
SangJunBak
force-pushed
the
jun/fix-http-multi-process-dumping
branch
from
July 23, 2026 15:30
f46e28b to
bc1e0c7
Compare
SangJunBak
marked this pull request as ready for review
July 23, 2026 16:16
SangJunBak
requested review from
alex-hunt-materialize
and removed request for
alex-hunt-materialize
July 23, 2026 16:16
alex-hunt-materialize
approved these changes
Jul 27, 2026
| labels={ | ||
| "app": "environmentd", | ||
| MZ_ORGANIZATION_NAME_LABEL: MZ_INSTANCE_NAME, | ||
| MZ_RESOURCE_ID_LABEL: "environmentd", |
Contributor
There was a problem hiding this comment.
The resource ID is a random string used for identifying which Materialize instance this object belongs to. It needs to match the resource ID used elsewhere.
This will not work if we ever instantiate multiple environments simultaneously.
| # `REPLICATION FACTOR 1` pins the cluster to exactly one replica, so the | ||
| # `scale=2` size yields exactly two clusterd pods behind one service. | ||
| mz.environmentd.sql( | ||
| f"CREATE CLUSTER {CLUSTER_NAME} SIZE 'scale={SCALE},workers=1', REPLICATION FACTOR 1" |
Contributor
There was a problem hiding this comment.
Can we also get a test with replication factor 2?
The `materialize.cloud/mz-resource-id` label ties a Kubernetes object to the Materialize instance it belongs to, so every object of an instance carries the same value and two instances never share one. Stamping the literal string `environmentd` on the environmentd Service met neither requirement. cloudtest now generates an organization name and a resource ID per MaterializeApplication, in the format the operator assigns, and stamps both on the environmentd Service and StatefulSet. mz-debug is handed the generated organization name rather than a shared constant.
mz-debug has to walk two dimensions to reach every clusterd pod: the processes of a replica, which share one service, and the replicas of a cluster, which each get their own service. The profiling test only covered the first, so parametrize it over a scale=2 replica and a replication factor 2 cluster. Each case now asserts that a profile arrived for every expected pod by name, rather than counting the profiles that carry a replica's id. Only profiles written by the run under test count, because mz-debug names its output directory after the current minute and runs seconds apart share one.
def-
added a commit
that referenced
this pull request
Jul 28, 2026
So we don't accidentally reuse an existing file from the same minute. Follow-up to #37816
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/mz-debug: Assert emulator run produces all default output files
mz-debug: Scrape http resources per process instead of per service
Because multi-process replicas can have multiple pods per service, we account for that when dumping resources.
Motivation
Bug found here #37283 (review)
Description
Rather than just port forward the first pod of a service, finds the related process pods too
Verification
Created an e2e orchestratord test to assert
Nightly: https://buildkite.com/materialize/nightly/builds/17563