Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
__pycache__/
*.pyc
41 changes: 41 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Tool-mlxreg

## Purpose
Crucible tool for monitoring Mellanox/NVIDIA ConnectX network adapter power consumption using Mellanox Firmware Tools (`mlxreg`) to query hardware voltage regulator and environmental power sensors.

## Languages
- Bash: start/stop and collection scripts (`mlxreg-start`, `mlxreg-stop`, `mlxreg-collect`)
- Python: post-processing and CDM metric emission (`mlxreg-post-process`)

## Key Files
| File | Purpose |
|------|---------|
| `mlxreg-start` | Validates parameters, resolves network interfaces to PCI addresses, and launches `mlxreg-collect` |
| `mlxreg-stop` | Sends SIGTERM to `mlxreg-collect` and compresses CSV files with xz |
| `mlxreg-collect` | Periodic hardware power sensor polling loop via `mlxreg` |
| `mlxreg-post-process` | Reads CSV logs and emits `power-watts` CDM metrics via `toolbox.cdm_metrics` |
| `rickshaw.json` | Rickshaw integration: collector scripts, blacklist/whitelist |
| `workshop.json` | Engine image build: installs MFT tools |
| `tool-metadata.json` | Machine-readable description and CDM-indexed status (consumed by `crucible tools list`) |
| `multiplex.json` | Parameter validation rules and `defaults` preset for multiplex (mirrors benchmark `multiplex.json`) |

## Configuration
- `--devices <list>` — Comma-separated list of PCI addresses (e.g. `0000:b5:00.0,0000:c3:00.0`) or network interface names (e.g. `ens7f0np0,ens8f0np0`)
- `--sensors <list>` — Comma-separated list of sensor indices to poll (default: `1,2,6,127`)
- `--interval <seconds>` — Polling interval in seconds (default: `2`)

## Architecture
- `mlxreg-start` — Validates device access (`mlxreg --device $device --reg_name MVCAP --get`), converts interface names to PCI addresses via sysfs (`/sys/class/net/$iface/device`), and launches `mlxreg-collect`
- `mlxreg-collect` — Queries each sensor on each device using `mlxreg --reg_name MVCR --indexes "sensor_index=$idx"`, parses hexadecimal power values into watts, and writes CSV records to `mlxreg-data/<device>.csv`
- `mlxreg-stop` — Sends SIGTERM to collector, waits for graceful exit, and compresses all CSV files to `.csv.xz`
- `mlxreg-post-process` — Reads `.csv.xz` files, constructs CDM metric samples with labels (`device`, `sensor_index`, `sensor_name`, `metric`), and logs `power-watts` metrics

## Testing
- Run post-processor locally: `cd <tool-data-dir> && TOOLBOX_HOME=/opt/crucible/subprojects/core/toolbox python3 /opt/crucible/subprojects/tools/mlxreg/mlxreg-post-process`
- Validate syntax: `python3 -c "import py_compile; py_compile.compile('mlxreg-post-process', doraise=True)"`
- Full integration: `crucible run <run-file.json>` with mlxreg configured

## Conventions
- Primary branch is `main`
- Standard Bash modelines and 4-space indentation
- Python code follows 4-space indentation with standard modelines
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
@AGENTS.md
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,19 +156,20 @@ Use the first PCI function (e.g., `0000:b5:00.0`) as the `--devices` parameter:
|-----------|----------|---------|-------------|
| `--devices` | **Yes** | - | Comma-separated device identifiers. Can be either:<br>• Network interface names: `ens7f0np0` or `ens7f0np0,ens8f0np0`<br>• PCI BDF addresses: `0000:b5:00.0` or `0000:b5:00.0,0000:c3:00.0`<br>The tool automatically converts interface names to PCI addresses |
| `--interval` | No | `2` | Collection interval in seconds |
| `--sensors` | No | `1,2,6` | Comma-separated MVCR sensor indices to collect |
| `--sensors` | No | `1,2,6,127` | Comma-separated MVCR sensor indices to collect |

### Sensor Indices

The tool queries the `MVCR` (Voltage/Current/Power) register from the BF-3 DPU. The BF-3 DPU exposes 21 power sensors (indices 1-21), but only a subset provides actual power consumption data:

#### Active Power Sensors (Default Collection: 1,2,6)
#### Active Power Sensors (Default Collection: 1,2,6,127)

| Sensor Index | Sensor Name | Description | Typical Value | Notes |
|--------------|-------------|-------------|---------------|-------|
| 1 | `Vr0Pwr` | Voltage Regulator 0 power | ~17W | Core power supply |
| 2 | `Vr1Pwr` | Voltage Regulator 1 power | ~28W | I/O power supply |
| 6 | `PwrEnv` | Environmental/Total power | ~124W | **Total device power** |
| 127 | `TOT PWR` | Total module power | ~124W | **Total board power** |

**Power Breakdown:**
- **Total BF-3 Power (PwrEnv):** ~124W
Expand All @@ -182,7 +183,7 @@ The tool queries the `MVCR` (Voltage/Current/Power) register from the BF-3 DPU.
- Network controller power
- Other internal components

For most use cases, collecting sensors **1, 2, and 6** provides both total power and component-level breakdown.
For most use cases, collecting sensors **1, 2, 6, and 127** provides both total power and component-level breakdown.

#### Additional Sensors (Status/Diagnostic)

Expand Down
25 changes: 25 additions & 0 deletions multiplex.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"presets": {
"defaults" : [
{ "arg": "sensors", "vals": ["1,2,6,127"] },
{ "arg": "interval", "vals": ["2"] }
]
},
"validations": {
"device_list" : {
"description" : "comma-separated list of PCI addresses (e.g. 0000:b5:00.0) or network interface names (e.g. ens7f0np0)",
"args" : [ "devices" ],
"vals": "^[A-Za-z0-9_.:@-]+(,[A-Za-z0-9_.:@-]+)*$"
},
"sensor_list" : {
"description" : "comma-separated list of sensor index integers (default: 1,2,6,127)",
"args" : [ "sensors" ],
"vals": "^[1-9][0-9]*(,[1-9][0-9]*)*$"
},
"positive_integer" : {
"description" : "a whole number greater than 0",
"args" : [ "interval" ],
"vals": "^[1-9][0-9]*$"
}
}
}
22 changes: 22 additions & 0 deletions tool-metadata.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"rickshaw-tool-metadata": {
"schema": {
"version": "2026.08.11"
}
},
"tool": "mlxreg",
"description": "Mellanox NIC power consumption monitoring via Mellanox Firmware Tools (mlxreg) accessing hardware power sensors (e.g. Vr0Pwr, Vr1Pwr, PwrEnv, TOT PWR).",
"cdm_indexed": true,
"cdm_sources": [
{
"source": "mlxreg",
"types": ["power-watts"]
}
],
"output_files": [
"mlxreg-data/<device>.csv.xz",
"mlxreg-pid.txt",
"mlxreg-start-stderrout.txt",
"mlxreg-stop-stderrout.txt"
]
}