Feature Request: Support multiple derived metrics in derive_metrics function

[ConvolutedDog]

Problem

Currently, the derive_metrics function only supports returning a single derived metric value. This limitation prevents users from computing multiple custom metrics in a single profiling run.

# Only one derived metric can be returned
def derive_metrics(a, b, c):
    return a + b  # single value only

Desired Behavior

Allow derive_metrics to return multiple values, either as:

• A tuple/list of values
• A dictionary of named metrics

# Option 1: Tuple/list return
def derive_metrics(a, b, c):
    return a + b, a * b, a / b  # multiple metrics

# Option 2: Dict return (named metrics)
def derive_metrics(a, b, c):
    return {"sum": a + b, "product": a * b, "ratio": a / b}

Use Cases

• Compute multiple related metrics (e.g., arithmetic intensity + bandwidth utilization)
• Return both absolute and normalized values

[Alok-Joshi]

# Option 2: Dict return (named metrics)
def derive_metrics(a, b, c):
    return {"sum": a + b, "product": a * b, "ratio": a / b}

Option 2 makes sense to us; we also were brainstorming on similar lines.

[Alok-Joshi]

Another thing we briefly discussed was how we the derived metrics can be included in the dataframe. The current dataframe for a derived metric looks like -

This works if we are deriving only a single metric. To support multiple derived metrics, we can get rid of the Transformed column altogether and have a row for each derived metric -

This approach works well as it allows us to give a different name for each derived metric. This can also work well with #9 (comment) . The original metric is also preserved.

[ConvolutedDog]

Since only a single derived metric is currently supported, I haven't exploded the dataframe in PR #9 when there is a derived metric (df["Transformed"] is not all bool(False)). So the "Metric" column still maintains the tuple form.

  Annotation     n       AvgValue        StdDev  NumRuns                                             Metric        Transformed Configuration
0     matmul  1024   16213.333333  0.000000e+00       10  (smsp__sass_inst_executed_op_shared_ld.sum, sm...  compute_avg_insts        n=1024
1     matmul  2048   97357.575758  1.455192e-11       10  (smsp__sass_inst_executed_op_shared_ld.sum, sm...  compute_avg_insts        n=2048
2     matmul  4096  767751.757576  0.000000e+00       10  (smsp__sass_inst_executed_op_shared_ld.sum, sm...  compute_avg_insts        n=4096

If we support multiple derived metrics in the future, exploding the dataframe (as you suggested) would indeed be the right approach.

But how should we handle plotting in that scenario? Should we provide an API for plotting multiple metrics on the same chart?

[ConvolutedDog]

@Alok-Joshi Does this DataFrame format for example-09 look okay? I removed the 'Transform' column.

Should we add a parameter for users to choose between listing all metrics or just the transformed ones in the "Metric" column?

   Annotation                                     Metric     n     AvgValue  StdDev     MinValue     MaxValue  ...  ComputeClock MemoryClock   CI95_Lower   CI95_Upper  RelativeStdDevPct  StableMeasurement       Geomean
0      matmul  smsp__sass_inst_executed_op_shared_ld.sum  1024    2131968.0     0.0    2131968.0    2131968.0  ...       2520000    10501000    2131968.0    2131968.0                0.0               True  1.420171e+07
1      matmul  smsp__sass_inst_executed_op_shared_st.sum  1024      32768.0     0.0      32768.0      32768.0  ...       2520000    10501000      32768.0      32768.0                0.0               True  3.000800e+05
2      matmul                           launch__sm_count  1024        128.0     0.0        128.0        128.0  ...       2520000    10501000        128.0        128.0                0.0               True  1.280000e+02
3      matmul  smsp__sass_inst_executed_op_shared_ld.sum  2048   13131776.0     0.0   13131776.0   13131776.0  ...       2520000    10501000   13131776.0   13131776.0                0.0               True  1.420171e+07
4      matmul  smsp__sass_inst_executed_op_shared_st.sum  2048     524288.0     0.0     524288.0     524288.0  ...       2520000    10501000     524288.0     524288.0                0.0               True  3.000800e+05
5      matmul                           launch__sm_count  2048        128.0     0.0        128.0        128.0  ...       2520000    10501000        128.0        128.0                0.0               True  1.280000e+02
6      matmul  smsp__sass_inst_executed_op_shared_ld.sum  4096  102309888.0     0.0  102309888.0  102309888.0  ...       2520000    10501000  102309888.0  102309888.0                0.0               True  1.420171e+07
7      matmul  smsp__sass_inst_executed_op_shared_st.sum  4096    1572864.0     0.0    1572864.0    1572864.0  ...       2520000    10501000    1572864.0    1572864.0                0.0               True  3.000800e+05
8      matmul                           launch__sm_count  4096        128.0     0.0        128.0        128.0  ...       2520000    10501000        128.0        128.0                0.0               True  1.280000e+02
9      matmul                          compute_avg_insts  1024      16912.0     0.0      16912.0      16912.0  ...       2520000    10501000      16912.0      16912.0                0.0               True  1.135572e+05
10     matmul                          compute_avg_insts  2048     106688.0     0.0     106688.0     106688.0  ...       2520000    10501000     106688.0     106688.0                0.0               True  1.135572e+05
11     matmul                          compute_avg_insts  4096     811584.0     0.0     811584.0     811584.0  ...       2520000    10501000     811584.0     811584.0                0.0               True  1.135572e+05

[Alok-Joshi]

Does this DataFrame format for example-09 look okay? I removed the 'Transform' column.

The dataframe makes sense to me

Should we add a parameter for users to choose between listing all metrics or just the transformed ones in the "Metric" column?

I see value in retaining all the metrics in the dataframe - collected as well as derived. Only issue is this would mean whenever we have a derive metric, the dataframe would have atleast two rows, and hence nsight.analyze.plot will error out as it cannot plot multiple metrics.

We could have a default behaviour for nsight.analyze.plot where it simply plots the first metric in the dataframe. Adding @acollins3 @nvssatoor @bastianhagedorn to the discussion

[acollins3]

I would suggest we add a parameter nsight.analyze.plot that specifies which metric to plot, i.e. plot(metric="launch__sm.count"). This makes the choice explicit, and unsurprising, compared to basing the choice on some ordering criteria.

If metric=... is not specified, then by default it could plot the single metric found in the dataframe, or produce an error if there are multiple metrics present.

This parameter could be extended in future to accept a list or similar, if and when we add support for plotting multiple metrics.

[ConvolutedDog]

Thanks for the suggestions. I've been a bit busy the last few days, but I'll work on these changes once I'm free.

Regarding the points discussed:

• The DataFrame should keep all metrics (both collected and derived).
• Add a metric parameter to nsight.analyze.plot to let users specify which metric to plot. If not specified, it can either:
  • Plot the single metric if only one exists in the DataFrame, or
  • Raise an error if multiple metrics are present.

I'll implement these updates accordingly.

[ConvolutedDog]

@Alok-Joshi @acollins3 @nvssatoor @bastianhagedorn
I have submitted the PR #16, please take a look.

[Alok-Joshi]

PR #16 implements this feature. Closing this issue. Thanks @ConvolutedDog !