Feature Request: Support accessing prior results when profiling multiple functions in a single script

[ConvolutedDog]

Problem

Nsight's profiling system currently profiles each @nsight.analyze.kernel decorated function in a separate script execution. While this allows individual functions to be profiled, it creates two major issues:

1. Results Isolation: Each profiling execution only returns results for the currently-profiled function. Other decorated functions in a single script return None during that execution.

2. Data Inaccessibility: When profiling multiple functions in a single script, results from earlier profiling executions are not accessible in later executions, making it impossible to access all profiling results within a single script run.

The root cause is that @nsight.analyze.kernel triggers a separate script execution for each decorated function it profiles.

Current Behavior

import torch
import nsight

sizes = [(2**i,) for i in range(11, 14)]

@nsight.analyze.kernel(configs=sizes, runs=10)
def kernel1(n: int) -> None:
    a = torch.randn(n, n, device="cuda")
    b = torch.randn(n, n, device="cuda")
    with nsight.annotate("matmul"):
        _ = a @ b

@nsight.analyze.kernel(configs=sizes, runs=10)
def kernel2(n: int) -> None:
    a = torch.randn(n, n, device="cuda")
    b = torch.randn(n, n, device="cuda")
    with nsight.annotate("matmul"):
        _ = a @ b

def main() -> None:
    # When profiling kernel1, this will be executed and returns a ProfilerResult object.
    # When profiling kernel2, this will also be executed (the second execution pass), but kernel1 returns None.
    result1 = kernel1()
    print("Kernel1 results:", result1.to_dataframe())  # When profiling kernel2, this cannot be accessed

    # This works - kernel2 returns a ProfilerResult object.
    result2 = kernel2()
    print("Kernel2 results:", result2.to_dataframe())  # When profiling kernel2, this can be accessed

if __name__ == "__main__":
    main()

Expected Behavior

Both profiling functions should return valid ProfilerResult objects that can be accessed independently:

# User can access multiple results.
results = []
results.append(kernel1())
results.append(kernel2())

[ConvolutedDog]

Proposal: Disk-Based Synchronization

I suggest implementing a simple disk-based synchronization mechanism:

1. When "NSPY_NCU_PROFILE" not in os.environ (main process):

• After launch_ncu() and extract_df_from_report() complete successfully
• Save the resulting DataFrame to disk (e.g., using pickle)
• Store it with a unique identifier (function name + run_id), but I'm not sure about this

2. When "NSPY_NCU_PROFILE" in os.environ (ncu subprocess):

• In the collect() method, when func.__name__ != name (not the function being profiled)
• Instead of returning None, load the previously saved DataFrame from disk
• Return the cached DataFrame, allowing access to prior profiling results

Benefits

• Compatible: Existing code continue to function without modification.
• Minimal Change: Leverages the existing execution flow with small additions.
• Simple implementation: Compared to AST-based script rewriting approaches, this solution requires less complex logic.

Example Implementation

def collect(self, func, configs, settings):
    # Check if running under ncu
    if "NSPY_NCU_PROFILE" not in os.environ:
        # Main process, run ncu and save results
        tag = f"{func.__name__}-{func._nspy_ncu_run_id}"
        report_path = f"{settings.output_prefix}ncu-output-{tag}.ncu-rep"
        
        # existing launch_ncu logic ...
        
        df = extraction.extract_df_from_report(...)
        
        # Save DataFrame to disk
        if df is not None:
            cache_path = f"{settings.output_prefix}cache-{tag}.pkl"
            df.to_pickle(cache_path)
            print(f"[NSIGHT-PYTHON] Cached results to {cache_path}")
        
        return df
    else:
        # ncu subprocess
        name = os.environ["NSPY_NCU_PROFILE"]
        
        if func.__name__ != name:
            # Try to load cached results from disk
            cache_pattern = f"{settings.output_prefix}cache-{func.__name__}-*.pkl"
            cache_files = glob.glob(cache_pattern)
            
            if cache_files:
                # Load the most recent cache
                latest_cache = sorted(cache_files)[-1]
                try:
                    cached_df = pd.read_pickle(latest_cache)
                    print(f"[NSIGHT-PYTHON] Loaded cached results from {latest_cache}")
                    return cached_df
                except:
                    pass
            
            # If no cache found, return empty DataFrame
            return pd.DataFrame()
        
        # existing logic for the function being profiled ...