[c10d][fr] wait counter for dump function (pytorch#140823)

Summary:
Add a wait counter for the dump function.
This is useful to see if we get stuck in the dump function and never return for a particular job.

Test Plan: Tested locally I and see `pytorch.wait_counter.NCCLTraceBuffer__dump.busy_time_us.sum.60` in ODS.

Differential Revision: D65823433

Pull Request resolved: pytorch#140823
Approved by: https://github.com/fduwjj

Author: c-p-i-o

torch/csrc/distributed/c10d/NCCLUtils.cpp

@@ -2,6 +2,7 @@
 #include <torch/csrc/distributed/c10d/ProcessGroupNCCL.hpp>
 #include <torch/csrc/distributed/c10d/control_plane/Handlers.hpp>
 
+#include <c10/util/WaitCounter.h>
 #include <c10/util/env.h>
 #include <fstream>
 
@@ -843,6 +844,7 @@ std::string NCCLTraceBuffer::dump(
     bool includeCollectives,
     bool includeStackTraces,
     bool onlyActive) {
+  STATIC_SCOPED_WAIT_COUNTER(pytorch.wait_counter.NCCLTraceBuffer__dump);
   auto result = new_dict();
   // common values
   result.insert(version_key, version_val);