[QUESTION] How to time the code

[Weifan1226]

Hi Megatron team!

I wanted to time some of Megatron's code blocks, but didn't find a good way to do it.
But I was surprised to find code like this:

          if config.timers is not None:
                  config.timers('layernorm-grads-all-reduce', log_level=1).start(
                      barrier=config.barrier_with_L1_time
                  )
              _allreduce_layernorm_grads(model, config)
              if config.timers is not None:
                  config.timers('layernorm-grads-all-reduce').stop()

Can I us config.timers() to time the code? If yes, how can i see the output from config.timers.

Thanks!

[Advait251206]

Yes — you can absolutely use config.timers to time code in Megatron-LM, and it is actually the official internal profiling system used by the repo.

However, the key missing piece is: timers only print if you explicitly enable logging / reporting.

---

1. What config.timers actually is

config.timers is a wrapper around Megatron’s internal Timers class.

It supports:

start("name")
stop("name")
log()
reset()

So this:

config.timers('layernorm-grads-all-reduce').start()
...
config.timers('layernorm-grads-all-reduce').stop()

records elapsed time for that named region.

---

2. Why you don’t see any output

By default:

timers collect data but DO NOT print anything

So if you only add:

config.timers(...)

you will see zero output unless logging is triggered.

---

3. How to enable timer output

Option A (most common): enable --timing-log-level

When launching training:

--timing-log-level 1

or higher verbosity:

--timing-log-level 2

This is the main switch that activates timer reporting.

---

Option B: ensure timers are enabled

Make sure config has:

config.timers is not None

This requires:

--timers-interval <value>

Example:

--timers-interval 10

---

Option C: force print manually (debugging)

Inside code:

config.timers.log()

or:

config.timers.log(timers_to_output=None)

---

4. Where output appears

Timer logs usually show up in:

(A) stdout (rank 0)

Example:

iteration 10 | layernorm-grads-all-reduce: 0.32 ms
attention-forward: 12.4 ms
mlp-forward: 8.1 ms

(B) TensorBoard (if enabled)

If using:

--tensorboard-dir ...

timing metrics can be logged there too.

---

5. Important subtlety

Timers in Megatron are:

✔ hierarchical

✔ per-rank

✔ not automatically synchronized

So:

• each GPU reports its own timing
• no automatic averaging unless configured

---

6. Best practice for custom timing

If you want to instrument your own code:

if config.timers is not None:
    config.timers('my-custom-block').start()

x = my_function(x)

if config.timers is not None:
    config.timers('my-custom-block').stop()

Then ensure:

--timing-log-level 1
--timers-interval 1

---

7. Alternative (more precise than Megatron timers)

If you want higher fidelity:

PyTorch profiler:

torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA
    ],
    record_shapes=True,
    profile_memory=True
)

or NVTX ranges (best for Nsight Systems):

import torch.cuda.nvtx as nvtx

nvtx.range_push("my-block")
...
nvtx.range_pop()

---

Final takeaway

✔ Yes, config.timers is the correct built-in tool
✔ But it does NOT print unless timing logging is enabled
✔ Use --timing-log-level 1 to see output
✔ Combine with --timers-interval for periodic reporting

---

If you want, I can show you exactly where in Megatron the timer summaries are aggregated and printed (it’s buried in the training loop and not obvious at all).

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206