Lost hp_metric when call self.trainer.test on fit end

[razgzy]

Bug description

In the init of L.LightningModule, I use

super().__init__()
self.save_hyperparameters()

to log hparams and in on_validation_end, i use
self.logger.experiment.add_scalar('hp_metric', hp_metric, global_step=self.current_epoch)
to log hp_metric. After fit, i call test once.

def on_fit_end(self):
        self.trainer.test(ckpt_path="last", datamodule=self.trainer.datamodule)

During training, everything is alright. The hp_metric iss logged correctly in Tensorboard. But after fit, the hp_metric in Tensorboard becomes (0, -1), all the logged hp_metrices are lost.
The code work well in lightning v2.2, but failed in v2.5

What version are you seeing the problem on?

v2.5

Reproduced in studio

No response

How to reproduce the bug

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- PyTorch Lightning Version (e.g., 2.5.0): 2.5.3
#- PyTorch Version (e.g., 2.7.1): 2.7.1
#- Python version (e.g., 3.12): 3.11.7
#- OS (e.g., Linux): Ubuntu 22.04
#- CUDA/cuDNN version: 11.8
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source): pip

More info

No response

cc @lantiga

[deependujha]

Hi @razgzy

Thanks for opening the issue. I'll shortly look into it.

You will get much faster help if you share a minimal reproducible example, ideally on Lightning Studio.

That way others can directly run your code in the same environment and confirm the regression between v2.2 and v2.5.

[SkafteNicki]

@razgzy can you try setting default_hp_metric when initializing TensorBoardLogger to False?

pytorch-lightning/src/lightning/fabric/loggers/tensorboard.py

Lines 57 to 58 in 3726e54

	default_hp_metric: Enables a placeholder metric with key `hp_metric` when `log_hyperparams` is
	called without a metric (otherwise calls to ``log_hyperparams`` without a metric are ignored).

[razgzy]

@razgzy can you try setting default_hp_metric when initializing TensorBoardLogger to False?

pytorch-lightning/src/lightning/fabric/loggers/tensorboard.py

Lines 57 to 58 in 3726e54

     default_hp_metric: Enables a placeholder metric with key `hp_metric` when `log_hyperparams` is 
         called without a metric (otherwise calls to ``log_hyperparams`` without a metric are ignored).

It works. The hp_metric is logged correctly, but now can not automatically log to HPARAMS by self.save_hyperparameters()

[SkafteNicki]

@razgzy in that case you can just call the log_hyparams method of the tensorboard logger yourself on the hparams attribute saved by save_hyperparameters. Probably do this in the same place you are logging hp_metric

self.logger.experiment.add_scalar('hp_metric', hp_metric, global_step=self.current_epoch)
self.logger.log_hyperparams(self.hparams, metrics=hp_metric, step=self.global_step)

I just want to make clear that the reason you are seeing this is due to you calling trainer.test on the same experiment. What happens is that when you call trainer.test it will automatically try again to log all hyperparameters, which ends up overwriting your hp_metrics with the default placeholder. Therefore an alternative is to do something like this

def on_fit_end(self):
    self.trainer.enable_autolog_hparams = False   # this is True by default, trainer argument
    self.trainer.test(ckpt_path="last", datamodule=self.trainer.datamodule)
    self.trainer.enable_autolog_hparams = True

where you for the test method disables the hyperparameter logging. For completness, it happens here in the trainer:

pytorch-lightning/src/lightning/pytorch/trainer/trainer.py

Lines 1010 to 1011 in e21b172

	if self.enable_autolog_hparams:
	_log_hyperparams(self)

[razgzy]

@SkafteNicki Thanks a lot for your comprehensive explanation. I'll close this issue.