Just bumped into this issue myself as well.

I suggest we record the open attribute as something special like _rr_open="modal" if showModal was called to open the dialog box in this way.
Also we can call dialogEl.matches('dialog:modal') to check if the element was triggered with showModal or not.

For popover elements can listen to the toggle event and check .matches('[popover]:popover-open') to see if popover was opened. Or we could monkey patch HTMLElement.hidePopover() HTMLElement.showPopover() HTMLElement.togglePopover() to listen for toggles.

requestFullscreen is a bit tricker.

+1 on this.
We should be aware that in some UIs, there are cases where several layers of modals open above each other and it looks like:

@hpvd good point, however I haven't been able to find any way of accessing the order of opened dialog boxes. For full snapshots that's unlucky, for incremental snapshots it should be much easier to capture/replay this.