With the Divination Wizard in my party, the basic idea has always been that the Wizard tells me beforehand if they want to mess with a roll.
This usually takes the form of “I’m going to cast this and replace their saving throw” or “If this guy attacks my Rogue buddy, I’m failing his roll” and if none of those things happen, I just roll as usual.
I always make sure to include enough story and description to give players an idea of what’s going to happen and give them a moment to interrupt.
So if the evil advisor to the king is about to deceive him into getting the party thrown into jail, I will first say “The advisor bends over to the king and whispers things in his ear while giving you the evil eye” and then wait a second or two and if nobody reacts with “I’m interrupting this” then I make the roll, whatever it is. Could be a Deception, could be a Saving Throw for the king, depends on what happens.
But the players will know what in-game fiction they can interrupt and if the Wizard asks “what kind of roll is this? I might want to change it”, then I tell them, and they can decide to apply portent.
It could also sometimes include rolls that the players might not otherwise know had been made.
This shouldn’t come up as often as you think. A portent can only be applied when you can see the target, and generally when you see them you’ll be aware of what they’re doing. In the few exceptions, I’d call it out beforehand.