dnd 5e – How can I prevent my players from teleporting out to escape any dungeon?

Kill, stun, or otherwise incapacitate the two spell casters

Or have your monsters/NPCs try to.
If your monsters have, as you say, seen or experienced stuff like this from this party or other beings ‘in world’ then their targeting of spell casters is smart play by the monsters/NPCs. But I’d be careful to consider how this set of monsters or NPCs know, or guess, about what the party may or may not do. Beyond that, counterspell is a perfectly valid choice for an enemy spell caster to take, so don’t limit yourself. Include it in the arsenal of some of the party’s opponents.

The incapacitated condition prevents taking actions. It requires an action to cast Teleport. Here are some spells to hit the party casters with:

Banishing Smite, Tasha’s Hideous Laughter, Imprisonment, Symbol, Power Word Stun (If they are stunned, they can’t take an action until your NPC’s next turn)

For a less certain case, cast confusion on them. 8/10 results don’t result in the PC taking an action.

If you want to be real sneaky, slap Bane on them and then cast the follow up (from above) if you have multiple casters on the enemy side. Bane lowers the target’s saving throw roll, however, since it targets Charisma, your Sorcerers and Clerics and Warlocks are not as likely to be influenced as a Wizard.

Stun them: the stunned condition prevents the stunned creature taking an action until the stunner’s next turn. There are a variety of monsters and/or spells that can stun. (Have a monk, or the NPC Martial Arts Adept attack them, for example).

Paralyzed creatures can’t take actions either. A variety of spells and poisons and monsters inflict the paralyzed condition.

A paralyzed creature is incapacitated (see the condition) and can’t
move or speak. (Appendix A)

There are a variety of ways to render the spell casters unconscious, and being Unconscious also prevents them taking actions. Do that.

Put a different way: don’t pull your punches as a DM when playing the monsters/Evil Overlords/Demon Lords/Whatever.

For some of the adventures, use GcL’s advice on demi-planes as that makes your job as DM easy. However, if all of your encounters above a certain level are only on demi-planes, you may get a few eyebrows raised by your players.

Lastly: put them on a ticking clock

If they teleport out of {this place} there is a time penalty often associated with getting back there to complete the task they had come there to complete, or thing to get. (Free the deva, recover the artifact, slay the demon prince, whatever). If they don’t get back in time, mission failure, and then (for example) the demon prince opens gate and ravening hordes of demons erupt in {party’s home country / someplace important} and so on.

Next mission becomes: stop ravening hordes of X (which admittedly is a bit tricky given that D&D 5e hasn’t yet arrived at a good “mass combat” methodology).