What’s a good way to inplement a "delete" function for my custom polymorphic widgets?

I’m currently trying to make a simple program with UI where a user can add, select and remove different objects inherited from an abstract common interface with a mouse.

What I plan to do is use a set<unique_ptr<Concrete_class*>> to store collections separately for each type of all the objects created and store one currently selected object in a unique_ptr<Abstract_class*>.

The problem is that if I want to delete the selected object there is no elegant way to do it: either I have to use a combination of .get(), dynamic casts and if/else chains to ensure the object is removed from the right set or store the selected object in a variant<Concrete_class1*, Concrete_class2*, ...> which makes using polymorphism pointless. It’s also impossible to store all the objects in one set since I plan to use their specific functions.

Is there a better way to approach this problem? I use dear imgui+sfml if that’s important.