design – What are the pros and cons of using a reference/pointer vs an ID

I’m writing in C++, but this problem applies to any language without GC and even to languages with a GC as well.

I have a structure in memory in which I create/add objects. The structure takes ownership of those objects. I should never need to use an object after it’s removed from the structure.

When I first implemented this data structure, it seemed natural to use an ID/key/name/handle for the objects stored in it. I’m using it like this:

id1 = structure.addObj(new Square());
id2 = structure.addObj(new Square());
id3 = structure.addObj(new Circle());

obj3 = structure.getObj(id3);
obj3.addFriend( id1 );
obj3.addFriend( id2 );

idMax = structure.findObjWithMostFriends();
objMax = structure.getObj(idMax);

After using it for a while, I’m thinking that it would be better to forget about the IDs and always use references to the objects instead. This way I wouldn’t need to pass a reference to the structure around every time.

I’m thinking about refactoring everything to only use references but I’m afraid of regretting it. I’d like to know more about what are the pros and cons of using IDs to decide whether I should proceed.

Memory details:

The objects are allocated on the heap and their address never changes.

The structure deallocates those objects when they’re removed (they could be released to the caller instead, but I don’t need this at the moment).

I’m not supposed to ever use objects that don’t belong to the structure. If my program is correct, I should never end up with a dangling ID or pointer. But it could happen if the program has bugs.

What are your experiences switching from IDs to references for similar problems? Which solution should I use?