It won’t be deleted.
When you remove an entire shared folder in one go (by clicking on the folder, and removing it), you won’t remove anything inside it. You can even add an entire shared folder to your My Drive, then later on remove the entire folder with one action, and the collaborators won’t lose anything. It simply just won’t be in your Drive anymore.
The same occurs with items in Shared with Me. When you remove an item from Shared with Me, you are simply taking it off the list you see yourself, and the item remains the same for everybody else.
You are correct in saying that items removed from within a shared folder do get removed, so please don’t do that!
A final point: one used to “add” shared items to one’s Drive; this is changing now to a shortcut being created by default, which won’t sync with your hard drive, or show the owner or last modified details in the My Drive section. You may get around this by pressing Shift + Z while the shared item is selected, if you want to add the original item to your Drive.