In Multisite the users profile picture keeps disappearing for “the other site”

I have the following setup: Site A with Domain A and Site B with Domain B

Site A is set to be the main site.

The users’ profile pictures have all been set via Site A and therefore Domain A.
They do show under “Network Admin > Users”, but they don’t show under “Site B > Users”.

If I now set an image for profile XYZ under “Site B > Users”, the URL for that profile pic changes to Domain B in the users table and there is no image anymore displayed for profile XYZ under Network Admin nor under Site A.

Any idea what the issue might be and is there a way around that?