Replacing assets in Unity real-time but keeping the references intact

I’m looking forward to adding mod support into my game by allowing users to easily replace some assets with their own versions.

The way I want it to work – if there’s, say, a certain texture in some folder in my compiled game, I replace the texture with the same name in my resources, but I keep references intact. That means that the texture will be replaced for every material using this texture. I don’t want to change materials themselves to reference the new texture, mainly because I’ll have to keep a list of materials for each texture, and because materials might be replaced by potential modder as well. It’s also important to note that I want to give people the ability to change basically anything (but scripts). Sounds, music, textures, meshes, materials, fonts, localization files, shaders, so the list of references might be absolutely changed upside down and that is absolutely not the solution.