uefi – How is an EFI partition safe?

I realize that similar questions have already been asked regarding the EFI system partition, but I just can not handle it or get a direct answer.

When I boot into Linux, my files are protected, though fundamentally, by permissions. The same goes for Windows and of course for most modern O.S.

However, the ESP uses a FAT file system that can be easily mounted and, in the case of Linux, hooked up and therefore very easily messed up.

Why? Just why? How can this be considered safe? Every user of a system can play with the ESP and do what he wants. It seems to defy all reasonable security measures.

Of course, I know Secure Boot and key signing, but this must be turned on to be useful. Nonetheless, the ESP is still FAT and can still be confused, possibly blocking the system (for a normal end user).

Or do I miss something? I really feel that I am the only person who does not "understand" this at all.