You cannot uninstall it, it’s part of the system and lots of services and apps rely on it. It not only stores passwords, but also certificates, keys and other information that needs to be stored securely. The
Keychain Access.app is merely a UI to access the keychain service; apps and services use dedicated APIs and for the shell, there’s the
security command. So even if you could remove
Keychain Access (the app), the service wouldn’t be affected. If you were to remove the service, lots of apps and system functionality would break.
Just don’t agree to store your password whenever you are asked.