encryption – How to ensure identity for server-to-server communication over IP

How would one go about ensuring that a server at a given IP address is who it says it is? I need to send confidential data to another server at another IP address. If the server had a public domain it is clear that a SSL certificate would be the right solution, but is this also the case for IP addresses?

I control a specific piece of software running on both servers, where Server1 will send confidential data to Server2. Server1 lives in a cloud I can control, but Server2 can live on any machine around the world as long as it is connected to the internet. At Server1, I have a registry of all Server2s that it can connect to.

I have thought about using a RSA key-pair, where the private key would live on Server2 and the public key would live in the registry on Server1 alongside the Server2 IP address. Then before sending the confidential data, I could present Server2 with the challenge of decrypting a random string and echoing back the result and check if it matches on Server1. The confidential data will of course also be encrypted with the public key, so an eventual man-in-the-middle attacker spoofing the IP address would get pure nonsense, but I don’t like the idea of someone eavesdropping on the communication.

TLDR; How do I make sure that a server at a specific IP address is not a man-in-middle attacker and is in fact a server that can be trusted with confidential data?