windows – RDP to a server behind load balancer – allow per original source IP

I have some Windows 2016 servers that are located behind a load balancer. I don’t have any control over the load balancer other than being able to connect to servers via it. This whole set up is in the cloud. I need to enable RDP connection on the windows servers – to allow connections for management/monitoring/etc. As I don’t have any control over the load balancer, I cannot put any security/firewall/etc. rules on it. If I allow RDP connections to the servers, that means that anybody from anywhere can RDP onto them (provided they have credentials, of course, but that doesn’t help in case of any 0-day in RDP protocol for example).

When I RDP onto these servers, I have to specify a load balancer cookie indicating which server I’m connecting to – that’s the only control I have.

I can set up firewall rules on the servers themselves to only allow RDP connection from certain IP addresses. But when I enable RDP and connect to the server, the RDP session lists the remote address as the internal IP of the load balancer (i.e. 10.x.x.x) .

So, is there any way I can restrict RDP on servers to specific external IP addresses?