A Load Balancer should act as a proxy, so the client sees the Load Balancer’s external IP as the destination for traffic to the Relay, and the Relay should see the Load Balancer’s internal IP as the source address for traffic coming from the client.
I’d first do a browser test by downloading https://relay:52311/masthead/masthead.afxm
In this case “relay” should resolve as the Load Balancer’s external IP address. That should either get the masthead file, or get a “403 Forbidden” if Relay Authentication is turned on. I’m not sure whether the Telnet test should work through a balancer.
I would guess at the moment the Load Balancer isn’t proxying the traffic correctly. Load Balancers in front of the relay can work, but be sure the Load Balancer uses a very sticky assignment - once a client has registered on one relay, it should not be transparently switched to a different internal relay.