We have 200K+ endpoints & 300+ operators but never faced such issue & never used such settings, its just working fine however initially it was taking lots of time to load but when we raised case HCL got that resolved with some additional stored procedures.
I would suggest open case for detailed investigation.