Unless I’m misunderstanding your request, it should be possible to accomplish this by using a custom server access URL; see “Custom server access URLs” in this document:
It doesn’t explicitly state it, but you can specify an IP address in the URL (or an FQDN). For example:
http(s)://203.0.113.10:32400 or
http(s)://plex.example.com:32400