From the sound of it, there might be an issue with your remote access setup causing your remote connection to depend on a Plex Relay server (indirect connection).
Does the connection show as remote or indirect on the „now playing“ section of your server’s dashboard?
https://support.plex.tv/articles/200871837-status-and-dashboard/
If so you might need to troubleshoot that remote access setup.
Did you already have a look at the related troubleshooting support article?
https://support.plex.tv/articles/200931138-troubleshooting-remote-access/
There’s also a guide with some more details on how to troubleshoot remote access issues.
