Plex remote play

Does streaming inside your network play things ok?

Have you set a static ip and port forwarded your server in the router?

It sounds like you may be using UPnP which can have the kind of unreliability you’re describing

You can review this guide