Immediately after start of playback, the client device is filling up its network “receive” buffer.
Naturally, the server has to work harder at playback start, to prepare transcoded chunks of your video so they are ready for transmission.
Take a look at your Remote Access settings in the server configuration. Chances are that your server is not directly accessible from outside of your home network, so that all remote connections are taking a detour over plex’s cloud servers (a so-called “Relay connection”).
These are limited to max 2mbps for Plex Pass owners. (and 1mbps without Plex Pass).
Use the Dashboard in the web app to confirm the connection type. If it says “indirect”, a Relay connection is used.
See The Basics of Remote Access Troubleshooting
To reduce bandwidth requirements, your server should be able to use hardware transcoding to HEVC/H.265 format. Even if your server hardware is capable of doing this, you may need to explicitly enable this in the server settings.
