I do not have a solution but I believe I am experiencing the exact same thing.
2019 Shield TV Pro connected to 1G ethernet. The server is showing direct play both audio and video, which are ~25Mbps 4k dolby vision / HDR with DD+ Atmos audio. Playback will buffer for a few seconds at random places from numerous files. My networking logging on the Shield shows similar events as yours.
I have two other devices connected to the same 1G switch, an Apple TV 4k and an Xbox Series X, both play back the same files in Plex with no buffering. The Apple TV does not pass through the Atmos audio but the Xbox does. Switching to Kodi on the same Shield directly playing the file off the network plays perfectly with no buffering.
Rebooting the Shield does not improve things for Plex. If I uninstall and reinstall the Plex app, it will work and play without buffering long enough for me to watch a one hour show. If I come back the next day and play the same file, it will randomly buffer again.
My server and apps are all the most recent public releases. All of this points to a regression somewhere in the Plex app for Android. This is a relatively recent problem I think in the last few months as I’ve been using this same setup for years now without issue.