only brief dropouts I’ve had seemed to be related to heavy cpu and network use during early pre-caching of tracks (there are Plexamp settings to address this) … issue you’re describing sounds different, like it takes a while before it shows up?
sometimes posting Plexamp logs (from the Pi, located in ~/.cache/Plexamp/log) that include the approximate time the issue occurred can give clues
