PS4 app buffers movies with remote server but not local
I read all the different threads about buffering and am not sure what I'm doing wrong. I was trying to get Plex on PS4 working at a friend's house with my remote server but it kept buffering, so I figured it was perhaps his network. I then tried on my own PS4 and realized I'm having the same issue, except PS4 is the only device that is having this problem and no other device (Roku, Chromecast, browser, etc). I've tried lowering quality, and my server is transcoding fine,has plenty of power to handle the transcode (the speed is usually 3-4), but there's still buffering. I tried 8mbps and even 4mbps which still stutters, although 4mbit buffers less.
Playing the same video over local network on PS4 I can direct play a 16mbit video just fine, and if I set it to 8mbit it also plays fine. The issue is only with remote play. But my network is not the issue, and my server is also fine. I can play the same video on my Roku 3 both direct play and transcoding, and both Roku and PS4 are connected through ethernet. I've tested lower bitrate movies, such as a 720p 4.4mbit movie which direct streams and still stutters occasionally.
I thought perhaps updating server software would make a difference but it didn't, currently on 0.9.15.0 on linux server. If I run a speed test in PS4 network settings I get around 30mbit download. If I play Netflix's example short movie that shows bitrate, it plays the 5800kbps 1080p movie just fine (the highest one)
I can post some logs, run some tests, etc if that would help narrow this down