It seems unlikely it is a connection issue, files that should transcode (e.g. old avi codec files) play through fine transcoding, and as mentioned, the files that should direct play continually switch between direct and transcode. It seems as if the server and/or roku app are either mis analyzing the files, or not making the correct decision about transcode/direct play. The roku log clearly says that the server decided it couldn't direct play the file. This is evidenced further by the fact that I've now turned direct play to 'off' in the roku app and for the moment everything seems to play properly (transcoding). This wouldn't happen if there was an intermittent connection issue right?
I've added exclusions to my media drives in my AV just in case. the server is just a windows PC with media on its own drives. The server and roku are both directly wired to my router, and all other devices (and indeed, netflix on the roku) work fine.
Does this occur when you try to direct play all of your media? Or does it only occur when certain media is direct played?
From your original statement and the Roku log, it is direct playing at the beginning, but then loses the connection (and states the connection failed) and only then does the Roku send the media to PMS to be transcoded. So the file is only transcoded when the Roku can no longer access the file directly. So the key is determining why the media can be accessed by the Roku at first, but that access is lost at some point during direct playback.
If you are hardwired and all the media is on the hard drive of your Plex Media Server computer, it should not be a connection issue. But if your media is stored on another device, it could be caused by that hard drive going to sleep after a period of time.
Plex Media Server may send more requests to the drive when transcoding media than the Roku may send when direct playing the media and that difference in the number of requests is keeping the connection alive and therefore you do not see the issue when media is transcoded. (I cannot be sure of this, but it would make sense that transcoded media is accessed more actively than media that is direct played).
And the error in the PMS log about the file being being used by another program seems to indicate that another program is trying to access that media and that is causing an access failure. The usual culprit of this is antivirus software, but it could be other programs that scan your media files. But this is more likely the case if the issue only happens some of the time.
I would not think that it is an issue with the media file itself unless it always occurs at the same part of the video and is not happening with every media file you play. If it does always occur at the same part of a particular video, you can always run that media file through Handbrake to try to resolve any issues with the media itself.