Do you have the server logs from any of these playback experiences?
I am on the Plex Pass branch, but have not seen this issue happening with the Xbox One app. I had an issue with certain files doing this ~2 years ago, (where the files would start on either a black screen or the first frame and say there) but haven’t seen the issue come back since then. At the time forcing the files to transcode (By disabling direct play/direct stream from the app settings) would fix the issue for me, but I would need more information to forward on to the developers for a long term fix.