So just to add further evidence that this is not an issue with my server, disk IO, Local network, I was able to play back 4K HDR video perfectly fine on my Galaxy Tab S3 which supports 4k HDR. The videos played back in original quality in HDR and looked great on the tablet no issues, no buffering.
Play the same Video on the Xbox One X and it buffers at 3 mins exactly
When can we have a Plex dev talk in depth about the Xbox buffer issues and an ETA when they will be fixed? Been going on a year now I think we should be able to get an ETA by now.