Bringing this full circle, yes, AMD’s encoders do not have as high as quality as NVIDIA’s or Intel’s latest generation encoder but let’s not also forget that Plex’s transcoder is a two-sided coin with hardware decoding as well. Also, as I had noted earlier, their hardware encoder does not exhibit the pixel distortion issue with either HEVC or h.264 in Handbrake which is also built directly from ffmpeg. To the point, AMD’s decoder is totally a different story from their encoder (see here: Does anyone review video decoding quality any more? | AnandTech Forums: Technology, Hardware, Software, and Deals). Their decoder has long been considered superior to certainly Intel’s, and slightly better than NVIDIA’s.
For me at least, AMD’s decoder comes very, very close to MadVR which is considered the gold standard for decoders by HTPC enthusiasts. That would mean of course that there is less quality loss when you use hardware decoding on AMD hardware. So yes, you may get better raw encoding quality with Intel or NVIDIA but that is tested generally without hardware decoding active, in the context of game streaming where you are capturing raw video from your screen. In contrast, in our use case, Plex uses both hardware encoding and decoding in the transcoding process and you want the least quality loss when decoding in real time in hardware before the source frames are passed to the hardware encoder.
