https://forums.plex.tv/t/hardware-transcoding-with-amd-cards/229434/22?u=feanor5
It seems work on AMD support on Plex side should be starting now!
Plex does not support currently vaapi (standard Linux transcoding API), only proprietary Intel/NVIDIA.