On macOS, Plex doesn’t use hardware acceleration directly. It uses the macOS ‘VideoToolbox’ functionality. I’m pretty sure that the D500 is not supported by VideoToolbox for hardware acceleration.
https://support.plex.tv/articles/115002178853-using-hardware-accelerated-streaming/
HandBrake Documentation — Apple VideoToolbox
6,1 H264 and advise for a new system | MacRumors Forums
I think Plex does have some (unsupported) ability to use AMD graphics in Windows only, and I believe the D500 does have the necessary UVD/VCE capability in Windows.
AMD for HW-transcoding, or Intel/Nvidia only? - #6 by Wiidesire
AMD FirePro D500 Specs | TechPowerUp GPU Database
There are Boot Camp/Windows drivers for the D500. It looks like it used to be a pain, but might be straightforward today.
Update AMD graphics drivers for Windows in Boot Camp - Apple Support
https://www.amd.com/en/support/kb/release-notes/apple-boot-camp-previous
nMP: D300/D500/D700 do not work in Windows 10 | MacRumors Forums (Old)
That’s still an impressive machine, but if you get hardware transcoding working, on AMD, inside Windows, in a VM, with passthrough, you’re pretty far outside anything that’s supported. Maybe everything will align and it will work! But maybe not.
I kinda hope you try it and report back. 
There are a few reports of the M1 Mac Mini being able to do at least one 4K transcode and a handful of 1080P transcodes.
