Just an FYI, I have validated the following setup DOES work as expected:
- x77 Ryzen based NAS
- Nvidia 1060-based GPU
- GPU Passthrough to Virtualization Station
- WIndows 10 x64 w/latest NVidia Drivers installed
- Latest PlexPass build of Plex
I tested HW Transcoding and it worked just fine in the above configuration (and actually it was fairly performant). Now I do have the VM files stored on an SSD volume which also helps with performance.
So Plex HW Transcoding (leveraging GPU Passthrough) does work just fine in a Win10 VM.