Plex tries, vaapi, nvenc, etc. to see if they’re supported. Then it uses whatever works (not sure about what priority but my guess is vaapi/qsv would be the first pick since it is officially supported). If all fail, it uses sw
The problem with nvidia decoding is that plex built their ffmpeg with support for the nvenc encoder, but not the cuvid decoder. That’s why it does hw encode, but sw decode
EDIT: what I wrote above applies to Linux. On windows, plex uses dxva to hw decode on nvidia, so it doesn’t need to use cuvid. But dxva is windows only.
EDIT2: see this post about the plex transcoder (ffmpeg) build details: https://forums.plex.tv/discussion/comment/1496184/#Comment_1496184
