I’ve had similar issues on my Proxmox server where everything was working fine, and then one day HW transcoding through nVidia stopped working for windows clients, and then for the rest.
I did start a thread at HW Transcoding for Windows Clients failing