not sure if windows has an nvidia-smi utility, but if it does and you check it out, you can monitor your transcode gpu processes and how much ram/utilization they use.
here is an example of nvidia-smi and plex on linux @ Guide: NVDEC Hardware Acceleration Patch for Plex Media Server on Linux
