I’ve seen lots of people posting here and on Reddit about RDP breaking HW transcoding and the only workaround is to use an RDP alternative like Google Remote Desktop or VNC.
After some extreme digging on the subject, I found the solution. First hint was found on an Emby support article linked below.
Open the group policy editor by running gpedit.msc
Navigate through the tree below:
Computer Configuration
Administrative Templates
Windows Components
Remote Desktop Services
Remote Desktop Session Host
Remote Session Environment
Open each of the following settings and set them to disabled
Use hardware graphics adapters for all Remote Desktop Services sessions
Prioritize H.264/AVC 444 graphics mode for Remote Desktop Connections
Configure H.264/AVC hardware encoding for Remote Desktop Connections
Restart Windows
I’ve tested this and found that while connected through RDP, QSV based HW transcoding seems to be working fine. There was one instance where a stream switched from HW to software when I switched from RDP to local login. I tried the same thing a few more times and couldn’t recreate the issue. HW decoding and HW encoding both running fine now
I believe this is a MS Windows problem as it I’ve read comments saying RDP is coded to take over the GPU completely and exclude all other apps from sharing it when remote sessions are active.
These settings were all “unset” (not disabled or enabled) on my server, but setting them to “disabled” seems to have done the trick for me. I can now HW decode and encode on the latest public PMS 1.25.5.5492.
There is a second RDP-related issue, which you may need to fix when you launch PMS from within your RDP session:
Open the registry editor of the server machine.
go to the branch HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Terminal Server\PerSessionTempDir
If it is set to 1 , change it to 0
Alternatively, you can also change the Group Policy or edit the RDP session properties:
To configure temporary folders on the RD Session Host server
On the RD Session Host server, open Remote Desktop Session Host Configuration. To open Remote Desktop Session Host Configuration, click Start , point to Administrative Tools , point to Remote Desktop Services , and then click Remote Desktop Session Host Configuration .
In the Edit settings area, under General , double-click either Delete temporary folders on exit or Use temporary folders per session .
On the General tab of the Properties dialog box, select or clear the Delete temporary folders on exit check box and the Use temporary folders per session check box based on what is appropriate for your environment, and then click OK .
If you clear the Use temporary folders per session check box, a user’s temporary files for all sessions on the RD Session Host server are stored in a common Temp folder under the user’s profile folder on the RD Session Host server.
You can also configure temporary folders on the RD Session Host server by using the Do not delete temp folder upon exit and the Do not use temporary folders per session Group Policy settings. These Group Policy settings are located in Computer Configuration\Policies\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Temporary folders and can be configured by using either the Local Group Policy Editor or the Group Policy Management Console (GPMC). Note that the Group Policy setting will take precedence over the setting configured in Remote Desktop Session Host Configuration.
