With 1.18.4 beta, there is now a new Server Advanced Transcoder setting for hardware encoding. One thing to try now is disabling this and keeping Hardware Acceleration enabled to see if this avoids the crashes when running as a Service or running headless via RDP and could be the way to configure hardware transcoding
See release notice for 1.18.4.2164
(Preferences) Enable “Use hardware-accelerated video encoding” by default.
