Do you have hardware accelerated transcoding enabled in your Transcoder settings?
Settings → Transcoder → Use hardware acceleration when available
If so, try disabling it to see if that’s what’s causing the issue. If so, you can then troubleshoot that side of things (try different video drivers, for example).
There were some pretty significant problems a while back on with PMS on Windows when hardware accelerated transcoding was in use. Here’s some background information:
