you don’t list a gpu, are using one ? is hardware acceleration enabled in the server settings ?
why are you transcoding in the first place? the main point of 4k is to have the best quality and transcoding obviously lowers quality
see also this may or may not help @ Plex, 4k, transcoding, and you - aka the rules of 4k - a FAQ
