CPU vs GPU Transcoding?

My experience seems to be a bit different, my plex server with 4 8280 xeons can start trans codes in under 3 seconds (4k even), with my RTX 4000, 13 seconds?

Am i missing a setting somewhere? I have 12 streams going on right now remotely, with the hardware encode/decode disabled, people report almost instant play, no quality issues, enable and they say 13 second waits before video files start.