your server status should tell you exactly what is happening, whether its the video or audio transcoding, or both.
atv4k does not pass through HD audio, so your first option is to simply use a different audio stream (IE dolby digital/dts 5.1 or stereo audio).
to get any benefit from 4k media, you want to avoid transcoding at all. direct play.
I don’t know anything about fedora, or how well HA works on that system, but assuming it does, then a 1050 should be sufficient for a couple 4k transcodes (which ideally you want to avoid in the first place), depending the video ram.
system ram won’t help unless your running out of ram from other applications, or need extra ram for disk caching of slow disks.
finally, you should be aware that gpu is not the miracle problem solver of 4k, it will only help with decode/encode of video, your system still has to be able to keep up with all the other things involved, including audio transcoding and subtitles, which are both single threaded so will be limited by your cpu capability.
you should carefully review the following threads.
Video Encode and Decode GPU Support Matrix | NVIDIA Developer
