this thread Plex, 4k, transcoding, and you - aka the rules of 4k - a FAQ might help explain, but from your provided info, it could be a couple issues;
if your server is not powerful enough to handle the 4k video (direct play) while also transcoding the audio and remuxing it back with the video stream, then that could very well be a cause to buffering.
it shouldn’t take a bunch of cpu to the above, but it is very IO intensive, lots of DISK and NETWORK io going trying to move the bits around.
smart tvs themselves can be problematic, as you say many have only 100m ethernet and 2.4 wifi, which is not very suitable to full bitrate 4k/hdr streaming.
you could try switching audio stream to dolbydigital 5.1 or stereo, this should let your tv direct play both video and audio.
it may also reduce the bandwidth needed to avoid any network overloading and buffering.
for the full 4k hdr atmos experience, you need a playback chain that can handle both the VIDEO and the AUDIO streams.
smart tvs have problems with the HD audio, due to ARC limitations, and network bandwidth limitations, cause all kinds of headaches for folks.
the simplest 4k solution is nvidia shield + 4k/atmos receiver + 4k tv
