Definitely best to use clients that support 4K if at all possible. Direct play/stream is easy and uses minimal server resources.
Upgrading the system is not difficult and not terribly expensive. Many people have added GPUs to their QNAP NAS. Something with a Pascal family chip (c. 2016) or later will work. This would provide the capability to transcode/tonemap 4K HDR. It also sets the system up for when Plex rolls out transcoding to HEVC.
Moving Plex to a new / different system can be reasonable. The N100 systems can handle 4K transcodes and can be found for $250 USD or less. Older model NUCs or other SFF PCs can sometimes be found on eBay for reasonable prices.
I went down the pre-transcode path when I first started with Plex. I spent a lot of time running DVD and BD rips through Handbrake to get around client and server limitations. Given how both Plex and hardware have evolved, I’m not sure that is still a good way to go.
A few years ago, I picked up a used PC on eBay that could handle Plex transcoding. Makes life a lot easier. I rip the disc, move it to my NAS, and enjoy. Plex transcodes if needed. I still use Handbrake, but only for video that is heavily interlaced, has compatibility problems, etc.
