Go with a 7th or 8th gen Intel CPU. They support h/w accelerated transcoding for H264 & H265 video on the internal GPU.
Do not add a discrete GPU unless you need it for something else. Nvidia consumer GPUs are limited to two simultaneous transcodes. Subsequent transcodes will hit the CPU, not the internal GPU. Here’s some info on using AMD GPUs from SlothTech: YouTube, Website
You don’t mention how many concurrent streams or transcodes. If the media direct plays, then you do not need much CPU. By using an Intel CPU, you can take advantage of h/w acceleration when transcoding H264/H265 media.
The Celeron in my Synology 918+ can easily direct play multiple streams. With h/w acceleration it can transcode 2 or 3 1080p streams. The i7-4790K in my desktop can handle at least six 1080p H264 transcodes using h/w acceleration (it does not have H265 capability).
The basic message is you don’t need an i9 or big Xeon server to run Plex. You might be able to get by with a current i5/i7, since the internal GPU is handling the video transcodes. Audio transcode will still be performed on the CPU, but the impact in minimal compared to video. Also, a few movies are in VC1 video, which is not h/w accelerated with Plex. A speedier CPU will help if you will be using the system for Handbrake, etc.
See Using Hardware-Accelerated Streaming for additional info.
8GB minimum. 16GB is nice.
Plex does not use that much RAM. Each instance of Plex Transcoder uses ~300MB on my Win10 system. The PMS process takes about 64MB RAM.
Having 16GB RAM gives you some extra headroom if you’ll be using the system for other things at the same time (ex: using Handbrake to process a video file while still serving up movies with PMS). Some people run related apps such as sonarr & radarr. You’ll need to take that into account if you plan to run such apps.
5400 will work. 7200 is better. See this post for more info. You can search the forum for “5400 7200” or similar and find many discussions.
I keep my media on a Synology 918+. It has a mix of HGST & Seagate IronWolf, both 7200 RPM. No problems with either. The HGST 10TBs are a bit noisy compared to the Seagate 12TB when moving a lot of data to/from the drives. I previously had HGST 4TB drives and they are very quiet. Rarely heard them during heavy read/write periods.
