You can try Plex Pass for $4.99 USD / month. If it works you can then choose monthly/yearly/lifetime subscription if desired. If it doesn’t work, cancel and you’re only out 5 bucks.
Your CPU is too old to use any embedded GPU. Per Using Hardware-Accelerated Streaming, you need a Sandy Bridge (2nd gen) CPU or later.
According to NVIDIA Video Encode and Decode GPU Support Matrix, your GPU does support H.264 decode & encode.
If you want x265 transcoding you’ll need a 7th gen or later CPU or a GTX 1050 or better Nvidia card.
Note 1: Nvidia consumer cards are limited to two simultaneous transcodes. The third simultaneous transcode hits the CPU.
Note 2: Audio is transcoded on the CPU. It doesn’t take many cycles, but just be aware it is there.
Just don’t do both at the same time, especially if transcoding. Doesn’t matter if you’ve an i7-xxx or an i9-7xxx. You can always stop PMS when you’re gaming. Just right-click on the icon in the taskbar and choose “Exit.”
Going forward:
Saw your post while writing this reply.
Give your current system a try and see how it works. Might be OK for you.
If you do go for a new system, you don’t need to go with an i9 & GTX 2080 for Plex. As mentioned earlier, you can use a lesser system. If you want the i9 & GTX 2080 for gaming, then go for it. Just not needed for Plex.
CPU: If you upgrade, go with a 7th gen or later CPU. The internal GPU can support x264 & x265 in hardware. You can check the CPU datasheets at Intel for specifics. Look in Datasheet, volume 1. Processor graphics usually in Section 2 (usually 2.4).
GPU: With a new Nvidia GPU, get a 1050 or better. See above linked Nvidia reference. At a minimum, you’ll want one that does x265 decoding. Plex currently encodes everything to x264. If they ever support encode to x265 then a 1050 or better will have you covered, as it supports x265 encoding (NVENC vs NVDEC tables).
Edit:
You don’t mention what Plex clients you’re using. When watching on a PC or Mac, use Plex Media Player (PMP) instead of Plex Web. PMP is much better. It supports more codecs natively, which means less transcoding. Plex Web is limited by the web browsers.
Nice feature of Plex Pass is it unlocks the mobile apps. No more one-minute timeouts.