Two things.
- Dedicate an SSD to your Plex Media Server data folders! (see below)
- Add a video card that has a good HW transcoding engine – I have a AMD Polaris 11 based video card (not officially supported by Plex currently, but works pretty reliably transcoding several streams at one time (NVidia is limited to 2).
From what experience, the way PMS stores and accesses its “database” and thumbnail catalog is by saving thousands of tiny ~16-32kb size images in dozens and dozens of nested folders (for my libraries which are quite extensive my “Media” folder is over 78GB, 143k+ files, 303k+ folders!) and accessing those files on Plex is very slow and inefficient (ridiculous) – even a 10k drive is will be pretty slow at seeking and locating so many tiny files, thumbnails and metadata.
C:\Users_yourusernamehere_\AppData\Local\Plex Media Server
I used the above settings in Windows Disk Management to map my dedicated Plex SSD
Also, by having the drive mapped to both it’s own drive letter AND the PMS path, I can more easily optimize the drive and schedule image backups of this very delicate data which is all too easily lost or corrupt.
This will leave your system drive(s) to caching, transcoding, VM, etc your system is doing at the moment and won’t impact PMS wile it quickly grabs your data from the SSD (no seek or rotations necessary) as you would on your 10k (but still 2.5") drive(s).
The other benefit of moving all of the above folders and files to a dedicated drive for Plex, is when you do upgrade to a new, better and fast “pc”, you don’t have to (slowly) backup/zip your PMS folders and restore them onto your next pc/server – your PMS data will be portable and on an SSD which is much quicker to backup so many small files.
While this may not help your transcoding x265 video, it will help prepare you and Plex for your inevitable.
** of course, you will also want to backup your Plex program folder as well so you can make use of your server backup as mentioned above C:\Program Files (x86)\Plex\Plex Media Server (yea, not x64 bit! LOL)
For me personally, most of my collection is still being ripped into x264 files and some of my discs are ripped in both x264 and x265 – but that’s really limited to experimentation purposes.
My next system upgrade will be to a Ryzen APU - it will have 8 threads which Plex will enjoy, plus the next gen transcoding engine from AMD used in their latest gen GPU.
I am hoping (trusting) that 10bit HEVC (x265) will be supported by Windows and Plex with AMD GPU/APU technology - by the time this happens!
My current builds are based around AMD FM2+ sockets with Polaris 11 video cards (XFX single slot RX460). So while not super quick, or super efficient with regards to voltage, they are good enough to get me through this year and early next year. BTW, this system also records HDTV from 12 tuners, edits and transcodes NTSC and ATSC using the Polaris VCE to catalog and store my series and BD rips – a far cry from you 12 cores! 
