The card is too old.
A transcode has two parts: Decoding from the existing format and Encoding to the desired format. Plex encodes everything to H.264. Look for a card that decodes the desired formats.
You’ll want a GTX 1050 or better to get HEVC transcoding. See Nvidia Encode/Decode Matrix.
Plex also supports Intel Quick Sync Video for hardware acceleration, which is available on your CPU. Set the PC to use on-board video as the primary video adapter (BIOS setting) or just pull the video card. Note that this will only accelerate H.264 video. You need a 7th Gen or later Intel CPU for HEVC video.
Edit: Additional Info:
https://support.plex.tv/articles/115002178853-using-hardware-accelerated-streaming/
