EDIT: I solved all my issues by installing the nvidia-450 driver as suggested here Converting to a lower quality fails with Hardware-Accelerated Streaming in Plex Web
Well. I tried the M2000 on a different board (with x16 and ubuntu 20.04) as I said and it works as it should, also the lower resolution options work fine. So I need to upgrade my server, no problem on the plex server-side.
