Trying to set up hardware transcoding (GTX980M)

[dan_s_hall]

Server Version#: 1.22.2.4282
Player Version#: I use multiple apps and devices to view so I’m not sure what player version is tbh

Alright so this might be a bit of a wall of text, so I apologize for that.

To start things off I have had plex running on an older ASUS G751JY (i7 4710, gtx 980m, 24gb ram so not underpowered Imo) for quite a while now. Never really had any issues with plex specifically and have always just used software transcoding because I don’t really ever have more than a few streams going at once. Now that I have some 4k content I have noticed that even one transcode can use about 80-85% of the cpu, so I decided to try and get the hardware transcoding set up.

My server is Ubuntu 20.04 and I have always just run it with the cli.

Anyway, I looked up a guide and started following one from linuxconfig (not sure if I can post links cause I’m new but I can do that later if needed) and added the repository and started installing nvidia-driver-460. I initially ran into a bit of a snag with uefi secure boot not liking 3rd party drivers, so it asked me to create a password and then re-enter it on reboot (MOK key? Something like that?) to confirm that I was authorized. So after the restart a prompt popped up and I accidentally (yes I know, stupid) selected the wrong option and was never prompted for the password I had created, and so the drivers didn’t install correctly. I uninstalled them with “apt remove nvidia-driver-460” and then also used autoremove to get rid of the dependencies etc.

So then I tried just installing the drivers using the .run file from nvidia’s website. It gave me a few complaints about secure boot and once again didn’t work. So I removed those as well and tried to use the repository again, but now I get a message saying I am “holding broken packages” and the installation fails.

Sooo I guess long story short I would just like to “start over” if that makes any sense. I didn’t think something like this would actually be difficult or complicated but hey, Linux likes to throw me a curveball every now and then.

What can I do to fix my apt install and get the secure boot password prompt again? Or would there be a better (or easier haha) way of doing it?

Thanks in advance!

EDIT

Alright so I managed to get the drivers installed. This time I selected nvidia-driver-460-server, and I also was able to input the mok key properly on restart. My video still isn’t transcoding in hardware though. Any idea what the issue would be? Both hardware acceleration boxes are checked off in the transcoder settings of course.

EDIT #2

Made a bit more progress. I was getting this error when checking nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

I found out that it was possibly an issue with an incorrect cuda version so I got that installed and voila! I now have a (hw) in my plex dashboard next to my media when transcoding. My only issue that I’m noticing is that when I’m transcoding a 4k video from 80mbps or so down to 10mbps, it is still using about 70-80% cpu usage, so I don’t think it’s actually working. Any thoughts on where to go from here?

[pl_5309]

Check the matrix(Video Encode and Decode GPU Support Matrix [NEW] | NVIDIA Developer) even if you get the drivers working you will see that the GTX 980M only handles H.264(AVCHD), anything above that won’t work.

As soon as you start talking 4K then anything older than iX-7xxx series Intel chip requires a more recent GPU.

[dan_s_hall]

Ah I see. Well thanks for that! At least it will work for the h264 stuff though. At least… Once it’s working again

After a server restart, it is no longer using the hardware transcoder. I have tried unchecking and checking the hardware acceleration options in transcoder settings and also stopped and started the service. Nvidia-smi shows the following

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 980M    On   | 00000000:01:00.0  On |                  N/A |
| N/A   37C    P8     5W /  N/A |      1MiB /  4035MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Any advice on where to look in the logs for what the problem might be? I looked in the transcode logs but it didn’t really tell me much.