Transcoding Failure - CUDA_ERROR_INVALID_VALUE: invalid argument

Server Version#: Version 1.30.1.6497
Player Version#: 5.52.3
Plex for Samsung
19_MUSEM_QTV

If I select a movie from my database to play on my Samsung TV, or the Plex Web interface i see a blank screen with a spinning icon. The media is supposed to be transcoded with nvenc hardware from HEVC to h264.

[Req#2c9e53/Transcode/majj58glmzmt09j10inbr8bj/4c15124f-dcab-465c-97e3-5c20a62c994d] [hevc @ 0x7fd3330b1540] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
[Req#2c9e55/Transcode/majj58glmzmt09j10inbr8bj/4c15124f-dcab-465c-97e3-5c20a62c994d] [hevc @ 0x7fd3330b1540] hardware accelerator failed to decode picture
[Req#2c9e58/Transcode/majj58glmzmt09j10inbr8bj/4c15124f-dcab-465c-97e3-5c20a62c994d] Error while decoding stream #0:0: Generic error in an external library
[Req#2c9e5b/Transcode/majj58glmzmt09j10inbr8bj/4c15124f-dcab-465c-97e3-5c20a62c994d] [hevc @ 0x7fd3330b1540] No decoder surfaces left

May I have the full logs ZIP please ?

Also please tell your Nvidia drivers version and GPU card

See attached and below.

Plex Media Server Logs_2023-01-02_18-13-31.zip (3.3 MB)

GPU 0: NVIDIA GeForce GTX 1660 (UUID: GPU-d3341101-7a97-1a26-a016-a3b8528ce6e3)
NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0

Is it difficult for you to install 515.86.01 and retest ?

I have a theory, based on a trend, which I can’t test here with the QNAP GT-1050

At the moment I have some encodes queued up (directly through ffmpeg) so I’ll have to wait until those are complete.
I can go down to 515.76, but that takes me back to September, this is an issue that just cropped up a week or two ago. The download date essentially correlates to install date.

-rwxr-xr-x 1 root root 343M Apr 29  2022 NVIDIA-Linux-x86_64-515.43.04.run
-rwxr-xr-x 1 root root 344M May 27  2022 NVIDIA-Linux-x86_64-515.48.07.run
-rwxr-xr-x 1 root root 347M Jun 24  2022 NVIDIA-Linux-x86_64-515.57.run
-rwxr-xr-x 1 root root 348M Jul 26 00:21 NVIDIA-Linux-x86_64-515.65.01.run
-rwxr-xr-x 1 root root 348M Sep 14 13:58 NVIDIA-Linux-x86_64-515.76.run
-rwxr-xr-x 1 root root 388M Oct  7 14:29 NVIDIA-Linux-x86_64-520.56.06.run
-rwxr-xr-x 1 root root 395M Nov 23 21:15 NVIDIA-Linux-x86_64-525.60.11.run

These would be the general upgrade dates for plex. 1.30.1 upgrade date better aligns with the timeline of the issue.

-rw-rw-r-- 1 zenon zenon  81M Nov 17 03:30 plexmediaserver_1.30.0.6406-f3f2fcc0a_amd64.deb
-rw-rw-r-- 1 zenon zenon  82M Dec  8 05:03 plexmediaserver_1.30.0.6486-629d58034_amd64.deb
-rw-rw-r-- 1 zenon zenon  82M Dec 14 15:06 plexmediaserver_1.30.1.6497-5fc2e0894_amd64.deb

I asked about 515.86.01 because I have it here as a known stable GPU function point.

Drivers version 525 upgrades Nvidia protocol from version 11.x to 12.0

What we’ll need to do is gather both DEBUG log ZIP sets (525 and 515) from Plex and then I’ll write up, using your log sets for the same video, and submit to Engineering / transcoder team.

If you can achieve full function and stability with Plex by using 515.86.01 then we can reasonably conclude there is an incompatibility with PMS and 12.0 Nvidia drivers.

515 drivers are more than enough for AV1 decode so you aren’t ‘boxed in a corner’ just yet.

This same thing happened when Nvidia went from 8->9, 9->10, 10->11 and here we are again at 12

It’s an eternal chase.

Ok, I’ve found the 515.86.01 driver but my encodes are still going so I can’t do the test yet.
When I do the test, you would like the same zip file I already sent, or do I need to do something different?

Just create another ZIP file from PMS (as PMS always does).
That’ll be enough.

I forgot to run another error log, so I’ll go back around with driver versions to do so.
515.86.01 now installed:

NVIDIA-SMI 515.86.01    Driver Version: 515.86.01    CUDA Version: 11.7

Video does not play in web browser, video does not play in Samsung TV app, log file attached.

Plex Media Server Logs_2023-01-04_09-06-46.zip (2.6 MB)

525.60.11 now installed:

NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0

Video does not play in web browser, video does not play in Samsung TV app, log file attached.

Plex Media Server Logs_2023-01-04_09-30-23.zip (2.2 MB)

Your logs are showing me either a failing device or a bad installation.

I have the same version, also on a xeon, only difference is your GT-1050 vs my P2200

failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 04, 2023 09:01:17.297 [0x7fef7c03fb38] ERROR - [Req#1a0/Transcode/5spzy3ghx4klgh81mohx90o8/fa7a703c-2543-4da5-ba32-a23359b484c6] [hevc @ 0x7f9c31a7d380] hardware accelerator failed to decode picture
Jan 04, 2023 09:01:17.298 [0x7fef7b88fb38] ERROR - [Req#1a1/Transcode/5spzy3ghx4klgh81mohx90o8/fa7a703c-2543-4da5-ba32-a23359b484c6] Error while decoding stream #0:0: Generic error in an external library
Jan 04, 2023 09:01:17.299 [0x7fef7e58fb38] ERROR - [Req#1a2/Transcode/5spzy3ghx4klgh81mohx90o8/fa7a703c-2543-4da5-ba32-a23359b484c6] [hevc @ 0x7f9c31a7d380] No decoder surfaces left
Jan 04, 2023 09:01:17.300 [0x7fef7d719b38] ERROR - [Req#1a3/Transcode/5spzy3ghx4klgh81mohx90o8/fa7a703c-2543-4da5-ba32-a23359b484c6] [hevc @ 0x7f9c31a7d380] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument

Play this controlled test file
Spears & Munsil - Test 01

I’m downloading the file now.
I tried re-encoding the same file with ffmpeg directly without issue.

I get about 1/3 through the test video and it locks up the Samsung app, the dashboard shows data transfer has stopped.
I looked at the original file again, it does show a transcoding process that’s doing something throughout the black screen, the dashboard shows that this transcode process is continuing along, but the bandwidth transfer doesn’t show any data being streamed.

Test Video
You can see that it locks-up the TV about 1/3 through the video.

Original Video
Here you can see the transcoder process is continuing along even though my TV shows a blank screen.

Wed Jan  4 13:10:57 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:83:00.0 Off |                  N/A |
| 45%   26C    P8     3W / 130W |    311MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     58441      C   ...diaserver/Plex Transcoder      308MiB |
+-----------------------------------------------------------------------------+

Here you can see the dashboard shows the movie is playing, but in fact nothing his been displayed on my screen and no data has been transferred out.

After about 10 minutes the dashboard shows the Samsung player disappear, and the transcode process also disappears from nvidia-smi.

This is all with the CUDA 12 driver, I’ll go back to the CUDA 11 driver and retry.

You can see my GPU is 26C, this is typical because it sits in a headless server and is only used for transcoding, if you think it’s truly a hardware issue I can try taking off the heatsink and reapplying with silver paste, but I really don’t think this will make a difference… What about if I re-encoded a movie several times with ffmpeg and did a bit comparison?
For example if a 2hr long media file the file is bit for bit identical I doubt it’s a hardware issue.

Plex Media Server Logs_2023-01-04_13-24-33.zip (2.4 MB)

Is this log file

Jan 04, 2023 13:11:46.570 [0x7f417aa1fb38] ERROR - [Req#268ee/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] [hevc @ 0x7ff04e215500] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 04, 2023 13:11:46.570 [0x7f417d6b9b38] ERROR - [Req#268ef/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] [hevc @ 0x7ff04e215500] hardware accelerator failed to decode picture
Jan 04, 2023 13:11:46.571 [0x7f417c09ab38] ERROR - [Req#268f0/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] Error while decoding stream #0:0: Generic error in an external library
Jan 04, 2023 13:11:46.574 [0x7f417edaab38] ERROR - [Req#268f1/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] [hevc @ 0x7ff04e215500] No decoder surfaces left
Jan 04, 2023 13:11:46.575 [0x7f417c82fb38] ERROR - [Req#268f2/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] [hevc @ 0x7ff04e215500] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 04, 2023 13:11:46.575 [0x7f417aa1fb38] ERROR - [Req#268f3/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] [hevc @ 0x7ff04e215500] hardware accelerator failed to decode picture
Jan 04, 2023 13:11:46.576 [0x7f417d6b9b38] ERROR - [Req#268f4/Transcode/c4v3y71oqqfys97r2s9trrpw/de0c969b-e4f1-48d1-8aaa-85ed16971fd7] Error while decoding stream #0:0: Generic error in an external library

from playing the video file I gave you?

If so, the problem is not Plex

If I take the message literally, it means your card cannot decode HEVC HDR @ 2160p

Who made it again ? (GT-1050’s are produced by a whole bunch of people each with their own firmware installed)

Don’t re-encode anything. That’s just ASKING for trouble.

At this point in time, I don’t care about the Samsung app.

Plex/web “basic” playback is more than enough.

Okay, I played in the web browser your test file and it failed at 6:27, see below. This is the 1st time I’ve seen an actual error when the stream stops playing. The log is attached.

I played the video 2x, it froze at the same point both times.

The video card was manufactured by MSI:

Plex Media Server Logs_2023-01-04_16-21-45.zip (2.7 MB)

This error is either :

  1. Transcoder temp is truly full
    -or-
  2. hanging up on the Dolby TrueHD problem (1.30.x)
Jan 04, 2023 16:19:01.989 [0x7f418111eb38] INFO - Plex Media Server v1.30.1.6497-5fc2e0894 - Ubuntu PC x86_64 - build: linux-x86_64 debian - GMT -06:00
Jan 04, 2023 16:19:01.989 [0x7f418111eb38] INFO - Linux version: 22.04.1 LTS (Jammy Jellyfish), language: en-US
Jan 04, 2023 16:19:01.989 [0x7f418111eb38] INFO - Processor: 24-core      Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
Jan 04, 2023 16:19:01.989 [0x7f418111eb38] INFO - Compiler is - Clang 11.0.1 (https://plex.tv 9b997da8e5b47bdb4a9425b3a3b290be393b4b1f)
Jan 04, 2023 16:19:01.989 [0x7f418111eb38] INFO - /usr/lib/plexmediaserver/Plex Media Server
Jan 04, 2023 16:19:01.985 [0x7f4170504b38] ERROR - [Req#55f9c/Transcode/scagxljzf6ommnkg9iv0oicg/99bcbee0-557e-4765-b82c-29515301d8a6] Error while decoding stream #0:1: No space left on device
Jan 04, 2023 16:19:01.986 [0x7f415fbf9b38] ERROR - [Req#55f9d/Transcode/scagxljzf6ommnkg9iv0oicg/99bcbee0-557e-4765-b82c-29515301d8a6] Error while decoding stream #0:1: No space left on device
Jan 04, 2023 16:19:01.986 [0x7f417a672b38] ERROR - [Req#55f9e/Transcode/scagxljzf6ommnkg9iv0oicg/99bcbee0-557e-4765-b82c-29515301d8a6] Error while decoding stream #0:1: No space left on device
Jan 04, 2023 16:19:01.989 [0x7f417a46fb38] ERROR - [Req#55f9f/Transcode/scagxljzf6ommnkg9iv0oicg/99bcbee0-557e-4765-b82c-29515301d8a6] Error while decoding stream #0:1: No space left on device
Jan 04, 2023 16:19:01.991 [0x7f415fbf9b38] ERROR - [Req#55fa1/Transcode/scagxljzf6ommnkg9iv0oicg/99bcbee0-557e-4765-b82c-29515301d8a6] Error while decoding stream #0:1: No space left on device

I watched the transcoding folder, it never exceeded 500 MB, so the device definitely has plenty of room. There has always been an issue with transcoded files stopping short a minute or two.
You’re saying this is a known issue with no fix?

Going back to my files that used to play, but won’t play anymore, any ideas?
Below are the video stream characteristics:

Video
ID                                       : 1
ID in the original source medium         : 4113 (0x1011)
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main@L5@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 2 h 0 min
Bit rate                                 : 29.9 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 0.602
Stream size                              : 25.1 GiB (90%)
Writing library                          : Lavc58.54.100 hevc_nvenc
Language                                 : English
Default                                  : No
Forced                                   : No
Color range                              : Limited

I downgraded the plex server and the video plays fine, logs attached.
Do you want me to install a different version to test?

plexmediaserver_1.30.0.6406-f3f2fcc0a_amd64.deb
Fri Jan  6 18:06:45 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:83:00.0 Off |                  N/A |
| 44%   29C    P0    17W / 130W |    297MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   1105983      C   ...diaserver/Plex Transcoder      294MiB |
+-----------------------------------------------------------------------------+

Plex Media Server Logs_2023-01-06_18-04-56.zip (4.8 MB)

This is a KNOWN issue.

The FIX is already set to be included in the next PMS release.