Hardware transcoding issue

Plex Media Server
22

Having the same issue,

Jan 31, 2023 12:04:49.863 [0x7f9ea6461b38] ERROR - [Req#1ec5/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] Could not find ref with POC 196
Jan 31, 2023 12:04:49.868 [0x7f9ea7dbdb38] ERROR - [Req#1ec6/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] No decoder surfaces left
Jan 31, 2023 12:04:49.869 [0x7f9ea6461b38] ERROR - [Req#1ec7/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 12:04:49.869 [0x7f9ea7dbdb38] ERROR - [Req#1ec8/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] hardware accelerator failed to decode picture
Jan 31, 2023 12:04:49.869 [0x7f9ea6461b38] ERROR - [Req#1ec9/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 12:04:49.874 [0x7f9ea7dbdb38] ERROR - [Req#1eca/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] No decoder surfaces left
Jan 31, 2023 12:04:49.874 [0x7f9ea6461b38] ERROR - [Req#1ecb/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 12:04:49.874 [0x7f9ea7dbdb38] ERROR - [Req#1ecc/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] hardware accelerator failed to decode picture
Jan 31, 2023 12:04:49.875 [0x7f9ea6461b38] ERROR - [Req#1ecd/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 12:04:49.877 [0x7f9ea7dbdb38] ERROR - [Req#1ece/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] Could not find ref with POC 200

I have had the same results with NVIDIA drivers 525.60.11 and 525.85.05. Issue occurs with PMS 1.30.2.6563 and I have tried the linked 1.31.1.6617 from this thread with the same results. For 1.31.1.6617 I have only tried with the 525.85.05 drivers.

Example file data for a file that generates the issue (note other hardware transcoding works fine, and transoding this file to 720p with hardware rather than 1080p resolves the issue).

Media

    Duration 2:41:18
    Bitrate 3577 kbps
    Width 1920
    Height 804
    Aspect Ratio 2.35
    Video Resolution 1080p
    Container MKV
    Video Frame Rate 24p
    Audio Profile dts
    Video Profile main

Part

    Duration 2:41:18
    File <removed>
    Size 4.04 GB
    Audio Profile dts
    Container MKV
    Indexes sd
    Video Profile main

    Codec HEVC
    Bitrate 2068 kbps
    Language English
    Language Tag en
    Bit Depth 8
    Chroma Location left
    Chroma Subsampling 4:2:0
    Coded Height 832
    Coded Width 1920
    Color Primaries bt709
    Color Range tv
    Color Space bt709

Unfortunately with my users the easiest solution currently is to disable my hardware transcoding as the failure on the end user side is just a spinner with no error being presented to them. Transcoding the files from hevc to x264 using hardware acceleration and ffmpeg generates no problems, its just with the plex transcoder where I run into the issues.

System is running direct hardwre:
Fedora release 36
6.1.8-100.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jan 24 20:32:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
32GB Ram
GeForce GTX 1660
Intel(R) Core™ i9-9900K CPU @ 3.60GHz

Current drivers and plex version:
plexmediaserver-1.31.1.6617 (version from this thread)
NVIDIA Driver Version: 525.85.05
CUDA: Version: 12.0

23

ALL.

Because of all the possible permutations involved, We need to reduce this to the least common factor.

  1. PMS server
  2. Plex/Web playback in the browser - Playback quality 20 Mbps - 1080p
  3. nvdecExtraFrames="2"

If it doesn’t work here, there’s no reason to proceed because the browser makes the server do all the work.

Test 1:

Test 2:

Test 3:

Test 4:

Test 5:

Please reply using:

Distro Name and version:
Graphics card:
Driver version installed:

Test 1: Pass/Fail
Test 2: Pass/Fail
Test 3: Pass/Fail
Test 4: Pass/Fail/NA (Card cannot transcode HEVC/HDR)
Test 5: Pass/Fail/NA (Card cannot transcode VP9)

I will compile everyone’s results into a matrix and discuss with Engineering.

( I posted this elsewhere in the forum but forgot here. Sorry)

24

Fedora 36 6.1.8-100.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jan 24 20:32:33 UTC 2023 x86_64 x86_64
GeForce GTX 1660
NVIDIA Driver Version: 525.85.05
CUDA: Version: 12.0

Test 1: Pass
Test 2: Fail (getting the errors mentioned in previous post)
Test 3: Pass
Test 4: Pass
Test 5: Pass

25

@pozunmv

This error: ?

Jan 31, 2023 12:04:49.874 [0x7f9ea6461b38] ERROR - [Req#1ecb/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 12:04:49.874 [0x7f9ea7dbdb38] ERROR - [Req#1eca/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] No decoder surfaces left
Jan 31, 2023 12:04:49.874 [0x7f9ea6461b38] ERROR - [Req#1ecb/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 12:04:49.874 [0x7f9ea7dbdb38] ERROR - [Req#1ecc/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] hardware accelerator failed to decode picture
Jan 31, 2023 12:04:49.875 [0x7f9ea6461b38] ERROR - [Req#1ecd/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 12:04:49.877 [0x7f9ea7dbdb38] ERROR - [Req#1ece/Transcode/lrhbqj6s18xw2gdtw3amhug1/cd5b5135-4491-4662-882c-d4b741671431] [hevc @ 0x7f0827c2a980] Could not find ref with POC 200

If no decoder surfaces left

  1. Stop PMS.
  2. Increase nvdecExtraFrames by 2 more extra frames (e.g 2+2 → 4 or 4 + 2 → 6)
  3. Retest
  4. Share result and final nvdecExtraFrames value.
26

Yes that is the error, its not failing because there are no decoders left, thats just the error its throwing.

Modifying the transcode from 1080p to 720p works fine in hardware. This issue happens when transcoding 1080p hevc 8bit content (which unfortunately is what a lot of my content is in) to 1080p h264 only.

The server was not hardware transcoding any other files at the time, and the failure of the transcode is independent of how many transcodes are happening on the system. I generally have unlimited set and have patched the drivers to allow unlimited as well. I routinely, before this problem, had 6 or so users simultaneously hw transcoding files. For the test I set nvdecExtraFrames to 2 as in the instructions, but removed it after running the test, but I can confirm at the time of testing mine was the only transcode going on while there was a single other direct play happening.

27

If you have the time, can you spend a few minutes and play with the value of
nvdecExtraFrames ?

I’m suggesting a series of 2 - 4 - 6 - 8 - 10 - 12 - 16 (16 will be a lot of GPU memory)

28

Yeah, sorry I replied and then went and did it. I stepped up from 2 all the way up to 32 as the value in settings. Every step behaved the same way, though some through up a transcoder error faster than others (though dont think it was related to the value).

With set at maybe 12
Jan 31, 2023 16:01:50.741 [0x7fa9777c3b38] ERROR - [Req#17c3/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] Could not find ref with POC 897
Jan 31, 2023 16:01:50.741 [0x7fa97dd9ab38] ERROR - [Req#17c4/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] No decoder surfaces left
Jan 31, 2023 16:01:50.741 [0x7fa9777c3b38] ERROR - [Req#17c5/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 16:01:50.742 [0x7fa97dd9ab38] ERROR - [Req#17c6/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] hardware accelerator failed to decode picture
Jan 31, 2023 16:01:50.742 [0x7fa9777c3b38] ERROR - [Req#17c7/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 16:01:50.742 [0x7fa97dd9ab38] ERROR - [Req#17c8/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] No decoder surfaces left
Jan 31, 2023 16:01:50.742 [0x7fa9777c3b38] ERROR - [Req#17c9/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 16:01:50.742 [0x7fa97dd9ab38] ERROR - [Req#17ca/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] [hevc @ 0x7f00f9a65240] hardware accelerator failed to decode picture
Jan 31, 2023 16:01:50.743 [0x7fa9777c3b38] ERROR - [Req#17cb/Transcode/bv2t9s4v82hg9drbbj40c4k3/0c81e713-919d-44f3-85e3-0fcd93c5f287] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 16:01:56.949 [0x7fa9777c3b38] WARN - [Req#184c/Transcode/bv2t9s4v82hg9drbbj40c4k3] Transcode runner appears to have died.



With set to 32
Jan 31, 2023 16:15:35.963 [0x7fea77926b38] ERROR - [Req#1793/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 16:15:35.963 [0x7fea7355eb38] ERROR - [Req#1794/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] [hevc @ 0x7fe5cb944240] Could not find ref with POC 897
Jan 31, 2023 16:15:35.963 [0x7fea77926b38] ERROR - [Req#1795/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] [hevc @ 0x7fe5cb944240] No decoder surfaces left
Jan 31, 2023 16:15:35.963 [0x7fea7355eb38] ERROR - [Req#1796/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] [hevc @ 0x7fe5cb944240] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 16:15:35.963 [0x7fea77926b38] ERROR - [Req#1797/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] [hevc @ 0x7fe5cb944240] hardware accelerator failed to decode picture
Jan 31, 2023 16:15:35.964 [0x7fea7355eb38] ERROR - [Req#1798/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 16:15:35.964 [0x7fea77926b38] ERROR - [Req#1799/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] [hevc @ 0x7fe5cb944240] No decoder surfaces left
Jan 31, 2023 16:15:35.964 [0x7fea7355eb38] ERROR - [Req#179a/Transcode/gw8ajow9sy1yy6zp277yn8ft/2c88cd52-c598-437f-a8bb-d576bf94eb86] [hevc @ 0x7fe5cb944240] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
29

Heres a bit more information. This is with nvdecExtraFrames set to 32. First is what it looks like as its failing (usage from nvidia-smi and a tail -10 of the plex logs). Followed up by the 4k hevc properly transcoding.

Failed hevc 8bit transcode
Tue Jan 31 16:21:25 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.05    Driver Version: 525.85.05    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   49C    P0    48W / 130W |    296MiB /  6144MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     35673      C   ...diaserver/Plex Transcoder      292MiB |
+-----------------------------------------------------------------------------+
Jan 31, 2023 16:21:25.228 [0x7fea77073b38] ERROR - [Req#3123/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 16:21:25.229 [0x7fea7479cb38] ERROR - [Req#3124/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] Could not find ref with POC 897
Jan 31, 2023 16:21:25.229 [0x7fea77073b38] ERROR - [Req#3125/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] No decoder surfaces left
Jan 31, 2023 16:21:25.229 [0x7fea7479cb38] ERROR - [Req#3126/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 16:21:25.229 [0x7fea77073b38] ERROR - [Req#3127/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] hardware accelerator failed to decode picture
Jan 31, 2023 16:21:25.229 [0x7fea7479cb38] ERROR - [Req#3128/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 16:21:25.230 [0x7fea77073b38] ERROR - [Req#3129/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] No decoder surfaces left
Jan 31, 2023 16:21:25.230 [0x7fea7479cb38] ERROR - [Req#312a/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 16:21:25.230 [0x7fea77073b38] ERROR - [Req#312b/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] [hevc @ 0x7fcf5508f240] hardware accelerator failed to decode picture
Jan 31, 2023 16:21:25.230 [0x7fea7479cb38] ERROR - [Req#312c/Transcode/ci9b8ylxcxrmmq7wmvevjiqt/bf21209f-cd9a-43a4-b76c-c17feaeb2ea0] Error while decoding stream #0:0: Generic error in an external library

Transcoding the 4k hevc 10bit (It plays properly not sure what the VAAPI errors are about, maybe that was when I changed the web player from auto to 20MB/s)
Tue Jan 31 16:23:38 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.05    Driver Version: 525.85.05    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   50C    P0    52W / 130W |   1406MiB /  6144MiB |      8%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     35973      C   ...diaserver/Plex Transcoder     1402MiB |
+-----------------------------------------------------------------------------+
Jan 31, 2023 16:22:29.609 [0x7fea77073b38] WARN - [Req#327d/Transcode] Got a transcode session ping without a valid session ID.
Jan 31, 2023 16:22:59.618 [0x7fea77073b38] WARN - [Req#329a/Transcode] Got a transcode session ping without a valid session ID.
Jan 31, 2023 16:23:24.752 [0x7fea77073b38] ERROR - [Req#32b5/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:25.047 [0x7fea77073b38] ERROR - [Req#32b5/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:32.537 [0x7fea77073b38] ERROR - [Req#3313/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:32.843 [0x7fea77073b38] ERROR - [Req#3313/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:33.145 [0x7fea77073b38] ERROR - [Req#3313/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:33.447 [0x7fea77073b38] ERROR - [Req#3313/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:33.685 [0x7fea77073b38] ERROR - [Req#3313/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jan 31, 2023 16:23:33.922 [0x7fea77073b38] ERROR - [Req#3313/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).

I have also tested downgrading my drivers to 515.76 which are based on CUDA 11.7 and have the same issue

Tue Jan 31 17:05:23 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   42C    P0    44W / 130W |    295MiB /  6144MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     49199      C   ...diaserver/Plex Transcoder      291MiB |
+-----------------------------------------------------------------------------+

Jan 31, 2023 17:05:22.091 [0x7f4b028afb38] ERROR - [Req#8c38/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 17:05:22.091 [0x7f4b03acab38] ERROR - [Req#8c39/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] Could not find ref with POC 897
Jan 31, 2023 17:05:22.091 [0x7f4aff7b4b38] ERROR - [Req#8c3a/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] No decoder surfaces left
Jan 31, 2023 17:05:22.092 [0x7f4b02cb5b38] ERROR - [Req#8c3b/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 17:05:22.092 [0x7f4afe3a3b38] ERROR - [Req#8c3c/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] hardware accelerator failed to decode picture
Jan 31, 2023 17:05:22.092 [0x7f4b02ab2b38] ERROR - [Req#8c3d/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] Error while decoding stream #0:0: Generic error in an external library
Jan 31, 2023 17:05:22.092 [0x7f4b030bbb38] ERROR - [Req#8c3e/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] No decoder surfaces left
Jan 31, 2023 17:05:22.093 [0x7f4aff9b7b38] ERROR - [Req#8c3f/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] decoder->cvdl->cuvidDecodePicture(decoder->decoder, &ctx->pic_params) failed -> CUDA_ERROR_INVALID_VALUE: invalid argument
Jan 31, 2023 17:05:22.093 [0x7f4afe1a0b38] ERROR - [Req#8c40/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] [hevc @ 0x7f038c81bc80] hardware accelerator failed to decode picture
Jan 31, 2023 17:05:22.093 [0x7f4b0182db38] ERROR - [Req#8c41/Transcode/wii7ye256muu1yabw2bzd27m/3decc53f-5c00-4cfb-a876-249a712a84a3] Error while decoding stream #0:0: Generic error in an external library```
30

There might be two choices – BUT I MUST CONFIRM WITH ENGINEERING:

  1. Nvidia drivers 515.86.01 (pre 12.0 CUDA)
  2. PMS 1.29.2

I don’t know which.

31

I have tried downgrading to 1.29.x. However the DB did not seem to agree with the downgrade and while starting the server appeared to work it would not bring the system online and I just kept seeing downgrading DB messages with some failures to downgrade the DB.

I think I may have a backup from around that version, but would prefer not to have to roll back the data to there.

Also I think my last edit went through after you posted your reply I have tried with 515.76 which is pre CUDA 12

32

@pozunmv

Do what I do.

  1. Rename Library → Library_version.number
  2. Now start PMS
  3. You will get a new Library and server instance
  4. Give it a friendly name PMS-TEST-ver.sion.number
  5. Now add your smallest library section (Your test videos?)

This way, you can have multiple pre-loaded PMS versions

33

My Library 547GB, I have it on a local nvme that is 1TB, backup and all media is stored on a nas. Keeping around local different versions of the library and/or re-importing all media is a large pain. So I backup the data on occassion and I am trying to avoid having to rebuild/recreate my DB (or pull from backup) if possible.

$ sudo du -d 0 -h /var/lib/plexmediaserver/Library/
547G    /var/lib/plexmediaserver/Library/
34

Please re-read what I said.

You can further turn off “thumbnails” and “chapter markers”.

those things are a big waste of space if you don’t actively use them

35

@ChuckPa
I understand what you wrote. I can do that if you are asking me to test out whether an older version works (which I know it will as it was working previously with the CUDA12 based drivers) for more data on the issue? If you are asking for that I can temporarily do that. However I am fine moving to non HW based transcoding until a real patch is completed, I mean as long as its not months in the making. My other thought on that would be copying off the transcoder binaries and seeing if they work with the newer library without issue.

36

Hold up. I’ll try. I have problems producing the problems but will do my best.

37

I swapped from a baremetal install to docker today and noticed this issue too, I checked my baremetal installation and faced the same errors there. The below information may help.

OS: Ubuntu 22.04.1 LTS
GPU NVIDIA P400
Driver Version: 470.161.03
CUDA Version: 11.4

If I rollback the versions of the docker image I can get it working again, the same driver has been used since november 2022 so predates the PMS updates I think.

Broken:
1.31.0.6654 (docker:plexpass)
1.30.2.6563 (baremetal:latest)
1.30.2.6563 (docker:latest)
1.30.1.6562 (docker)

Working:
1.30.0.6486 (docker)
1.29.2.6364 (docker)
1.29.0.6244 (docker)

38

Any CUDA 11.x should be fine.

I was stable on 11.7 (515.86.01 drivers).

Because I work for Plex, I had to risk upgrading. I’m lucky. 525.60.13 works for me

Given the amount of failures in the 1.30.x family –

Go 1.29.2.6364 and sit there until it’s resolved.
(for me at least, 1.29.2.6364 works on CUDA 11.7 and CUDA 12.0 drivers)

39

Damn…Is this how PLEX fix issues…a multi million $ company…think about all the crap options they loaded into a paid application (that users did NOT even ask for…
and then give them ■■■■ about main product not working after a update…WAUW I SAY !..now for 8 days my PLEX server has not worked, FIX your software PLEX !!!

1 Like
40

No, this isn’t how it’s done but I’m here in the trenches with all of you and I’m trying to:

  1. Find the root cause
  2. Keep all of you working as best as I possibly can.

ALL:

We (users and I) have been doing more testing with PMS 1.29.2.6364 and Nvidia drivers 525 (CUDA 12).

Everything works so far.

Please load 1.29.2.6364 (I’ll get whatever is needed).

Let’s retest using 1.29.2.6364.

It’s the last datapoint I need.

NOTE: Even Trillion dollar companies have their problems. cough cough (I hate trolls)

41

They’re working on it and provided workarounds in the meantime.

  1. Downgrade Plex.
  2. Downgrade Nvidia driver.

Choose one and stop being unrealistic. Every software encounters issues out of their control. There are obviously some edge cases where it does not always happen.