1.23.0.4438 crash with HEVC files

walderston · April 27, 2021, 5:02pm

Server Version#: 1.23.0.4438-5eef0fd80
Player Version#: Web client
Device#: QNAP TVS-1277 4.5.2.1630

Playing video seems to cause the PMS to crash with the following in the console :

****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Crash Reports/1.23.0.4438-5eef0fd80/PLEX MEDIA SERVER/0fa78caa-eba5-43e2-ff51269a-e5459f29.dmp

Note that the dmp file doesn’t exist.

update: seems to happen with HEVC encoded files only. H264 encoded files play fine.

ChuckPa · April 27, 2021, 6:19pm

you’re getting core dump as the transcoder is attempting to load the Nvidia modules.

Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuSignalExternalSemaphoresAsync
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuWaitExternalSemaphoresAsync
Apr 27, 2021 17:47:05.844 [0x7fcbf96dab38] DEBUG - [Transcode/JobRunner] Job running: '/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/CrashUploader' '--directory=/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Crash Reports/1.23.0.4438-5eef0fd80' '--version=1.23.0.4438-5eef0fd80' '--platform=Linux' '--platformVersion=QTS 4.5.2.1630' '--serverUuid=d3b77deba96079eabb18c89a88bbd7422441d420' '--userId=adam@walderston.com' '--sentryUrl=https://sentry.io/api/1233455/minidump' '--sentryKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' '--vendor=QNAP' '--model=x86_64' '--device=TS-1277' '--allowRetries=0'
Apr 27, 2021 17:47:05.846 [0x7fcbf96dab38] DEBUG - [Transcode/JobRunner] Jobs: Starting child process with pid 22259

Do you have up to date Nvidia drivers ?
If so, please stop PMS
Go into the Codecs directory and remove the HEVC codecs
Restart PMS
Play again. PMS will download fresh HEVC codecs.

If this does not work:

Nvidia drivers issue
Card problem.

My best guess is the Nvidia drivers are damaged / out of revision but we’ll go with the HEVC driver first.

walderston · April 27, 2021, 7:05pm

@ChuckPa

deleted everything in /share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Codecs and still get the same issue.

If I revert back to the previous build (1.22.3.4392-d7c624def) I have no issues. All apps are up to date according to the QNAP AppCenter.

I do have a GPU installed but its in ‘Container Station Mode’ so shouldn’t be picked up by PMS. Looking at the QNAP control panel it shows the GPU usage as 0 - so its not that ?

ChuckPa · April 27, 2021, 7:22pm

If you have the card in Container Station mode, there is no reason this should be detected – but it is being detected.

Now I’m understanding the fault.

PMS sees the card (this is the problem)
It starts loading drivers in preparation for communicating
Communication with the card faults (permission/access)

Which version of QTS and NVIDIA_GPU_DRIVERS do you have installed ?

Apr 27, 2021 17:47:05.777 [0x7fcbf96dab38] DEBUG - [Transcode] Codecs: testing h264_nvenc (encoder)
Apr 27, 2021 17:47:05.777 [0x7fcbf96dab38] DEBUG - [Transcode] Codecs: hardware transcoding: testing API nvenc
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded lib: libcuda.so.1
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuInit
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDeviceGetCount
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDeviceGet
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDeviceGetAttribute
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDeviceGetName
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDeviceComputeCapability
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuCtxCreate_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuCtxSetLimit
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuCtxPushCurrent_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuCtxPopCurrent_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuCtxDestroy_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMemAlloc_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMemAllocPitch_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMemsetD8Async
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMemFree_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMemcpy2D_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMemcpy2DAsync_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGetErrorName
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGetErrorString
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuCtxGetDevice
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuStreamCreate
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuStreamQuery
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuStreamSynchronize
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuStreamDestroy_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuStreamAddCallback
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuEventCreate
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuEventDestroy_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuEventSynchronize
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuEventQuery
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuEventRecord
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuLaunchKernel
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuLinkCreate
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuLinkAddData
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuLinkComplete
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuLinkDestroy
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuModuleLoadData
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuModuleUnload
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuModuleGetFunction
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuTexObjectCreate
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuTexObjectDestroy
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGLGetDevices_v2
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGraphicsGLRegisterImage
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGraphicsUnregisterResource
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGraphicsMapResources
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGraphicsUnmapResources
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuGraphicsSubResourceGetMappedArray
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDeviceGetUuid
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuImportExternalMemory
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDestroyExternalMemory
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuExternalMemoryGetMappedBuffer
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuExternalMemoryGetMappedMipmappedArray
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMipmappedArrayGetLevel
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuMipmappedArrayDestroy
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuImportExternalSemaphore
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuDestroyExternalSemaphore
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuSignalExternalSemaphoresAsync
Apr 27, 2021 17:47:05.778 [0x7fcbf96dab38] DEBUG - [Transcode] [FFMPEG] - Loaded sym: cuWaitExternalSemaphoresAsync
Apr 27, 2021 17:47:05.844 [0x7fcbf96dab38] DEBUG - [Transcode/JobRunner] Job running: '/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/CrashUploader' '--directory=/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Crash Reports/1.23.0.4438-5eef0fd80' '--version=1.23.0.4438-5eef0fd80' '--platform=Linux' '--platformVersion=QTS 4.5.2.1630' '--serverUuid=d3b77deba96079eabb18c89a88bbd7422441d420' '--userId=adam@walderston.com' '--sentryUrl=https://sentry.io/api/1233455/minidump' '--sentryKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' '--vendor=QNAP' '--model=x86_64' '--device=TS-1277' '--allowRetries=0'
Apr 27, 2021 17:47:05.846 [0x7fcbf96dab38] DEBUG - [Transcode/JobRunner] Jobs: Starting child process with pid 22259

walderston · April 27, 2021, 7:25pm

QTS 4.5.2.1630
NVIDIA GPU Driver 4.0.2
NvKernelDriver 4.5.2.1630

ChuckPa · April 27, 2021, 7:27pm

Is NvKernelDriver needed?

It’s been my experience, and what I code for, only the QNAP driver is needed.

The Nv driver creates a secondary path to the device.

walderston · April 27, 2021, 7:28pm

I didn’t install the NvKernelDriver… Can’t seem to stop or remove it either (at least not from the GUI)

ChuckPa · April 27, 2021, 7:34pm

They changed something in 4.5.2

It looks like they broke out the kernel module to be free standing

would you mind SSH into the QNAP and typing this command ?

echo /sys/module/nvidia/drivers/pci:nvidia/*:*:*.*

Next, Let’s look at the GPU itself

gpuhal_app -l

Look at the ID number returned (first column)

gpuhal_app -s $ID

Let me know what it reports please

walderston · April 27, 2021, 7:38pm

/sys/module/nvidia/drivers/pci:nvidia/0000:0f:00.0

f:NVIDIA Corporation:GP106 [GeForce GTX 1060 3GB]:external:7

active_status=4,real_status=4,driverqpkg=NVIDIA_GPU_DRV,driver_installed=1,in_used=container-station,transcode=0,display=0,pcie_slot=1,pcie_vp=1,va_type=none

ChuckPa · April 27, 2021, 8:21pm

Thank you.

The card is not in QTS mode: real_status=4

I do not add the Nvidia drivers to the PMS path unless real_status=2 (QTS mode)

Somehow it is still finding the drivers.

I need investigate this on our LAB system (also a Ryzen)

walderston · April 27, 2021, 8:23pm

no worries… give me a shout if you need anything else or to test anything else… for now, I’ll stick with the older 1.22.3.4392 release which seems to be working.

ChuckPa · April 28, 2021, 4:20am

@walderston

We identified where & how it’s happening. It’s the result of changes made in 1.23.0.
It’s a small issue with it not obeying the word “No”
We’re tracking it down where to fix it now. Should be easy to remedy. I’ll keep you in the loop.

ChuckPa · April 29, 2021, 2:04am

@walderston

Would you be willing to give this a test drive ?

I changed how it checks for permission to use the GPU.
In what I see here, it’s working good for me with or without Nvidia.

The crash you saw was because the card was busy doing something when PMS came in and started accessing it. (QNAP’s enforcement of permissions isn’t the best).

Please confirm it does not engage the Nvidia when not in QTS mode.
Please also confirm proper (hw) utilization.

walderston · April 30, 2021, 4:55pm

@ChuckPa sorry for the delay… Yep seems to be all good now and working as expected.

broncogr · April 30, 2021, 5:49pm

@ChuckPa
Mine is still crashing even with your .4438 custom version linked above as well as with the latest .4459.
It only crashes when it needs to transcode through the Nvidia GPU, direct play has no issue playing on your version.
Last version with no issues was PlexMediaServer-1.22.3.4392-d7c624def-x86_64 and any plex pass versions after that from the 1.22 branch.

I am on quts hero 4.5.2.1638 using an Nvidia GTX 1650 in QTS mode
I have removed all codecs from the codecs folder and new ones were downloaded but the issue persists

walderston · April 30, 2021, 6:39pm

just tried mine in QTS mode and it plays correctly without issue transcoding from HEVC to h264

broncogr · April 30, 2021, 8:06pm

Unfortunately, for me it crashes without providing a crash log for me to share.
This happens with 1.23.0.4438, the custom 4438 attached above, as well as 4459 which was released today.
No such issues with 1.22.3.xxxx

ChuckPa · April 30, 2021, 8:34pm

I need DEBUG log files (the ZIP) which captures this happening.

broncogr · April 30, 2021, 8:42pm

Sure.
So i should get those from download logs under troubleshooting?
I will reinstall your 4438 version tomorrow as it is getting late here now.
Since plex crashes after trying to use transcoding should I stop/ start the app and then provide the zip?
Should I just post it here afterwards?
Thanks

ChuckPa · April 30, 2021, 8:53pm

Yes… Settings - Server - Troubleshooting - Download logs.

Topic		Replies	Views
QuTS / QTS - 4.5.2 / 4.5.3 - Nvidia GPU failures NAS & Devices server-qnap	74	1458	May 18, 2021
New PlexServer releases no NVIDIA GPU encoding on QNAP NAS & Devices server-qnap	51	1791	June 8, 2021
Hardware transcoding on Qnap TVS-672XT NAS & Devices server-qnap	85	3106	January 5, 2021
After updating qts qnap and gpu nvidia KO NAS & Devices server-qnap	50	456	May 5, 2021
Potential issue with QNAP transcoding new NVIDIA_GPU_DRV_5.1.0.0822_x86_64 NAS & Devices server-qnap	47	1258	March 18, 2023

1.23.0.4438 crash with HEVC files

Related topics