No Linux Transcoding Headless Server in ESXi

Plex Media Server Desktops & Laptops
1

Server Version#: 1.14.1.5488
Player Version#: Android Plex App Exoplayer2 or browser player

Here’s where I am at:
Virtual Server in vmware ESXi 6.5
Guest OS: Ubuntu Server 18.04 LTS
GPU: Nvidia Quadro P400 in pass thru to the Plex

I’m not seeing the (hw) on the web client when transcoding to either the browser player or my phone. I know the Nvidia card can’t to decoding, but I assumed I would see (hw) after the part that shows the encoded stream.

Do I need to use the full desktop Linux in order to get hardware encoding? Am I doing something wrong with my setup in ESXi? Any help would be greatly appreciated. I see my CPU maxing out (with 8 “cores” allocated of my e5-2695v2 to plex). I can add more cores, but I would prefer to leave more for other VMs if at all possible.

2

I need to see the logs.

PMS itself relies on you to have all the nvidia drivers properly installed.
If they aren’t, PMS won’t detect the card.

Also, did you enable passthrough of the specific card?

3

Here’s at least some parts of the log that might be helpful.

Feb 09, 2019 21:28:42.290 [0x7f568b7ff700] DEBUG - Codecs: testing h264_nvenc (encoder)
Feb 09, 2019 21:28:42.316 [0x7f568b7ff700] WARN - avcodec_open2 returned -1313558101 for encoder ‘h264_nvenc’
Feb 09, 2019 21:28:42.318 [0x7f568b7ff700] DEBUG - Codecs: testing h264_nvenc (encoder)
Feb 09, 2019 21:28:42.328 [0x7f568b7ff700] WARN - avcodec_open2 returned -1313558101 for encoder ‘h264_nvenc’
Feb 09, 2019 21:28:42.328 [0x7f568b7ff700] DEBUG - Streaming Resource: Determining preferred transcode encoders through transcode only decision.
Feb 09, 2019 21:28:42.328 [0x7f568b7ff700] DEBUG - Codecs: testing h264_nvenc (encoder)
Feb 09, 2019 21:28:42.336 [0x7f568b7ff700] WARN - avcodec_open2 returned -1313558101 for encoder ‘h264_nvenc’
Feb 09, 2019 21:28:42.337 [0x7f568b7ff700] DEBUG - Codecs: testing h264_nvenc (encoder)
Feb 09, 2019 21:28:42.343 [0x7f568b7ff700] WARN - avcodec_open2 returned -1313558101 for encoder ‘h264_nvenc’
Feb 09, 2019 21:28:42.345 [0x7f568b7ff700] DEBUG - Codecs: testing h264_nvenc (encoder)
Feb 09, 2019 21:28:42.353 [0x7f568b7ff700] WARN - avcodec_open2 returned -1313558101 for encoder ‘h264_nvenc’
Feb 09, 2019 21:28:42.355 [0x7f568b7ff700] DEBUG - Codecs: testing h264_nvenc (encoder)
Feb 09, 2019 21:28:42.363 [0x7f568b7ff700] WARN - avcodec_open2 returned -1313558101 for encoder ‘h264_nvenc’
Feb 09, 2019 21:28:42.364 [0x7f568b7ff700] DEBUG - Streaming Resource: Reached Decision id=1159 codes=(General=1001,Direct play not available; Conversion OK. Direct Play=3000,App cannot direct play this item. Direct play is disabled. Transcode=1001,Direct play not available; Conversion OK.) media=(id=1337 part=(id=1338 decision=transcode container=mp4 protocol=dash streams=(Video=(id=20836 decision=transcode bitrate=1323 encoder=libx264 width=840 height=480) Audio=(id=20837 decision=transcode bitrate=102 encoder=aac channels=2 rate=48000))))
Feb 09, 2019 21:28:42.529 [0x7f5687ffe700] DEBUG - TPU: hardware transcoding: final decoder: , final encoder:
Feb 09, 2019 21:28:42.530 [0x7f5687ffe700] DEBUG - Job running: EAE_ROOT=’/tmp/pms-98f820d4-671c-4ad1-a3a8-d85c391a5a87/EasyAudioEncoder’ FFMPEG_EXTERNAL_LIBS=’/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/531e313-1328-linux-ubuntu-x86_64/’ XDG_CACHE_HOME=’/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache’ XDG_DATA_HOME=’/usr/lib/plexmediaserver/Resources’ X_PLEX_TOKEN=‘xxxxxxxxxxxxxxxxxxxx’ ‘/usr/lib/plexmediaserver/Plex Transcoder’ ‘-codec:#0x01’ ‘h264’ ‘-codec:#0x02’ ‘aac_lc’ ‘-probesize’ ‘10000000’ ‘-i’ ‘/nfs/Storage/DVD/Chasing Amy.m4v’ ‘-filter_complex’ ‘[0:#0x01]scale=w=840:h=480[0];[0]format=pix_fmts=yuv420p|nv12[1]’ ‘-filter_complex’ ‘[0:#0x02] aresample=async=1:ocl=’’‘stereo’’’:osr=48000[2]’ ‘-map’ ‘[1]’ ‘-codec:0’ ‘libx264’ ‘-crf:0’ ‘24’ ‘-maxrate:0’ ‘1323k’ ‘-bufsize:0’ ‘2646k’ ‘-r:0’ ‘23.998999999999999’ ‘-preset:0’ ‘veryfast’ ‘-x264opts:0’ ‘subme=2:me_range=4:rc_lookahead=20:me=hex’ ‘-force_key_frames:0’ ‘expr:gte(t,0+n_forced7)’ ‘-map’ ‘[2]’ ‘-metadata:s:1’ ‘language=eng’ ‘-codec:1’ ‘aac’ ‘-b:1’ ‘102k’ ‘-f’ ‘dash’ ‘-min_seg_duration’ ‘7000000’ ‘-skip_to_segment’ ‘1’ ‘-time_delta’ ‘0.0625’ ‘-manifest_name’ ‘http://127.0.0.1:32400/video/:/transcode/session/srgul8xiijfmr8tc0zo3dcc1/e12a5f67-b809-40ae-87d8-570e7f0c74ec/manifest’ ‘-avoid_negative_ts’ ‘disabled’ ‘-map_metadata’ ‘-1’ ‘-map_chapters’ ‘-1’ ‘dash’ ‘-start_at_zero’ ‘-copyts’ ‘-vsync’ ‘cfr’ ‘-y’ ‘-nostats’ ‘-loglevel’ ‘quiet’ ‘-loglevel_plex’ ‘error’ ‘-progressurl’ ‘http://127.0.0.1:32400/video/:/transcode/session/srgul8xiijfmr8tc0zo3dcc1/e12a5f67-b809-40ae-87d8-570e7f0c74ec/progress’
Feb 09, 2019 21:28:44.303 [0x7f56897ff700] DEBUG - TPU: hardware transcoding: final decoder: , final encoder:
Feb 09, 2019 21:28:44.304 [0x7f56897ff700] DEBUG - Job running: EAE_ROOT=’/tmp/pms-98f820d4-671c-4ad1-a3a8-d85c391a5a87/EasyAudioEncoder’ FFMPEG_EXTERNAL_LIBS=’/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/531e313-1328-linux-ubuntu-x86_64/’ XDG_CACHE_HOME=’/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache’ XDG_DATA_HOME=’/usr/lib/plexmediaserver/Resources’ X_PLEX_TOKEN=‘xxxxxxxxxxxxxxxxxxxx’ ‘/usr/lib/plexmediaserver/Plex Transcoder’ ‘-codec:#0x01’ ‘h264’ ‘-codec:#0x02’ ‘aac_lc’ ‘-ss’ ‘2408’ ‘-probesize’ ‘10000000’ ‘-i’ ‘/nfs/Storage/DVD/Chasing Amy.m4v’ ‘-filter_complex’ ‘[0:#0x01]scale=w=840:h=480[0];[0]format=pix_fmts=yuv420p|nv12[1]’ ‘-filter_complex’ ‘[0:#0x02] aresample=async=1:ocl=’’‘stereo’’’:osr=48000[2]’ ‘-map’ ‘[1]’ ‘-codec:0’ ‘libx264’ ‘-crf:0’ ‘24’ ‘-maxrate:0’ ‘1323k’ ‘-bufsize:0’ ‘2646k’ ‘-r:0’ ‘23.998999999999999’ ‘-preset:0’ ‘veryfast’ ‘-x264opts:0’ ‘subme=2:me_range=4:rc_lookahead=20:me=hex’ ‘-force_key_frames:0’ 'expr:gte(t,2408+n_forced7)’ ‘-map’ ‘[2]’ ‘-metadata:s:1’ ‘language=eng’ ‘-codec:1’ ‘aac’ ‘-b:1’ ‘102k’ ‘-f’ ‘dash’ ‘-min_seg_duration’ ‘7000000’ ‘-skip_to_segment’ ‘345’ ‘-time_delta’ ‘0.0625’ ‘-manifest_name’ ‘http://127.0.0.1:32400/video/:/transcode/session/srgul8xiijfmr8tc0zo3dcc1/559b25ea-559b-4f61-93cc-c3eebab6625c/manifest’ ‘-avoid_negative_ts’ ‘disabled’ ‘-map_metadata’ ‘-1’ ‘-map_chapters’ ‘-1’ ‘dash’ ‘-start_at_zero’ ‘-copyts’ ‘-y’ ‘-nostats’ ‘-loglevel’ ‘quiet’ ‘-loglevel_plex’ ‘error’ ‘-progressurl’ ‘http://127.0.0.1:32400/video/:/transcode/session/srgul8xiijfmr8tc0zo3dcc1/559b25ea-559b-4f61-93cc-c3eebab6625c/progress’

And yes I have validated that Nvidia drivers version 390 are installed and yes my Quadro P400 is in pass through and it shows up in my devices in my headless command line only Plex Server. I’ve also disabled the “svga” that ESXi adds so that the p400 is the only gpu (now I can only access via SSH and the web client).

03:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP107GL [Quadro P400]
Physical Slot: 160
Flags: bus master, fast devsel, latency 64, IRQ 18
Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at 4000 [size=128]
Capabilities:
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

4

Thanks,
This confirms PMS could not talk to the nvidia driver to open a communication channel. As a result, the final encoder shown is empty (software)

When it can access the card through the drivers, it will clearly show nvenc and will also show in the FFMPEG command line.

I can only tell you that on a bare machine (e.g. a QNAP with GT1060) I am able to make it work.
During the PMS startup there, I must validate the card’s existence, verify the drivers are installed and, perhaps the key point for you as well, append the drivers’ path to the LD_LIBRARY_PATH before launching PMS.

5

LD_LIBRARY_PATH was specified and validated. Not sure what I am doing wrong. Either way, for now I created a Windows 10 vm (oy) and was able to get it up and running with hw encoding in about 2 hours (validated that I was seeing the hw on the server). Not ideal as that’s a lot of overhead, but then again I have the resources on my VM Server, so throwing a few extra GB’s of RAM isn’t the end of the world if it means I reduce my CPU usage.

6

The way PMS works with nVidia is through their API first.
If their API (driver modules) can see the card, it reports back to PMS.
PMS doesn’t open the card directly like it does with QSV. It relies on the API.

I do not use ESXi but ask if both VMs have the same passthrough options enabled?

7

As far as passthrough, they are both configured the same way. I probably just configured something wrong on the linux side for the drivers. It’ll be something I’ll play with in the future, but for now I have a solution (albeit less than ideal).

8

Quick question on the P400. How did you get the driver of esxi 6.5? After adding my card it is greyed out in the vphere web client.

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.