Hardware transcoding stopped working with QuickSync (VAAPI) on an Intel iGPU (Intel NUC)

ddrones · June 3, 2023, 11:58pm

Server Version#: 1.32.2.7100

I am not sure which server version broke the hw transcoding on my Intel NUC iGPU but I suspect it started a few versions ago (noisy fan because of software transcoding).

To give a bit more context:

HW transcoding was working before
Suddenly it did not
Was using Ubuntu 20.04, I though that plex did receive an update requiring more update date packages
Upgraded to Ubuntu 22.04 - still not working
Everything fine with users/groups/access to /dev/dri/renderD128
manual run of ffmpeg with vaapi is working fine ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i /path/to/file.mkv -f null -
Updated to latest stable all intel drivers/packages (compute, media and display runtime from here which gave a nice boost of hw capability: got 6x in ffmpeg with ubuntu packages, ~8x with intel latest) - still not working in plex

VAAPI not working in plex

The journey started with debug/verbose logs:

Jun 03, 2023 21:17:52.436 [140500955691832] DEBUG - [Req#15ba/Transcode] Codecs: testing hevc (decoder) with hwdevice vaapi
Jun 03, 2023 21:17:52.436 [140500955691832] VERBOSE - [Req#15ba/Transcode] [FFMPEG] - Rescanning for external libs: '/data/plex/Plex\ Media\ Server/Codecs/8217c1c-4565-linux-x86_64/'
Jun 03, 2023 21:17:52.436 [140500955691832] DEBUG - [Req#15ba/Transcode] Codecs: hardware transcoding: testing API vaapi
Jun 03, 2023 21:17:52.437 [140500955691832] VERBOSE - [Req#15ba/Transcode] [FFMPEG] - Trying to use DRM render node for device 0.
Jun 03, 2023 21:17:52.437 [140500955691832] VERBOSE - [Req#15ba/Transcode] [FFMPEG] - libva: VA-API version 1.18.0
Jun 03, 2023 21:17:52.437 [140500955691832] VERBOSE - [Req#15ba/Transcode] [FFMPEG] - libva: User requested driver 'i965'
Jun 03, 2023 21:17:52.437 [140500955691832] VERBOSE - [Req#15ba/Transcode] [FFMPEG] - libva: Trying to open /data/plex/Plex Media Server/Cache/va-dri-linux-x86_64/i965_drv_video.so
Jun 03, 2023 21:17:52.437 [140500955691832] VERBOSE - [Req#15ba/Transcode] [FFMPEG] - libva: va_openDriver() returns -1
Jun 03, 2023 21:17:52.437 [140500955691832] ERROR - [Req#15ba/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jun 03, 2023 21:17:52.437 [140500955691832] DEBUG - [Req#15ba/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error

It seems that plex is trying to open /data/plex/Plex Media Server/Cache/va-dri-linux-x86_64/i965_drv_video.so to use vaapi which fails and therefor fallback to software decoding.

I/O error is actually nothing fancy:

d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$ ls -lha
total 8.0K
drwxr-xr-x 1 plex plex 66 Jun 3 21:52 .
drwxr-xr-x 1 plex plex 692 Jun 3 21:43 ..
lrwxrwxrwx 1 plex plex 77 Jun 3 21:52 iHD_drv_video.so -> '/data/plex/Plex Media Server/Drivers/imd-74-linux-x86_64/dri/iHD_drv_video.so'
d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$

The file simply does not exist for the i965 driver. Only the lib for the iHD driver does.

iHD vs i965

Not quite sure about the difference between these 2 Intels drivers but it got me thinking…

d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$ sudo vainfo
Trying display: wayland
error: XDG_RUNTIME_DIR not set in the environment.
Trying display: x11
error: can't connect to X server!
Trying display: drm
libva info: VA-API version 1.18.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_18
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.18 (libva 2.18.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 23.1.6 ()
vainfo: Supported profile and entrypoints
      VAProfileNone : VAEntrypointVideoProc
      VAProfileNone : VAEntrypointStats
      VAProfileMPEG2Simple : VAEntrypointVLD
      VAProfileMPEG2Simple : VAEntrypointEncSlice
      VAProfileMPEG2Main : VAEntrypointVLD
      VAProfileMPEG2Main : VAEntrypointEncSlice
      VAProfileH264Main : VAEntrypointVLD
      VAProfileH264Main : VAEntrypointEncSlice
      VAProfileH264Main : VAEntrypointFEI
      VAProfileH264Main : VAEntrypointEncSliceLP
      VAProfileH264High : VAEntrypointVLD
      VAProfileH264High : VAEntrypointEncSlice
      VAProfileH264High : VAEntrypointFEI
      VAProfileH264High : VAEntrypointEncSliceLP
      VAProfileVC1Simple : VAEntrypointVLD
      VAProfileVC1Main : VAEntrypointVLD
      VAProfileVC1Advanced : VAEntrypointVLD
      VAProfileJPEGBaseline : VAEntrypointVLD
      VAProfileJPEGBaseline : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3 : VAEntrypointVLD
      VAProfileVP8Version0_3 : VAEntrypointEncSlice
      VAProfileHEVCMain : VAEntrypointVLD
      VAProfileHEVCMain : VAEntrypointEncSlice
      VAProfileHEVCMain : VAEntrypointFEI
      VAProfileHEVCMain10 : VAEntrypointVLD
      VAProfileHEVCMain10 : VAEntrypointEncSlice
      VAProfileVP9Profile0 : VAEntrypointVLD
      VAProfileVP9Profile2 : VAEntrypointVLD
d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$

My Intel NUC iGPU seems to use the iHD driver, not the i965.

Poking around

→ Real i965 lib

Found out that Intel did provided the file from its package, tried to symlink it where Plex was expecting to find it:

d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$ sudo ln -s /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so i965_drv_video.so

But it did not work. VAAPI successfully initialized but reported that no profile (decoding/encoding I assumed) was found and therefor a software fallback occured. I removed that symlink.

→ Real iHD lib

I tried to trick plex into using the real iHD lib from Intel but clearly lib versions/headers did not match and therefor did not work.

d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$ sudo ln -s /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so i965_drv_video.so

resulted in

Jun 03, 2023 21:49:25.103 [140169198832440] VERBOSE - [Req#e80/Transcode/rtolksxx1fh0i35oa0zdmj6e] [FFMPEG] - libva: Trying to open /data/plex/Plex Media Server/Cache/va-dri-linux-x86_64/i965_drv_video.so
Jun 03, 2023 21:49:25.105 [140169198832440] ERROR - [Req#e80/Transcode/rtolksxx1fh0i35oa0zdmj6e] [FFMPEG] - libva: dlopen of /data/plex/Plex Media Server/Cache/va-dri-linux-x86_64/i965_drv_video.so failed: Error relocating /usr/lib/x86_64-linux-gnu/libstdc++.so.6: __wmemmove_chk: symbol not found

→ Plex iHD lib

From the cache directory where plex was looking for the i965_drv_video.so there was a iHD_drv_video.so file pointing to /data/plex/Plex Media Server/Drivers/imd-74-linux-x86_64/dri/iHD_drv_video.so : this one, as bundled with plex, should align with plex transcoder binary… So I tried to trick Plex one last time:

d@nucsrv:/data/plex/Plex Media Server/Cache/va-dri-linux-x86_64$ sudo ln -s iHD_drv_video.so i965_drv_video.so

And it worked !

Jun 03, 2023 22:01:32.362 [139959308180280] DEBUG - [Req#6c3/Transcode] Codecs: testing hevc (decoder) with hwdevice vaapi
Jun 03, 2023 22:01:32.362 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - Rescanning for external libs: '/data/plex/Plex\ Media\ Server/Codecs/8217c1c-4565-linux-x86_64/'
Jun 03, 2023 22:01:32.363 [139959308180280] DEBUG - [Req#6c3/Transcode] Codecs: hardware transcoding: testing API vaapi
Jun 03, 2023 22:01:32.363 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - Trying to use DRM render node for device 0.
Jun 03, 2023 22:01:32.363 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - libva: VA-API version 1.18.0
Jun 03, 2023 22:01:32.363 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - libva: User requested driver 'i965'
Jun 03, 2023 22:01:32.363 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - libva: Trying to open /data/plex/Plex Media Server/Cache/va-dri-linux-x86_64/i965_drv_video.so
Jun 03, 2023 22:01:32.363 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - libva: Found init function __vaDriverInit_1_18
Jun 03, 2023 22:01:32.364 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - libva: va_openDriver() returns 0
Jun 03, 2023 22:01:32.364 [139959308180280] VERBOSE - [Req#6c3/Transcode] [FFMPEG] - Initialised VAAPI connection: version 1.18

Summary

Plex tries to load Plex Media Server/Cache/va-dri-linux-x86_64/i965_drv_video.so to enable vaapi but fails as the file does not exist. What is strange is the file Plex Media Server/Cache/va-dri-linux-x86_64/iHD_drv_video.so does exist and perfectly work with my NUC which seems to be logical as vainfo reports using iHD too and not i965.

The logs seem to indicate that the i965 is specifically set/called by plex (libva: User requested driver 'i965') which could be the issue here as plex currently runs on an iHD intel plateform (and provides/bundles its own iHD while not having its own i965 ¯\(ツ)/¯).

My workaround is simple: it feeds to plex the right file under the wrong name, wrong filename plex is actually trying to load:

$ cd <your_plex_base_path>/Plex Media Server/Cache/va-dri-linux-x86_64
$ sudo ln -s iHD_drv_video.so i965_drv_video.so

But I am pretty sure this should not be needed as Plex should try to load the correct file on its own and hece is a bug.

pshanew · June 4, 2023, 12:07am

Possibly a known issue.

If so, there’s an engineering build posted in that thread which you can test.

By the way, what processor is installed in your NUC?

ddrones · June 4, 2023, 12:26am

Good to know thank you, even if I will stick with my workaround for now to stay on stable builds.

For the record, my NUC CPU is a i7-8650U.

ChuckPa · June 4, 2023, 1:17am

May I see server DEBUG logs which capture the failure?

I have an i7-8809G CPU here.

ddrones · June 4, 2023, 8:56am

You won’t be able to see the issue (at least why it is failling) with only debug, verbose is required. The only relevant lines are the first I included in my post (about vaapi initialization) but I extracted a more complete log sample as requested.

I can not send it to you tho:

ChuckPa · June 4, 2023, 3:58pm

@ddrones

I have opened a PM for you to attach logs

In almost all cases, especially with these mature CPUs like ours, I can see what’s happening by examining the DEBUG logs and knowing the following information:

if VaapiDriver hidden pref is set.
If the server is running in a VM or container (Docker or LXC)

ddrones · June 4, 2023, 4:00pm

Running baremetal with systemd. Thanks for opening the DM, logs sent.

ddrones · June 4, 2023, 8:09pm

I found the responsible… Me, 3 years ago:

It was burried within the Preference.xml file, in the middle of others options. Thanks ChuckPa for spotting its activation.

Why this 3 years old workaround was working until now and not anymore : no idea.

But by removing this old workaround and the symlink workaround from my 1st post (which was a workaround of the first workaround in the end ) everything is back to normal, hw transcoding working, no workarounds involved

ChuckPa · June 4, 2023, 8:12pm

The i965 driver is only downloaded for those CPUs which need it

That’s why it fails. Your CPU doesn’t need the i965.

I’m sorry I looked at your prefs and missed it too. I must need new

tollgate · June 4, 2023, 8:24pm

I had a similar problem to this. One day hw accel was working. Next time I checked a couple of days later it wasn’t. Found there was a bad symlink in the cache directory va-dri-linux-x86_64. An extra “dri” in the path. Arch linux on an i3-3770. Not the same issue you had but your issue helped me troubleshoot my issue and fix it.

ddrones · June 4, 2023, 8:27pm

It does not have to be same as long as it helps others

Congrats on finding your bug

pommesmatte · June 7, 2023, 8:18pm

i965 is needed for getting subtitle burning working, as that is broken with iHD.

ChuckPa · June 7, 2023, 8:53pm

The iHD doesn’t do any subtitle burning. It’s all done with the CPU.

As posted in your other thread – The ApolloLake update should fix all this for everyone.

pommesmatte · June 7, 2023, 9:15pm

Yes, but its not working with the iHD driver, while it worked flawless with the i965 driver until 1.32.2 broke that also.

What exactly that fix is supposed to fix?

ddrones · June 7, 2023, 9:15pm

But it does break zero copy transcoding (hw decoding → hw encoding, directly within quicksync) which decreases performance even with hardware transcoding (hw decoding → back to userland to modify image with CPU → back to hw encoding).

I avoid subtitle burning at all costs

ChuckPa · June 7, 2023, 11:30pm

May I see logs which capture this ?

I have an 8809G (the same GPU as you with the addition of an AMD Radeon onboard)

Also, we’re very much going to toss build 7100 out the Window. There was a great deal of regression in there.

With the logs, I’m going to retest using my files on my i7-8809g with the same PMS version I see in the logs.

If I can replicate the fault – Great.
If I cannot replicate the fault – more investigation is needed.

ChuckPa · June 7, 2023, 11:32pm

By definition, When subtitles are involved, Zero copy IS broken.

The process goes from

Decode each frame to a raw binary image (like a bitmap)
Use the CPU to write (burn) the subtitle into the bitmap image
Encode that standalone frame and send it out.

Zero-copy means Input → Output in one shot.

TheBigChay · June 11, 2023, 2:02am

This effected my instance as well with an Intel i5-8500T. I originally tried upgrading kernel from 5.4.0 to latest with no luck. I downgraded to PMS to 1.32.1.6999 and hw transcoding immediately started working again.

pommesmatte · June 12, 2023, 7:21am

Yes, but we are back at square one, because you never could explain why this process won’t even keep up realtime when using iHD driver (making subtitle burn-in infeasible), but using i965 its about ten times faster and thus even making concurrent subtitle burn-ins possible.

So its very unfortunate that i965 got broken and I really hope that fixes for that are included in the upcoming releases.

mitzone · June 12, 2023, 10:18pm

Hello,
Just noticed hardware transcoding stopped working on my server as well. (was using latest version 1.32.4.7195).

Docker install : plexinc/pms-docker
CPU is Intel J3455

Noticed this kind of errors in the logs :

Jun 13, 2023 00:42:32.494 [140482633325368] DEBUG - [Req#5d6/Transcode] TranscodeUniversalRequest: using augmented profile Firefox
Jun 13, 2023 00:42:32.494 [140482633325368] DEBUG - [Req#5d6/Transcode] Downloading document http://127.0.0.1:32400/library/metadata/498574?includeBandwidths=1&offset=-1&X-Plex-Incomplete-Segments=1&X-Plex-Session-Identifier=2eyscik1jbnx8q8ni6eqaaza
Jun 13, 2023 00:42:32.508 [140482633325368] DEBUG - [Req#5d6/Transcode/Req#5e2] We're going to try to auto-select an audio stream for account 1.
Jun 13, 2023 00:42:32.508 [140482633325368] DEBUG - [Req#5d6/Transcode/Req#5e2] Selecting best audio stream for part ID 568785 (language: en)
Jun 13, 2023 00:42:32.508 [140482633325368] DEBUG - [Req#5d6/Transcode/Req#5e2] We're going to try to auto-select a subtitle.
Jun 13, 2023 00:42:32.508 [140482633325368] DEBUG - [Req#5d6/Transcode/Req#5e2] Selected a foreign audio track: en
Jun 13, 2023 00:42:32.508 [140482633325368] DEBUG - [Req#5d6/Transcode/Req#5e2] Audio Stream: 443032, Subtitle Stream: -1
Jun 13, 2023 00:42:32.512 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: testing h264_nvenc (encoder)
Jun 13, 2023 00:42:32.513 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: hardware transcoding: testing API nvenc
Jun 13, 2023 00:42:32.513 [140482633325368] ERROR - [Req#5d6/Transcode] [FFMPEG] - Cannot load libcuda.so.1
Jun 13, 2023 00:42:32.513 [140482633325368] ERROR - [Req#5d6/Transcode] [FFMPEG] - Could not dynamically load CUDA
Jun 13, 2023 00:42:32.513 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: Operation not permitted
Jun 13, 2023 00:42:32.513 [140482633325368] DEBUG - [Req#5d6/Transcode] Could not create hardware context for h264_nvenc
Jun 13, 2023 00:42:32.513 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: testing h264 (decoder) with hwdevice vaapi
Jun 13, 2023 00:42:32.514 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: hardware transcoding: testing API vaapi
Jun 13, 2023 00:42:32.536 [140482633325368] ERROR - [Req#5d6/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
Jun 13, 2023 00:42:32.537 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
Jun 13, 2023 00:42:32.537 [140482633325368] DEBUG - [Req#5d6/Transcode] Could not create hardware context for h264
Jun 13, 2023 00:42:32.537 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: testing h264 (decoder) with hwdevice nvdec
Jun 13, 2023 00:42:32.537 [140482633325368] DEBUG - [Req#5d6/Transcode] Codecs: hardware transcoding: testing API nvdec
Jun 13, 2023 00:42:32.538 [140482633325368] ERROR - [Req#5d6/Transcode] [FFMPEG] - Cannot load libcuda.so.1
Jun 13, 2023 00:42:32.538 [140482633325368] ERROR - [Req#5d6/Transcode] [FFMPEG] - Could not dynamically load CUDA

I started downgrading until I found that latest version where my hw transcoding works is :

plexmediaserver_1.32.1.6999-91e1e2e2c_amd64.deb

Thanks for looking into this.

Topic		Replies	Views
Latest build 7100 not transcoding on Intel QSV Desktops & Laptops server-linux , server-unraid	45	1146	September 11, 2023
Hardware Transcoding - Linux - artefacts General Discussions	79	794	January 7, 2020
Hardware transcoding broken on Intel NUC with J5005 CPU on Ubuntu 18.04 LTS Desktops & Laptops server-linux	69	9342	November 30, 2020
I5 9600k / Intel UHD 630 / Ubuntu 18.10 / Hardware Acceleration Not Working Desktops & Laptops server-linux	69	3923	November 25, 2019
Hardware Transcoding w/ Intel Quicksync not working Plex Media Server server-linux	29	510	October 30, 2020

Hardware transcoding stopped working with QuickSync (VAAPI) on an Intel iGPU (Intel NUC)

Related topics