Plex Transcoder SEGFAULT (SIGSEGV) immediately on invocation

Plex Media Server
1

Server Version#: 1.32 through 1.35
Player Version#: (bundled with the above) 4.108 or whatever

A client – either in a Chrome web browser or on an Android device – requests to transcode the video+audio stream from original quality to something of lower quality. Plex Transcoder is spawned, begins attempting to transcode, and immediately segfaults.

I have replicated this via command line by copying the Plex Transcoder command as recorded in the logs:

EAE_ROOT=/tmp/pms-XXXXXXXX-XXXX-4XXX-XXXX-XXXXXXXXXXXX/EasyAudioEncoder \
  FFMPEG_EXTERNAL_LIBS='/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/xxxxxxx-4xxx-linux-x86_64/' \
  X_PLEX_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
  "/usr/lib/plexmediaserver/Plex Transcoder" \
  -codec:0 h264 -codec:1 eac3_eae -eae_prefix:1 xxxxxxxxxxxxxxxxxxxxxxxxx \
  -analyzeduration 20000000 -probesize 20000000 -i "/path/to/media/to/transcode.mkv" \
  -filter_complex "[0:0]scale=w=720:h=404:force_divisible_by=4[0];[0]format=pix_fmts=yuv420p|nv12[1]" -map "[1]" \
  -codec:0 libx264 -crf:0 17 -maxrate:0 2613k -bufsize:0 5226k \
  -r:0 23.975999999999999 -preset:0 veryfast \
  -x264opts:0 subme=6:me_range=4:rc_lookahead=10:me=hex:8x8dct=1 \
  -force_key_frames:0 "expr:gte(t,n_forced*8)" \
  -filter_complex "[0:1] aresample=async=1:ochl='stereo':rematrix_maxval=0.000000dB:osr=48000[2]" \
  -map "[2]" -metadata:s:1 language=eng -codec:1 aac -b:1 232k -f dash \
  -seg_duration 8 -dash_segment_type mp4 \
  -init_seg_name 'init-stream$RepresentationID$.m4s' \
  -media_seg_name 'chunk-stream$RepresentationID$-$Number%05d$.m4s' \
  -window_size 5 -delete_removed false -skip_to_segment 1 -time_delta 0.0625 \
  -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 dash \
  -start_at_zero -copyts -vsync cfr -y -nostats -loglevel debug -loglevel_plex debug

The above segfaults immediately. Enabling core dumps and loading via GDB reveals pretty much nothing in terms of a stack trace.

Running on a 2x Intel(R) Xeon(R) Platinum 8160 within ESXi. Please advise how to further debug this crash.

2
  1. Need DEBUG logs please to look further. A single fragment is insufficient
  2. Given it fails on multiple versions, I suspect a system issue (which the DEBUG logs should show)

Please create one time time, with DEBUG logging enabled.
Download the Logs ZIP file

Attach that ZIP
(I will open a PM to you if needed)

3

Attached.

Plex Media Server Logs_2023-08-31_16-44-44.zip (45.3 KB)

4

Is there any progress on this or further steps I can take to debug?

5

There was something important about the fact this was running on an instance of Ubuntu Server. The same setup on the same server but with Ubuntu Desktop works. I eventually figured this out by running the Transcoder directly with verbose logging and checking dmesg output.

Note that I’ve gotten Plex to run on Ubuntu Server fine within headless VirtualBox for years but clearly something different happens with the ESXi hypervisor.

Edit: The above is all incorrect – this had nothing to do with Desktop versus Server. It’s a Transcoder option issue.

6

I’m sorry. I wasn’t notified of your post until just now.

In looking at your logs, I see you only proved the current ‘Plex Media Server.log’

The one of most interest is the one which captures the crash (the ‘.1’)

Might it be possible to get the full set of logs ?

7

I have already deleted that VM instance, since it wasn’t working.

However, this was the process used to capture the above zip file:

  1. Shut down the plexmediaserver service.
  2. Deleted all files in the Logs directory.
  3. Started the plexmediaserver service again.
  4. Reproduced the issue.
  5. Used the Plex web UI to download the logs as a zip file.

This zip does contain just Plex Media Server.log, but the crash is indeed contained within it. Line 1115 is about where the media is started to play at original quality. Line 1563 is about where the request for a lower bitrate is begun and line 1611 is where the segfault is indicated. Here are lines 1611-1617:

Aug 31, 2023 16:44:31.615 [139645816347448] DEBUG - Jobs: '/usr/lib/plexmediaserver/Plex Transcoder' exit code for process 3307762 is -11 (signal: Segmentation fault)
Aug 31, 2023 16:44:31.615 [139645715585848] DEBUG - Streaming Resource: Terminating session 0x7f01cdafd938:fexcm0mkpafo613hcider89v which is using transcoder slot.  Used slots is now 0
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Killing job.
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Signalling job ID 3307762 with 9
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Job was already killed, not killing again.
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Stopping transcode session fssydvb929ngan63cxzmapij
Aug 31, 2023 16:44:31.615 [139645715585848] DEBUG - Streaming Resource: Terminated session 0x7f01cdafd938:fexcm0mkpafo613hcider89v with reason Conversion failed. The transcoder exited due to an error.
8

I see it. THANK YOU (tired eyes here)

I see it failing the ‘fast seek’ step.
This is where it restarts, changes transcode parameters and seeks back into the file to continue.

Aug 31, 2023 16:44:31.125 [139645733210936] DEBUG - [Req#1a7/Transcode/Req#1a8] We're going to try to auto-select an audio stream for account 1.
Aug 31, 2023 16:44:31.125 [139645733210936] DEBUG - [Req#1a7/Transcode/Req#1a8] Selecting best audio stream for part ID 4196 (language: )
Aug 31, 2023 16:44:31.125 [139645733210936] DEBUG - [Req#1a7/Transcode/Req#1a8] Audio Stream: 26146, Subtitle Stream: -1
Aug 31, 2023 16:44:31.126 [139645733210936] DEBUG - [Req#1a7/Transcode] Found session GUID of fssydvb929ngan63cxzmapij in session start.
Aug 31, 2023 16:44:31.127 [139645733210936] DEBUG - [Req#1a7/Transcode] Cleaning directory for session fssydvb929ngan63cxzmapij ()
Aug 31, 2023 16:44:31.127 [139645733210936] DEBUG - [Req#1a7/Transcode] Starting a transcode session fssydvb929ngan63cxzmapij at offset -1.0 (state=3)
Aug 31, 2023 16:44:31.128 [139645733210936] DEBUG - [Req#1a7/Transcode/JobRunner] In directory: "/tmp/Transcode/Sessions/plex-transcode-fssydvb929ngan63cxzmapij-084de49b-29cf-4515-9c04-2f1b9689716d"
Aug 31, 2023 16:44:31.128 [139645733210936] DEBUG - [Req#1a7/Transcode/JobRunner] Jobs: Starting child process with pid 3307762
Aug 31, 2023 16:44:31.137 [139645694491448] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=startup (6 live) #19c Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.137 [139645811747640] DEBUG - Completed: [127.0.0.1:59462] 204 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=startup (6 live) #19c 0ms 203 bytes (pipelined: 1) (range: bytes=0-) 
Aug 31, 2023 16:44:31.141 [139645719804728] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=startup (6 live) #1ab Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.141 [139645813857080] DEBUG - Completed: [127.0.0.1:59462] 204 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=startup (6 live) #1ab 0ms 203 bytes (pipelined: 2) (range: bytes=0-) 
Aug 31, 2023 16:44:31.142 [139645769710392] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=opening (6 live) #1ac Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.142 [139645813857080] DEBUG - Completed: [127.0.0.1:59462] 204 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=opening (6 live) #1ac 0ms 203 bytes (pipelined: 3) (range: bytes=0-) 
Aug 31, 2023 16:44:31.173 [139645696600888] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=opened (6 live) #1ad Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.173 [139645811747640] DEBUG - Completed: [127.0.0.1:59462] 204 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?status=opened (6 live) #1ad 0ms 203 bytes (pipelined: 4) (range: bytes=0-) 
Aug 31, 2023 16:44:31.174 [139645722352440] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/stream?index=0&id=0&codec=h264&type=video (6 live) #1ae Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.174 [139645811747640] DEBUG - Completed: [127.0.0.1:59462] 200 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/stream?index=0&id=0&codec=h264&type=video (6 live) #1ae 0ms 195 bytes (pipelined: 5) (range: bytes=0-) 
Aug 31, 2023 16:44:31.174 [139645694491448] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/stream?index=1&id=0&codec=eac3&type=audio (6 live) #1af Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.175 [139645811747640] DEBUG - Completed: [127.0.0.1:59462] 200 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/stream?index=1&id=0&codec=eac3&type=audio (6 live) #1af 0ms 195 bytes (pipelined: 6) (range: bytes=0-) 
Aug 31, 2023 16:44:31.221 [139645719804728] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=0&id=0&codec=h264&type=video&profile=High&width=1920&height=1080&interlaced=0&sar=1:1&level=40&frameRate=23.976&disp_default=1 (6 live) #1b0 Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.222 [139645813857080] DEBUG - Completed: [127.0.0.1:59462] 200 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=0&id=0&codec=h264&type=video&profile=High&width=1920&height=1080&interlaced=0&sar=1:1&level=40&frameRate=23.976&disp_default=1 (6 live) #1b0 0ms 195 bytes (pipelined: 7) (range: bytes=0-) 
Aug 31, 2023 16:44:31.222 [139645769710392] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=1&id=0&codec=eac3&type=audio&language=eng&channels=6&layout=5.1%28side%29&sampleRate=48000&disp_default=1 (6 live) #1b1 Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.223 [139645813857080] DEBUG - Completed: [127.0.0.1:59462] 200 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=1&id=0&codec=eac3&type=audio&language=eng&channels=6&layout=5.1%28side%29&sampleRate=48000&disp_default=1 (6 live) #1b1 0ms 195 bytes (pipelined: 8) (range: bytes=0-) 
Aug 31, 2023 16:44:31.223 [139645696600888] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=2&id=0&codec=subrip&type=subtitle&language=eng (6 live) #1b2 Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.223 [139645813857080] DEBUG - Completed: [127.0.0.1:59462] 200 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=2&id=0&codec=subrip&type=subtitle&language=eng (6 live) #1b2 0ms 195 bytes (pipelined: 9) (range: bytes=0-) 
Aug 31, 2023 16:44:31.223 [139645722352440] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=3&id=0&codec=subrip&type=subtitle&language=eng&disp_hearing_impaired=1 (6 live) #1b3 Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.224 [139645813857080] DEBUG - Completed: [127.0.0.1:59462] 200 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress/streamDetail?index=3&id=0&codec=subrip&type=subtitle&language=eng&disp_hearing_impaired=1 (6 live) #1b3 0ms 195 bytes (pipelined: 10) (range: bytes=0-) 
Aug 31, 2023 16:44:31.226 [139645694491448] DEBUG - Request: [127.0.0.1:59462 (Loopback)] PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?duration=2342.176000 (6 live) #1b4 Token () (range: bytes=0-) 
Aug 31, 2023 16:44:31.226 [139645733210936] DEBUG - [Req#1a7/Transcode] Started session successfully: fssydvb929ngan63cxzmapij
Aug 31, 2023 16:44:31.226 [139645811747640] DEBUG - Completed: [127.0.0.1:59462] 204 PUT /video/:/transcode/session/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d/progress?duration=2342.176000 (6 live) #1b4 0ms 203 bytes (pipelined: 11) (range: bytes=0-) 
Aug 31, 2023 16:44:31.227 [139645813857080] DEBUG - Completed: [192.168.1.3:51234] 200 GET /video/:/transcode/universal/start.mpd?hasMDE=1&path=%2Flibrary%2Fmetadata%2F4178&mediaIndex=0&partIndex=0&protocol=dash&fastSeek=1&directPlay=0&directStream=1&subtitleSize=100&audioBoost=100&location=lan&maxVideoBitrate=2000&addDebugOverlay=0&autoAdjustQuality=0&directStreamAudio=1&mediaBufferSize=102400&session=fssydvb929ngan63cxzmapij&subtitles=burn&Accept-Language=en (6 live) #1a7 GZIP 112ms 952 bytes (pipelined: 49)
Aug 31, 2023 16:44:31.232 [139645719804728] DEBUG - Request: [192.168.1.3:51234 (Allowed Network (Subnet))] GET /status/sessions (6 live) #1b8 GZIP
Aug 31, 2023 16:44:31.232 [139645719804728] DEBUG - [Req#1b8] [Now] Adding 1 sessions.
Aug 31, 2023 16:44:31.234 [139645811747640] DEBUG - Completed: [192.168.1.3:51234] 200 GET /status/sessions (6 live) #1b8 GZIP 2ms 2921 bytes (pipelined: 50)
Aug 31, 2023 16:44:31.248 [139645769710392] DEBUG - Request: [192.168.1.3:51234 (Allowed Network (Subnet))] GET /video/:/transcode/universal/session/fssydvb929ngan63cxzmapij/1/header (6 live) #1b9 GZIP
Aug 31, 2023 16:44:31.248 [139645696600888] DEBUG - Request: [192.168.1.3:51272 (Allowed Network (Subnet))] GET /video/:/transcode/universal/session/fssydvb929ngan63cxzmapij/0/header (6 live) #179 GZIP
Aug 31, 2023 16:44:31.576 [139645722352440] DEBUG - [Req#1b6/Transcode/fssydvb929ngan63cxzmapij/084de49b-29cf-4515-9c04-2f1b9689716d] Transcoder: session fssydvb929ngan63cxzmapij indicated fallback to software decoding
Aug 31, 2023 16:44:31.615 [139645816347448] DEBUG - Jobs: '/usr/lib/plexmediaserver/Plex Transcoder' exit code for process 3307762 is -11 (signal: Segmentation fault)
Aug 31, 2023 16:44:31.615 [139645715585848] DEBUG - Streaming Resource: Terminating session 0x7f01cdafd938:fexcm0mkpafo613hcider89v which is using transcoder slot.  Used slots is now 0
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Killing job.
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Signalling job ID 3307762 with 9
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Job was already killed, not killing again.
Aug 31, 2023 16:44:31.615 [139645748067128] DEBUG - Stopping transcode session fssydvb929ngan63cxzmapij
Aug 31, 2023 16:44:31.615 [139645715585848] DEBUG - Streaming Resource: Terminated session 0x7f01cdafd938:fexcm0mkpafo613hcider89v with reason Conversion failed. The transcoder exited due to an error.
Aug 31, 2023 16:44:31.615 [139645715585848] DEBUG - Cleaning directory for session fssydvb929ngan63cxzmapij (/tmp/Transcode/Sessions/plex-transcode-fssydvb929ngan63cxzmapij-084de49b-29cf-4515-9c04-2f1b9689716d)
Aug 31, 2023 16:44:31.617 [139645748067128] DEBUG - Transcoder: Cleaning old transcode directories.
Aug 31, 2023 16:44:31.617 [139645748067128] DEBUG - Whacked session fssydvb929ngan63cxzmapij, 0 remaining.
Aug 31, 2023 16:44:31.621 [139645694491448] DEBUG - Request: [192.168.1.3:51250 (Allowed Network (Subnet))] GET /status/sessions (5 live) #14d GZIP
Aug 31, 2023 16:44:31.621 [139645694491448] DEBUG - [Req#14d] [Now] Adding 1 sessions.

I will write this up and start testing it.

Confirm please the type of media (H264 or H265 ) ?
Also, looks like an Android / Nvidia Shield player device ?

9

Media info as pertinent:

Bitrate 7934 kbps
Width 1920
Height 1080
Container MKV
Video Frame Rate 24p
Video Profile high

Codec H264
Bitrate 7934 kbps
Bit Depth 8
Chroma Location left
Chroma Subsampling 4:2:0
Coded Height 1088
Coded Width 1920
Color Primaries bt709
Color Range tv
Color Space bt709
Color Trc bt709
Frame Rate 23.976 fps
Height 1080
Level 4.0
Profile high
Ref Frames 4
Scan Type progressive
Width 1920
Display Title 1080p (H.264)
Extended Display Title 1080p (H.264)

Codec EAC3
Channels 6
Bitrate 640 kbps
Language English
Language Tag en
Audio Channel Layout 5.1(side)
Sampling Rate 48000 Hz
Display Title English (EAC3 5.1)
Extended Display Title English (EAC3 5.1)
Forced true

Codec SRT
Language English
Language Tag en
Title Forced
Display Title English Forced (SRT)
Extended Display Title Forced (English SRT)

Codec SRT
Language English
Language Tag en
Display Title English (SRT)
Extended Display Title English (SRT)

Codec SRT
Language English
Language Tag en
Hearing Impaired true
Title SDH
Display Title English SDH (SRT)
Extended Display Title SDH (English SRT)

Running on an Ubuntu Server 22.04 virtual machine on VMware ESXi 8.0U1. I really think the fact no display was attached had something to do with it (since it works on the same version of Ubuntu Desktop). If I remember to, I’ll recreate the VM this weekend and see if I can dig up more information.

10

Would you be willing to test / try this?

We found build problems and corrected them.

Given what we’ve seen, it might even fix the segfault you’re seeing.

11

I don’t think it was HW transcoding related. Once I rebuilt with Ubuntu Desktop, I figured having a virtual display would at least let me get QSV or PCI passthrough for the GPU working. However, as it is running in a VM, QSV isn’t exposed and I couldn’t get PCI passthrough to work despite several attempts.

The VM is limited to software transcoding only. Still, I can try it out.

12

If you can try the basics (simple low-bitrate) stuff, that’ll confirm the build integrity.

It should not segfault either way.

We did find compilation problems in our build system.
That was at the root of all this. (not sure how long those flags were in there)

All I can say is we’re working again – starting to move forward

13

I figured out the problem by pure chance. Tested on both the latest release and the proposed release you asked me to test above. After installing on a fresh VM, I realized I didn’t quite match the settings I had on the first few installs of Plex, and I didn’t get a segfault. It has nothing to do with Server versus Desktop editions of Ubuntu. I knew the crash was due to libx264 based on the thread in gdb, but it would have been a lot easier to diagnose if the binaries weren’t stripped and I were able to get a decent stack trace.

After a lot of messing around with the settings, it boils down to this:

  • Transcoder quality set to “Automatic” – no segfault
  • Transcoder quality set to “Make my CPU hurt” – segfault

At the simplest level, it’s the subme=2 vs subme=6 change to x264opts that makes the difference:

root@plex:~# /usr/lib/plexmediaserver/Plex\ Media\ Server --version
v1.32.6.7515-873995547

root@plex:~# mediainfo convert.mp4 
General
Complete name                            : convert.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 757 KiB
Duration                                 : 5 s 5 ms
Overall bit rate                         : 1 239 kb/s
Writing application                      : Lavf59.20.101

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 5 s 5 ms
Bit rate                                 : 1 236 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.025
Stream size                              : 755 KiB (100%)
Writing library                          : x264 core 161
Encoding settings                        : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=2 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=4 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0 / threads=24 / lookahead_threads=8 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=1 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=20 / rc=crf / mbtree=1 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : avcC

root@plex:~# EAE_ROOT=/tmp/pms-91fd6af2-348a-4abf-aa86-627b2f0e8cc0/EasyAudioEncoder FFMPEG_EXTERNAL_LIBS='/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/a31c019-4611-linux-x86_64/' "/usr/lib/plexmediaserver/Plex Transcoder" -codec:0 h264 -i convert.mp4 -codec:0 libx264 -preset:0 veryfast -x264opts:0 subme=6:me_range=4:rc_lookahead=20:me=hex -f mp4 convert2.mp4 &>/dev/null; echo $?  # Make my CPU hurt
Segmentation fault (core dumped)
139

root@plex:~# EAE_ROOT=/tmp/pms-91fd6af2-348a-4abf-aa86-627b2f0e8cc0/EasyAudioEncoder FFMPEG_EXTERNAL_LIBS='/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/a31c019-4611-linux-x86_64/' "/usr/lib/plexmediaserver/Plex Transcoder" -codec:0 h264 -i convert.mp4 -codec:0 libx264 -preset:0 veryfast -x264opts:0 subme=2:me_range=4:rc_lookahead=20:me=hex -f mp4 convert2.mp4 &>/dev/null; echo $?  # Automatic
0
14

if you had lldb , and ran it in the debugger, you could get a module stack traceback but that’s how they want it.

The symbols are removed (stored separately)

15

Turns out you really couldn’t, due to how libx264 initializes its context and calls threads.

Oh, I know. They’ve stripped it, even though it is GPL, so that makes no sense. But if someone really cared, they could reverse-engineer liblibx264_encoder.so, bindiff it to the open-source copy (which doesn’t suffer from this bug, at least the one in the Ubuntu repo doesn’t) to, perhaps, figure out that the crash happens in x264_8_cabac_block_residual_rd_internal_avx512:

Thread 34 "Plex Transcoder" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 4135590]                      
0x00007ffff6291aae in x264_8_cabac_block_residual_rd_internal_avx512 (param_1=140737189778240, param_2=0) from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/a31c019-4611-linux-x86_64/liblibx264_encoder.so

(gdb) backtrace
#0  0x00007ffff6291aae in x264_8_cabac_block_residual_rd_internal_avx512 (param_1=140737189778240, param_2=0) from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/a31c019-4611-linux-x86_64/liblibx264_encoder.so
#1  0x0000000000000000 in ?? ()

(gdb) disassemble
Dump of assembler code for function x264_8_cabac_block_residual_rd_internal_avx512:
   0x00007ffff6291a70 <+0>:     push   %rbx
   0x00007ffff6291a71 <+1>:     push   %rbp
   0x00007ffff6291a72 <+2>:     push   %r14
   0x00007ffff6291a74 <+4>:     push   %r15
   0x00007ffff6291a76 <+6>:     sub    $0x38,%rsp
   0x00007ffff6291a7a <+10>:    lea    -0x1a21(%rip),%r15        # 0x7ffff6290060 <x264_8_cabac_encode_decision_asm>
   0x00007ffff6291a81 <+17>:    shl    $0x4,%esi
   0x00007ffff6291a84 <+20>:    add    %edx,%esi
   0x00007ffff6291a86 <+22>:    movzwl -0x2f460(%r15,%rsi,2),%r9d
   0x00007ffff6291a8f <+31>:    movzwl -0x2f420(%r15,%rsi,2),%r10d
   0x00007ffff6291a98 <+40>:    movzwl -0x2f3e0(%r15,%rdx,2),%r11d
   0x00007ffff6291aa1 <+49>:    mov    %rdi,%r8
   0x00007ffff6291aa4 <+52>:    and    $0xfffffffffffffffd,%r8
   0x00007ffff6291aa8 <+56>:    vpabsw (%r8),%ymm16
=> 0x00007ffff6291aae <+62>:    vmovdqa32 %ymm16,(%rsp)
   0x00007ffff6291ab5 <+69>:    xor    %rdi,%r8
   0x00007ffff6291ab8 <+72>:    add    %rsp,%r8
   0x00007ffff6291abb <+75>:    lea    -0x3d9a0(%r15),%rsi
   0x00007ffff6291ac2 <+82>:    movslq (%rsi,%rdx,4),%rax
   0x00007ffff6291ac6 <+86>:    add    %rsi,%rax
   0x00007ffff6291ac9 <+89>:    call   *%rax
   0x00007ffff6291acb <+91>:    mov    0x40(%rcx),%edi
   0x00007ffff6291ace <+94>:    add    $0x44,%rcx
   0x00007ffff6291ad2 <+98>:    add    %rcx,%r9

One could also perhaps then determine that to work around the bug, one could place the following option in their Preferences.xml file:

TranscoderH264OptionsOverride="asm=mmx2,sse2,sse2fast,ssse3,sse4.2,avx,fma3,bmi2,avx2"

Regardless, I hope this is filed as a bug.

16

I will take this to the transcoder engineer.

Looking at it is confusing.

Please confirm you’re using Software transcoding here ?

17

Yes, this is software transcoding. The reason the vmovdqa32 instruction is getting a general protection fault is because it is supposed to be 32-byte aligned but is 16-byte aligned.

https://www.felixcloutier.com/x86/movdqa:vmovdqa32:vmovdqa64

When the source or destination operand is a memory operand, the operand must be aligned on a 16 (EVEX.128)/32(EVEX.256)/64(EVEX.512)-byte boundary or a general-protection exception (#GP) will be generated.

0x00007ffff6291aae <+62>:    62 e1 7d 28 7f 04 24    vmovdqa32 %ymm16,(%rsp)

Fairly certain this is the EVEX.256.66.0F.W0 7F /r VMOVDQA32 ymm2/m256 {k1}{z}, ymm1 variant.

Example of current code:

(gdb) disassemble /r $rip,+6
Dump of assembler code from 0x7ffff6291aae to 0x7ffff6291ab4:
=> 0x00007ffff6291aae <x264_8_cabac_block_residual_rd_internal_avx512+62>:      62 e1 7d 28 7f 04 24    vmovdqa32 %ymm16,(%rsp)
End of assembler dump.
(gdb) info reg $rsp
rsp            0x7fffee3401d0      0x7fffee3401d0
(gdb) stepi

Thread 34 "Plex Transcoder" received signal SIGSEGV, Segmentation fault.
0x00007ffff6291aae in x264_8_cabac_block_residual_rd_internal_avx512 (param_1=140737189774144, param_2=0) from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/a31c019-4611-linux-x86_64/liblibx264_encoder.so

Changing to a 32-byte boundary:

(gdb) disassemble /r $rip,+6
Dump of assembler code from 0x7ffff6291aae to 0x7ffff6291ab4:
=> 0x00007ffff6291aae <x264_8_cabac_block_residual_rd_internal_avx512+62>:      62 e1 7d 28 7f 04 24    vmovdqa32 %ymm16,(%rsp)
End of assembler dump.
(gdb) info reg $rsp
rsp            0x7fffee3411d0      0x7fffee3411d0
(gdb) set $rsp -= 0x10
(gdb) info reg $rsp
rsp            0x7fffee3411c0      0x7fffee3411c0
(gdb) stepi
0x00007ffff6291ab5 in x264_8_cabac_block_residual_rd_internal_avx512 (param_1=140737189778240, param_2=0) from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/a31c019-4611-linux-x86_64/liblibx264_encoder.so
(gdb) # No segfault
18

Thank you.

Been a bit since I had to second guess the compiler and the validity of FFMPEG core code.

Does this seem to be a structure (source code) problem or a compilation problem?

To me , it seems like, as you show. there’s a 16 byte alignment when 32-byte is needed which is an error in how the memory is allocated.

Can you cut a sample of the file (if needed) or provide some instructions on how we can replicate it under the debugger so it can be properly fixed ?

19

I have been using the attached file to consistently test this bug.

On the machine that encounters this 100% of the time, I also installed ffmpeg from the standard Ubuntu repositories:

$ ffmpeg -version
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 70.100 / 56. 70.100
libavcodec     58.134.100 / 58.134.100
libavformat    58. 76.100 / 58. 76.100
libavdevice    58. 13.100 / 58. 13.100
libavfilter     7.110.100 /  7.110.100
libswscale      5.  9.100 /  5.  9.100
libswresample   3.  9.100 /  3.  9.100
libpostproc    55.  9.100 / 55.  9.100

The version of libx264 that was installed with that is /usr/lib/x86_64-linux-gnu/libx264.so.163.

Normally, I’d say this is a compiler bug, since the compiler should know that if it is going to use a vmovdqa32 instruction with %rsp as the destination, then %rsp needs to be 32-byte aligned.

However, the disassembly of x264_8_cabac_block_residual_rd_internal_avx512 in libx264.so.163 (core 163) is identical to the core 161 version in Plex Transcoder, i.e., there is no logic for the stack to be 32-byte aligned inside of x264_8_cabac_block_residual_rd_internal_avx512. That logic must be elsewhere and that’s a bit too deep down the rabbit hole for me to chase right now.

sample.zip (755.1 KB)

20

Are there any updates on replicating this issue?