Segfault in transcoder process with ~2 min left in video

Recently playback has been crashing with ~2 minutes left in files. Debug logs show the transcoder is crashing with a segmentation fault.

I’ve got a Synology DS918+ running DSM 6.2.3-25426
Plex Server Version#: 1.25.0.5282
Player Version#: Roku 6.9.12.7411-08dc87cc9-Plex

This seems to happen consistently with mkv files (h.264 / AAC 5.1 audio) with default transcoder settings. I’ve tried with hw acceleration disabled and results are the same.

Plex Media Server Logs_2021-11-30_21-33-51.zip (2.5 MB)

Attaching logs with debug turned on. Can anyone help figure out what’s going on? Thanks!

2 Likes

Example of the segfault happens at the following timestamp, in case this helps:
(Nov 30, 2021 21:26:50.011 [0x7feb40532b38] DEBUG - Jobs: ‘/volume1/@appstore/Plex Media Server/Plex Transcoder’ exit code for process 23674 is -11 (signal: Segmentation fault)

Anyone have an idea on how I can troubleshoot this? It’s really frustrating to have to watch the last few minutes of shows on a laptop. : )

I don’t have any answers, but the same thing is happening to me. I am running Plex on Ubuntu 20.04 and MKV or MP4 files with h.264 / AAC 5.1 audio segfault close to the end of the video when played back on my Roku Ultra. The same video when played back on my iPad or Windows PC plays fine. Any help would be greatly appreciated.

I am also having this happen with a TCL Roku TV. Doesn’t happen on my LG tv or iPhone. It is always right when there is 2 min left of whatever show or movie.

As a test, I re-encoded a few videos to be stereo instead of 5.1 and Plex didn’t segfault.

How big is the video ?

My brother has a TCL Roku TV and is running into this issue. The video is direct stream but the audio is being transcoded from AAC to EAC3 for whatever reason. The last 2 minutes of anything he tries to watch with the audio transcoding fails with this error message.

May I see some DEBUG logs ZIP files which capture this failure please ?

I was going to capture logs last night but of course nothing we watched caused it to crash. I’ll continue to monitor and as soon as it happens again I will upload them.

EDIT: Looks like 1.25.2.5319 was released 2 days ago…wondering if this potentially resolved the issue and is why I didn’t see the crash last night.

I am running 1.25.2.5319 and seeing this issue. Transcoding segfaults at the end of the file only.

This is what i’ve got when running under the official Docker container on 1.25.2.5319, and playing on a Roku TV and the standard (not preview) app

Dec 16, 2021 19:53:55.260 [0x14c59721ab38] DEBUG - [Transcode] Session c405880c-e42c-42de-8190-4ed963183bed-3139 (2) is throttling
Dec 16, 2021 19:53:55.261 [0x14c5938eab38] DEBUG - [Transcoder] Throttle - Going into sloth mode.
Dec 16, 2021 19:53:58.631 [0x14c598886b38] DEBUG - Request: [127.0.0.1:57278 (Loopback)] GET /identity (18 live) Signed-in
Dec 16, 2021 19:53:58.632 [0x14c59c236b38] DEBUG - Completed: [127.0.0.1:57278] 200 GET /identity (18 live) 0ms 398 bytes (pipelined: 1)
Dec 16, 2021 19:54:01.390 [0x14c593aadb38] DEBUG - [Transcode] Session c405880c-e42c-42de-8190-4ed963183bed-3139 (2) is unthrottling
Dec 16, 2021 19:54:01.391 [0x14c595eb8b38] DEBUG - [Transcoder] Throttle - Getting back to work.
Dec 16, 2021 19:54:01.510 [0x14c5938a5b38] DEBUG - [Transcode] Transcoder segment range: 0 - 822 (822)
Dec 16, 2021 19:54:01.511 [0x14c59721ab38] DEBUG - [Transcode] Transcoder segment range: 0 - 823 (822)
Dec 16, 2021 19:54:01.718 [0x14c59e93fb38] DEBUG - Jobs: ‘/usr/lib/plexmediaserver/Plex Transcoder’ exit code for process 23220 is -11 (signal: Segmentation fault)
Dec 16, 2021 19:54:01.719 [0x14c59737db38] DEBUG - Notifying consumer that data is complete.
Dec 16, 2021 19:54:01.719 [0x14c59520bb38] DEBUG - Killing job.
Dec 16, 2021 19:54:01.719 [0x14c59520bb38] DEBUG - Signalling job ID 23220 with 9
Dec 16, 2021 19:54:01.719 [0x14c59520bb38] DEBUG - Job was already killed, not killing again.
Dec 16, 2021 19:54:01.719 [0x14c59520bb38] DEBUG - Stopping transcode session c405880c-e42c-42de-8190-4ed963183bed-3139
Dec 16, 2021 19:54:01.719 [0x14c595821b38] DEBUG - Cleaning directory for session c405880c-e42c-42de-8190-4ed963183bed-3139 (/transcode/Transcode/Sessions/plex-transcode-c405880c-e42c-42de-8190-4ed963183bed-3139-d115744c-5aae-4ccb-a9a6-bde5f83fdfee)
Dec 16, 2021 19:54:01.719 [0x14c59737db38] DEBUG - Streaming Resource: Terminated session 0x14c59a8cd648:9b5588a828dceeb772e8f9623da4bc7a with reason Conversion failed. The transcoder process crashed.
Dec 16, 2021 19:54:01.735 [0x14c59520bb38] DEBUG - Whacked session c405880c-e42c-42de-8190-4ed963183bed-3139, 1 remaining.

Not sure which files to attach from my debug log zip file, but i’ve only seen this hit 2 episodes so far

The ZIP file is easier for both you and me.

I often look in multiple log files tracking down a problem.

Plex Media Server Logs_2021-12-16_19-55-33.zip (5.5 MB)

done

it ran off the end of the file.

didn’t end well (that’s a problem) but the file length did not match what it had on file.

Did you recently update the file and forget to analyze it ?

f01256fd7c9a-17&maxVideoBitrate=4000&mediaBufferSize=50000&offset=1018&partIndex=0&videoResolution=1280x720&directPlay=0&path=%2Flibrary%2Fmetadata%2F65111&directStream=1&skipSubtitles=1 (29 live) TLS Signed-in Token (Sky1821) (range: bytes=983040-1310719) 
Dec 16, 2021 19:55:23.788 [0x14c59813db38] DEBUG - [Transcode/0bd35b7d-aad7-48f7-837e-f01256fd7c9a-17] Asked for segment 1097 from session.
Dec 16, 2021 19:55:23.788 [0x14c59813db38] DEBUG - [Transcode/0bd35b7d-aad7-48f7-837e-f01256fd7c9a-17] Returning segment 1097 from session
Dec 16, 2021 19:55:23.788 [0x14c59813db38] WARN - Overzealous client asked for end range of 1310719, content size is 944888; we'll clip.
Dec 16, 2021 19:55:23.788 [0x14c59813db38] WARN - Range could not be satisfied 983040 - 944887 (total size=944888)
Dec 16, 2021 19:55:23.789 [0x14c59813db38] DEBUG - Completed: [76.27.14.142:34049] 200 GET /video/:/transcode/universal/session/0bd35b7d-aad7-48f7-837e-f01256fd7c9a-17/base/01097.ts?protocol=hls&mediaIndex=0&waitForSegments=1&videoQuality=100&session=0bd35b7d-aad7-48f7-837e

that was a separate stream, no idea if it faulted similarly. the file I was playing back has not changed in a significant time (several months back at the latest)

Line 5489 (I think) is the start of the segfault for this particular transcode session, c405880c-e42c-42de-8190-4ed963183bed-3139

Here is the zip of my logs which contain the following failure.

Plex Media Server Logs_2021-12-18_00-58-33.zip (5.4 MB)

Dec 18, 2021 00:55:58.183 [0x7f9d473f8b38] DEBUG - Jobs: '/snap/plexmediaserver/239/Plex Transcoder' exit code for process 883854 is -11 (signal: Segmentation fault)
Dec 18, 2021 00:55:58.183 [0x7f9d40936b38] DEBUG - Streaming Resource: Terminating session 0x7f9d4690c188:bfd5ec18b0cd9c8caeb23928b3a13c90 which is using transcoder slot.  Used slots is now 0
Dec 18, 2021 00:55:58.183 [0x7f9d40bfdb38] DEBUG - Killing job.
Dec 18, 2021 00:55:58.183 [0x7f9d40bfdb38] DEBUG - Signalling job ID 883854 with 9
Dec 18, 2021 00:55:58.183 [0x7f9d40bfdb38] DEBUG - Job was already killed, not killing again.
Dec 18, 2021 00:55:58.183 [0x7f9d40bfdb38] DEBUG - Stopping transcode session cb003699-9005-4e12-8e6d-80b3090a26a2-1557

Plex version: 1.25.2.5319

This occurs when I am streaming on my Roku TV, but only once it gets close to the end of the file (timing doesn’t seem consistent, but anytime within the last 5 minutes).

After it crashes playback stops, I can restart playing but it will again crash when it gets close to the end. It does not crash at the same place every time.

I have to explicitly change the stream settings to be Direct Play → Force in Roku in order to watch a video without it crashing.

When it segfaults is when it is transcoding:

Video

1080p (H.264) → 1080P (H264)—Transcode

Audio

English (AAC Stereo) → EAC3—Transcode

System info:

user@plex:~$ uname -a
Linux plex 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

user@plex:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

user@plex:~$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      25
Model:                           80
Model name:                      AMD Ryzen 9 5900HX with Radeon Graphics
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         1794.358
CPU max MHz:                     3300.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        6587.55
Virtualization:                  AMD-V
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        4 MiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm
                                  constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave av
                                 x f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_
                                 nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdse
                                 ed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbno
                                 invd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip pku o
                                 spke vaes vpclmulqdq rdpid overflow_recov succor smca

Would it be possible to obtain a link to a copy of the file so it can be recreated & captured ?

Yes, here is the file.

This is the transcode process running:

user@plex:~$ ps aux | grep "Plex Transcoder"
root     1146328  944  4.0 1027148 643208 ?      Sl   13:39   1:43 /snap/plexmediaserver/239/Plex Transcoder -codec:0 h264 -codec:1 aac -analyzeduration 20000000 -probesize 20000000 -i /media/Plex/Shows/Taskmaster/Season 1/Taskmaster - S01E03 - The Poet and the Egg WEBDL-1080p.mkv -filter_complex [0:0]scale=w=1920:h=1080:force_divisible_by=4[0];[0]format=pix_fmts=yuv420p|nv12[1] -map [1] -codec:0 libx264 -crf:0 17 -maxrate:0 7683k -bufsize:0 15366k -r:0 25 -preset:0 veryfast -level:0 4.1 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=hex:8x8dct=0:partitions=none -force_key_frames:0 expr:gte(t,n_forced*1) -filter_complex [0:1] aresample=async=1:ocl='5.1(side)':rematrix_maxval=0.000000dB:osr=48000[2] -map [2] -metadata:s:1 language=eng -codec:1 eac3_eae -eae_prefix:1 c95099fe-2ecb-4339-80b7-f22a7e5e3f9f-50_ -b:1 699k -segment_format mpegts -f ssegment -individual_header_trailer 0 -segment_time 1 -segment_start_number 0 -segment_copyts 1 -segment_time_delta 0.0625 -segment_list http://127.0.0.1:32400/video/:/transcode/session/c95099fe-2ecb-4339-80b7-f22a7e5e3f9f-50/4e53094c-d69c-4dda-8007-a173fd2922b5/manifest?X-Plex-Http-Pipeline=infinite -segment_list_type csv -segment_list_size 5 -segment_list_separate_stream_times 1 -segment_list_unfinished 1 -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 media-%05d.ts -start_at_zero -copyts -vsync cfr -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/c95099fe-2ecb-4339-80b7-f22a7e5e3f9f-50/4e53094c-d69c-4dda-8007-a173fd2922b5/progress

If it helps as well.

This does not happen on the web or mobile (Android) apps, it does direct play instead of transcoding it.

I forgot to mention before I did recently “migrate” my Plex server from docker on Synology to a stand alone server (I just spun up a new server and started fresh, no db migration or anything). Previously I had never seen an AAC → EAC3 transcode before. This could very much be a red herring, but thought it would be worth mentioning.

Plex is installed as a snap and I mount the media files via an smb share from my Synology.

//nas.ip.addr.here/Media /media/Plex cifs credentials=/home/user/.smbcredentials,iocharset=utf8,vers=1.0,uid=0,gid=0 0 0

I know this is about a synology nas but i believe I’m having the same issue but running Truenas with server 1.25.2.5319. I can provide logs or create a new post about it if that’s preferred. It seems to only affect videos in surround sound being transcoded on my roku. If i force the roku to stereo it plays fine.