PMS Segfault During Transcode - Server Crash

Plex Media Server Desktops & Laptops
1

I am rather new to plex and just made the decision to use it as my primary media server for our collection of tablets and other connected devices.  I have been having issues with segfaults within PMS as well as hard server crashes.

 

Server OS: Ubuntu 14.04 (Server)

CPU: Pentium D 2.8 Ghz

Mem: 4GB

Primary Media Storage: Drobo via USB2.0

 

** I have parts for a new plex server in the mail.

 

PMS: 0.9.11.5.774-760

 

Troubleshooting...

  1) Performed fsck on all disks.

  2) Booted into recovery mode and ran memtest (pass)

  3) Run cpuburn on both cores for 1 hour+

  4) Wiped my server OS and reinstalled from scratch.

    a) Fresh install of PMS and recreated libraries.

 

After doing a server rebuild I was hopeful.  The kids had the server down hard within an hour.  They were trying to load a video on an iPad.  The last crash came with no kernel dump on the screen.  I had enabled debug logging within the server prior to the crash although nothing jumps out at me.

 

kern.log

Dec  7 15:46:13 ogre kernel: [15416.208039] Plex New Transc[24380]: segfault at 1 ip 0000000000000001 sp 00007fff033818b8 error 14 in Plex New Transcoder[400000+1151000]

Plex Server Log

Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG -  * X-Plex-Device => iPad
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG -  * X-Plex-Platform => iOS
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG -  * X-Plex-Product => Plex/iOS
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - We found auth token (xxxxxxxxxxxxxxxxxxxx), enabling token-based authentication.
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - Came in with a super-token, authorization succeeded.
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - Client [56B41342-8BD4-46BF-B76C-39C357CCA05D] reporting timeline state paused, progress of 109633/5916875ms for guid=, ratingKey=150 url=, key=/library/metadata/150, containerKey=, metadataId=150
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - Play progress on 150 'The Croods' - got played 109633 ms by account 1!
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - [Now] User is usulsuspct (ID: 1)
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - [Now] Device is iOS (Kelly's iPad).
Dec 07, 2014 15:50:00 [0x7fba177fa700] DEBUG - [Now] Updated play state for /library/metadata/150.
Dec 07, 2014 15:50:02 [0x7fba17ffb700] DEBUG - Request: [192.168.1.107:46881] GET /:/progress?key=51&identifier=com.plexapp.plugins.library&time=5087000 (15 live)
Dec 07, 2014 15:50:02 [0x7fba17ffb700] DEBUG -  * key => 51
Dec 07, 2014 15:50:02 [0x7fba17ffb700] DEBUG -  * identifier => com.plexapp.plugins.library
Dec 07, 2014 15:50:02 [0x7fba17ffb700] DEBUG -  * time => 5087000
Dec 07, 2014 15:50:05 [0x7fba177fa700] DEBUG - Request: [192.168.1.180:53109] GET /video/:/transcode/universal/session/56B41342-8BD4-46BF-B76C-39C357CCA05D-00/base/00000.ts (15 live)
Dec 07, 2014 15:50:05 [0x7fba177fa700] DEBUG - Asked for segment 0 from session.
Dec 07, 2014 15:50:07 [0x7fba17ffb700] DEBUG - Request: [192.168.1.107:46882] GET /:/progress?key=51&identifier=com.plexapp.plugins.library&time=5087000 (15 live)
Dec 07, 2014 15:50:07 [0x7fba17ffb700] DEBUG -  * key => 51
Dec 07, 2014 15:50:07 [0x7fba17ffb700] DEBUG -  * identifier => com.plexapp.plugins.library
Dec 07, 2014 15:50:07 [0x7fba17ffb700] DEBUG -  * time => 5087000
Dec 07, 2014 15:50:12 [0x7fba17ffb700] DEBUG - Request: [192.168.1.107:46883] GET /:/progress?key=51&identifier=com.plexapp.plugins.library&time=5087000 (15 live)
Dec 07, 2014 15:50:12 [0x7fba17ffb700] DEBUG -  * key => 51
Dec 07, 2014 15:50:12 [0x7fba17ffb700] DEBUG -  * identifier => com.plexapp.plugins.library
Dec 07, 2014 15:50:12 [0x7fba17ffb700] DEBUG -  * time => 5087000
Dec 07, 2014 15:50:15 [0x7fba17ffb700] DEBUG - Request: [192.168.1.180:53110] GET /video/:/transcode/universal/session/56B41342-8BD4-46BF-B76C-39C357CCA05D-00/base/00000.ts (15 live)
Dec 07, 2014 15:50:15 [0x7fba17ffb700] DEBUG - Asked for segment 0 from session.
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^

Media

  • Video Resolution 720p
  • Duration 1:38:36
  • Bitrate 3959 kbps
  • Width 1280
  • Height 544
  • Aspect Ratio 2.35
  • Container AVI
  • Video Frame Rate 24p

Video

  • Codec MPEG4
  • Bitrate 3501 kbps
  • Bit Depth 8
  • Chroma Subsampling 4:2:0
  • Color Space yuv
  • Duration 1:38:36
  • Frame Rate 23.976 fps
  • Gmc 0
  • Height 544
  • Level 0.5
  • Profile asp
  • Qpel 0
  • Scan Type progressive
  • Width 1280

Audio

  • Codec AC3
  • Channels 5.1
  • Bitrate 448 kbps
  • Bit Depth 16
  • Bitrate Mode CBR
  • Dialog Norm -31 dB
  • Duration 1:38:36
  • Sampling Rate 48000 Hz
  • Stream Identifier 1
2

I had another kernel oops last night.  I am starting to think this may be IO related.

[42605.184223] kernel tried to execute NX-protected page - exploit attempt? (uid: 103)
[42605.184288] BUG: unable to handle kernel paging request at ffff8800b6333ec8
[42605.184341] IP: [] 0xffff8800b6333ec8
[42605.184384] PGD 1fdf067 PUD 13bffe067 PMD 80000000b62001e3 
[42605.184429] Oops: 0011 [#1] SMP 
[42605.184459] Modules linked in: des_generic md4 nls_utf8 cifs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache ppdev snd_hda_codec_analog i915 snd_hda_intel snd_hda_codec dcdbas snd_hwdep snd_pcm video serio_raw joydev snd_page_alloc drm_kms_helper snd_timer drm snd lpc_ich soundcore i2c_algo_bit mac_hid parport_pc lp parport hid_generic usbhid hid tg3 ptp psmouse usb_storage pps_core
[42605.184795] CPU: 1 PID: 12618 Comm: Plex New Transc Not tainted 3.13.0-32-generic #57-Ubuntu
[42605.184853] Hardware name: Dell Inc.                 OptiPlex 745                 /0RF705, BIOS 2.0.8  11/30/2006
[42605.184922] task: ffff8801344f2fe0 ti: ffff8800b6332000 task.ti: ffff8800b6332000
[42605.184973] RIP: 0010:[]  [] 0xffff8800b6333ec8
[42605.185028] RSP: 0018:ffff8800b6333d50  EFLAGS: 00010202
[42605.185066] RAX: 0000000000000020 RBX: 0000000000000850 RCX: ffff88013bff8000
[42605.185114] RDX: 000000001e00a0af RSI: 000000000001c15f RDI: 0000000000000020
[42605.185163] RBP: ffffffff8115add1 R08: 0000000000000000 R09: 000000000002e771
[42605.185211] R10: 000000000002d80e R11: 000000000002d80e R12: ffff8800b6333d80
[42605.185260] R13: ffff88007ad78200 R14: ffff880036252800 R15: 0000000000000001
[42605.185309] FS:  00007f8836a65740(0000) GS:ffff88013bc40000(0000) knlGS:0000000000000000
[42605.185364] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[42605.185404] CR2: ffff8800b6333ec8 CR3: 00000000999dd000 CR4: 00000000000007e0
[42605.185453] Stack:
[42605.185469]  0000000000000005 ffff880036252878 ffff88007ad78200 0000000000000005
[42605.185528]  ffff880036252800 ffffea0001e6ab40 ffff8800b6333db8 ffffffff8115b110
[42605.185586]  ffffffff81365a2e ffff8800b6333db8 ffffffff8114e33e ffff88007ad78200
[42605.185644] Call Trace:
[42605.185671]  [] ? page_cache_async_readahead+0x80/0xa0
[42605.185723]  [] ? radix_tree_lookup_slot+0xe/0x10
[42605.185770]  [] ? find_get_page+0x1e/0xa0
[42605.185812]  [] ? generic_file_aio_read+0x468/0x700
[42605.185861]  [] ? do_sync_read+0x5a/0x90
[42605.185902]  [] ? vfs_read+0x95/0x160
[42605.185941]  [] ? SyS_read+0x49/0xa0
[42605.185982]  [] ? tracesys+0xe1/0xe6
[42605.186018] Code: 00 00 00 00 28 25 36 00 88 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0 2f 4f 34 01 88 ff ff <00> 00 00 00 00 00 00 00 e8 c7 84 00 00 00 00 00 00 80 00 00 00 
[42605.186323] RIP  [] 0xffff8800b6333ec8
[42605.186365]  RSP 
[42605.186390] CR2: ffff8800b6333ec8
[42605.188008] ---[ end trace 8474bb180a2317db ]---
3

Still no clue what is happening here.  I have generated IO manually via DD to both local and USB/network mounted drives all error free both reading and writing.  All of the individual hardware components have been tested based on what I know how to do all checked out okay.

Seemingly okay to direct play content, watched 2 hours last night via direct play on a Amazon Fire Stick.

4

My hypothesis is that the CPU is struggling with transcoding and is not adequately powered or cooled. From what I understand you're only segfaulting when transcoding something (CPU intensive), and not when direct playing it (minimal CPU usage). But if you have some crash logs please upload them.

5

I have run cpu burn in process on the system to peg the cores to 100% for 30-40 minutes as I had originally though the same.  I did some further testing today which has me looking outside of Plex.

If I try and encode a file via handbrake-cli I also get a segfault.  Backtraces on that process may indicate an issue with libx264 possibly.  I am not really sure where to go from there as this is a fresh off the ISO install of ubuntu 14.04.1.  Seeing as the encode also fails outside of plex my guess is the issue is not with plex rather something more system wide.

6

Hmm that is an unusual issue. Have you tried to encode a video with FFmpeg too? That is the video/audio library that Plex's transcoder uses. If you upload any crash logs you might have that will help a dev confirm it's not an issue with Plex itself.

7

I'm having the same issue...  and from what I can tell, Plex appears to be doing this in the middle of the night (2am)... and then my server locks up.  When I look at syslog, the last thing I see is the message from the original post.

I had to end up removing Plex, since I do have other things running on this machine.

I also don't know why plex was running a transcode job in the middle of the night, since no one is using the machine at that time.

I tried this for 3 nights... and for 3 nights I got a consistent lock up.   Since I removed plex, no more lock ups.  I'd really just to get this working, since I am a plex pass member, and I'd like to take advantage of plex.

8

I'm having the same issue...  and from what I can tell, Plex appears to be doing this in the middle of the night (2am)... and then my server locks up.  When I look at syslog, the last thing I see is the message from the original post.

I had to end up removing Plex, since I do have other things running on this machine.

I also don't know why plex was running a transcode job in the middle of the night, since no one is using the machine at that time.

I tried this for 3 nights... and for 3 nights I got a consistent lock up.   Since I removed plex, no more lock ups.  I'd really just to get this working, since I am a plex pass member, and I'd like to take advantage of plex.

I noticed my server was running transcode processes around 3am the other night. Prompted me to turn off DLNA and GDM but the threads still kept popping up even if i killed them manually. So I shut plex down and brought it back up. The transcode threads went away.

9

I noticed my server was running transcode processes around 3am the other night. Prompted me to turn off DLNA and GDM but the threads still kept popping up even if i killed them manually. So I shut plex down and brought it back up. The transcode threads went away.

you likely just have indexing enabled in your scheduled tasks.

10

I noticed my server was running transcode processes around 3am the other night. Prompted me to turn off DLNA and GDM but the threads still kept popping up even if i killed them manually. So I shut plex down and brought it back up. The transcode threads went away.

you likely just have indexing enabled in your scheduled tasks.

Thanks guys, I'll re-install plex, and turn off those things, and see if that solves my lock up issues.

11

So which of these options would cause the Transcoder to run... since that appears to be causing my lockup issues. 

I'm thinking generate missing media index files, but I'm not sure why that would cause the Transcoder to run.

 Backup database every three days
 Optimize database every week
 Remove old bundles every week
 Remove old cache files every week
 Generate missing media index files during maintenance

Index files provide thumbnail previews in Now Playing and while seeking on supported clients. These consume additional disk space.

 Refresh local metadata every three days
 Refresh all libraries during maintenance
 Upgrade media analysis during maintenance
12

I'd like to second this topic. Recently I've been having multiple crashes when transcoding files on the server. I'm running a SuperMicro X7DVL-I (might not be right, off the top of my head) with dual quad core CPUs and 8GB of RAM. 

I've been running ffmpeg processes on this box for days with no shutdowns or problems batch converting a large group of old MPeg2 videos to MP4/x264 videos, using a command line script.

However, as soon as my kids start watching Plex on the TV upstairs, the server will peg one core (usually #4 or #8) and restart. One time, and one time only, I caught the line "Cannot synchronize CPU cores" on the console screen before the reboot.

This also happens with the HandBrakeCLI client. In fact, it crashed so quickly with HandBrakeCLI, I switched to using ffmpeg instead.

I have another identical server running the same conversion tasks (I have a *lot* of old MP2 files) that never crashes (over 200 hours running ffmpeg) -- the only difference is that it is *not* running a Plex Media Server.

Indexing is turned off on the server, so that's not the issue. This happens on certain files (usually MKV/x264 files) and not on others. Sometimes it will happen 2-3 times watching a one hour video.

Any help would be appreciated. The reboots are failing the conversion process, and I'm having to re-run every failed conversion, sometimes losing tens of hours of 100% CPU time.

Running Ubuntu 14.04 Server

Distributor ID: Ubuntu
Description:    Ubuntu 14.04.2 LTS
Release:        14.04
Codename:       trusty
 
processor       : 0,1,2,3,4,5,6,7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           L5420  @ 2.50GHz
stepping        : 10
cpu MHz         : 2499.997
 
 dpkg -s plexmediaserver
Package: plexmediaserver
Status: install ok installed
Priority: extra
Section: video
Maintainer: Christian Svedin (gurka)
Architecture: amd64
Version: 0.9.12.1.1079-b655370-debian
 
13

Did you ever fix this?

14

Old thread. should have been closed.
Please open a fresh thread.

Closing this one.