Server-linux Plex Media Scan segmentation faults

Server Version#: 1.16.6.1592
Player Version#:

Plex Media Scan process keeps crashing while scanning one of my libraries.

[57839.521328] Plex Media Scan[27417]: segfault at a1 ip 00007fc1c4e30097 sp 00007fc1aae2d0d0 error 4 in libcrypto.so.1.0.0[7fc1c4d1f000+204000]
[57839.521335] Code: 8b 4f 1c 31 d2 4c 89 e0 48 f7 f1 49 8b 07 48 63 ca 4c 8b 2c c8 4d 85 ed 74 37 49 8b 6f 08 48 8d 1c c8 90 49 ff 87 a0 00 00 00 <4d> 39 65 10 75 11 49 ff 47 68 49 8b 7d 00 4c 89 f6 ff d5 85 c0 74
[61229.306851] traps: Plex Media Scan[335] general protection fault ip:7f2d50ab4097 sp:7f2d36abcfe0 error:0 in libcrypto.so.1.0.0[7f2d509a3000+204000]
[62906.519718] traps: Plex Media Scan[3032] general protection fault ip:7f072b969097 sp:7f071195a0d0 error:0 in libcrypto.so.1.0.0[7f072b858000+204000]
[63069.576128] traps: Plex Media Scan[3305] general protection fault ip:7f8d5b57f097 sp:7f8d41590fe0 error:0 in libcrypto.so.1.0.0[7f8d5b46e000+204000]
[64122.293960] traps: Plex Media Scan[5002] general protection fault ip:7fa039f9f097 sp:7fa01fdf8fe0 error:0 in libcrypto.so.1.0.0[7fa039e8e000+204000]
[65920.013204] traps: Plex Media Scan[8129] general protection fault ip:7f6cf063d097 sp:7f6cd663dfe0 error:0 in libcrypto.so.1.0.0[7f6cf052c000+204000]
[66528.301498] traps: Plex Media Scan[9398] general protection fault ip:7faa2e311097 sp:7faa13ffdfe0 error:0 in libcrypto.so.1.0.0[7faa2e200000+204000]
[67369.018161] traps: Plex Media Scan[11092] general protection fault ip:7f65e023e097 sp:7f65c6244fe0 error:0 in libcrypto.so.1.0.0[7f65e012d000+204000]
[71034.341685] traps: Plex Media Scan[18399] general protection fault ip:7f328d7a6097 sp:7f32737a8fe0 error:0 in libcrypto.so.1.0.0[7f328d695000+204000]
[72925.160729] traps: Plex Media Scan[25513] general protection fault ip:7ff97cc32097 sp:7ff962c2dfe0 error:0 in libcrypto.so.1.0.0[7ff97cb21000+204000]

May I have the full logs please? (DEBUG logging – not VERBOSE – ZIP file after you make it segfault).

Restarting PMS and then scanning would help me more.

The last crash at 14:57 MT was uploaded to Plex:
Sep 08, 2019 14:57:52.855 [0x7f3caaae8740] INFO - Successfully reported 7589e734-50fc-5ebe-251781b7-42459e4b

dmesg from crash:
[81714.886374] traps: Plex Media Scan[16132] general protection fault ip:7f84f1290097 sp:7f84d729afe0 error:0 in libcrypto.so.1.0.0[7f84f117f000+204000]

The crash at 14:57 is recorded in Plex Media Server.1.log

logs.zip (3.7 MB)

That makes no sense other than what I’ve seen in Fedora 30 itself.

I use Fedora 28 here because I cannot use 29 or 30 for production. They are far too unstable.
For libcrypto to trap, this makes no sense. Do you have a custom certificate?

I also get this on Ubuntu 18.04 LTS, currently using mainline kernel 5.2.11. I am on Server Version#: 1.16.6.1592. No custom certificate.

What does mediainfo report of the file (it shows the headers) and what is the exact file name.

I will try to reproduce using the exact conditions (same distros and versions as well)

No custom certificate on my Fedora server.

I didn’t have the same issue with the previous version and maybe I need to downgrade.
plexmediaserver-1.16.3.1433-359b06978.x86_64.rpm

Scanning those logs for ERROR I find a

Plex Media Scanner.log:1677:Sep 08, 2019 15:17:49.652 [0x7f5301a74740] ERROR - Exception caught determining whether we could skip 'Doctor.Who' ~ Null value not allowed for this type

and I find a handful of FFmpeg errors like these:

Plex Media Scanner.2.log:18170:Sep 08, 2019 15:17:48.653 [0x7fa9c77fe700] ERROR - [FFMPEG] - moov atom not found
Plex Media Scanner.2.log:18172:Sep 08, 2019 15:17:48.653 [0x7fa9c77fe700] ERROR - Exception analyzing media file '/local/clinton/Download/mirc/Outcast.S01E06.WEB-DL.x264-FUM.mp4' (Could not parse /local/clinton/Download/mirc/Outcast.S01E06.WEB-DL.x264-FUM.mp4 (error=-1094995529): Invalid data found when processing input)
Plex Media Scanner.2.log:18180:Sep 08, 2019 15:17:48.700 [0x7fa9c77fe700] ERROR - [FFMPEG] - moov atom not found
Plex Media Scanner.2.log:18182:Sep 08, 2019 15:17:48.700 [0x7fa9c77fe700] ERROR - Exception extracting thumbnail from media file '/local/clinton/Download/mirc/Outcast.S01E06.WEB-DL.x264-FUM.mp4' (Could not parse /local/clinton/Download/mirc/Outcast.S01E06.WEB-DL.x264-FUM.mp4 (error=-1094995529): Invalid data found when processing input)

Can we remove the offending files from PMS and try again?
I used a command like this (on macOS):

find . -name "*.log" | xargs -I{} fgrep {} --mmap -n -H -e ERROR > errors.txt

I need to see the entire ZIP file if you want me to debug.

ERROR statements alone tell me nothing as I have no diagnosis path to complete RCA.

I moved the trouble files into another directory that PMS isn’t monitoring.

This command works fine for me:
find /var/lib/plexmediaserver -name “*.log” | xargs -I{} fgrep {} -n -H -e ERROR

@sclintow

You will find ALL logs under /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs . There is no need to search for them.

@sclintow Please stick with Chuck on this.
I was eager to point out the ERROR messages I saw in the logs you uploaded.
I shouldn’t have asked you to move files before you finished following his direction.
Not that moving a few files is bad, but he knows better :slightly_smiling_face:

Moving suspected bad files to a different area is a good idea.
The scanner isn’t infallible. If the media files are damaged, it could cause a problem. A libcrypto fault (meaning a wild memory pointer) is not what I would expect though from data)

I downgraded to the older version and new media files are now being discovered properly. I haven’t had another segmentation fault since downgrading.

plexmediaserver-1.16.3.1433-359b06978.x86_64.rpm

Plex Media Scanner crashes regularly here. PMS stays running and appears to be functioning correctly.
Latest Plex Pass version, Ubuntu 18.04.3 LTS

[25529.684941] Plex Media Scan[7281]: segfault at 7fc9c2d45090 ip 00007fc9cbd6a71a sp 00007fc9c374c918 error 7 in libc-2.27.so[7fc9cbcaf000+1e7000]
[25529.684948] Code: 4d 29 c1 4c 29 c2 48 3b 15 7b 4f 33 00 0f 87 af 00 00 00 0f 10 01 0f 10 49 f0 0f 10 51 e0 0f 10 59 d0 48 83 e9 40 48 83 ea 40 <41> 0f 29 01 41 0f 29 49 f0 41 0f 29 51 e0 41 0f 29 59 d0 49 83 e9
[28225.427048] Plex Media Scan[7920]: segfault at 100000010 ip 00007fddffe12097 sp 00007fddf1e3efe0 error 4 in libcrypto.so.1.0.0[7fddffd01000+204000]
[28225.427057] Code: 8b 4f 1c 31 d2 4c 89 e0 48 f7 f1 49 8b 07 48 63 ca 4c 8b 2c c8 4d 85 ed 74 37 49 8b 6f 08 48 8d 1c c8 90 49 ff 87 a0 00 00 00 <4d> 39 65 10 75 11 49 ff 47 68 49 8b 7d 00 4c 89 f6 ff d5 85 c0 74
[36629.132029] Plex Media Scan[8740]: segfault at 31 ip 00007f2769163097 sp 00007f275b2aefe0 error 4 in libcrypto.so.1.0.0[7f2769052000+204000]
[36629.132038] Code: 8b 4f 1c 31 d2 4c 89 e0 48 f7 f1 49 8b 07 48 63 ca 4c 8b 2c c8 4d 85 ed 74 37 49 8b 6f 08 48 8d 1c c8 90 49 ff 87 a0 00 00 00 <4d> 39 65 10 75 11 49 ff 47 68 49 8b 7d 00 4c 89 f6 ff d5 85 c0 74
[39325.664950] Plex Media Scan[9195]: segfault at 41 ip 00007fe0c13b3097 sp 00007fe0b336f0d0 error 4 in libcrypto.so.1.0.0[7fe0c12a2000+204000]
[39325.664958] Code: 8b 4f 1c 31 d2 4c 89 e0 48 f7 f1 49 8b 07 48 63 ca 4c 8b 2c c8 4d 85 ed 74 37 49 8b 6f 08 48 8d 1c c8 90 49 ff 87 a0 00 00 00 <4d> 39 65 10 75 11 49 ff 47 68 49 8b 7d 00 4c 89 f6 ff d5 85 c0 74
[41725.597414] Plex Media Scan[9543]: segfault at 100000010 ip 00007f6db1d1d097 sp 00007f6da3df8fe0 error 4 in libcrypto.so.1.0.0[7f6db1c0c000+204000]
[41725.597423] Code: 8b 4f 1c 31 d2 4c 89 e0 48 f7 f1 49 8b 07 48 63 ca 4c 8b 2c c8 4d 85 ed 74 37 49 8b 6f 08 48 8d 1c c8 90 49 ff 87 a0 00 00 00 <4d> 39 65 10 75 11 49 ff 47 68 49 8b 7d 00 4c 89 f6 ff d5 85 c0 74
[57029.113710] traps: Plex Media Scan[11654] general protection fault ip:7fb00f0bc097 sp:7fb001153fe0 error:0 in libcrypto.so.1.0.0[7fb00efab000+204000]
[91529.040677] Plex Media Scan[15424]: segfault at 71 ip 00007f317929a097 sp 00007f316b36f0d0 error 4 in libcrypto.so.1.0.0[7f3179189000+204000]
[91529.040686] Code: 8b 4f 1c 31 d2 4c 89 e0 48 f7 f1 49 8b 07 48 63 ca 4c 8b 2c c8 4d 85 ed 74 37 49 8b 6f 08 48 8d 1c c8 90 49 ff 87 a0 00 00 00 <4d> 39 65 10 75 11 49 ff 47 68 49 8b 7d 00 4c 89 f6 ff d5 85 c0 74

Please downgrade to an earlier PMS build while we investigate further.

After searching for error 4 in libcrypto.so.1.0.0 for another post,
I came across a Github project that resolved the same error message in their project,
narrowing it down to an interaction with SSL. Would someone familiar with C++
let us know if this looks promising?

Thanks,
Where can I find earlier versions of Plex for download?

You didn’t keep any of the previous ones?

IIRC, Debian keeps downloaded repo packages in its cache. I would need look that up.

Found version plexmediaserver_1.15.4.993-bb4a2cb6c_amd64.deb and downgraded.
It would be great if there were a repository available for easy downgrading.
Thanks for a fantastic product - keep up the great work! I know you guys will get this fixed eventually.