I’m not sure in which category to put this. Putting it in Mobile, since the server never crashes if I don’t touch Plex on iOS for a while. No crashes in 3 months right after I cut down iOS usage, then multiple crashes came back right after I’d started using it again - all within the same server version (and DSM version). Extensive usage of Plex for Windows, Plex Web or Plex MPV Shim never leads to any crashes.
My server:
Synology DS216+II
DSM 6.2.3-25426 Update 3
PMS Intel 64-bit v1.23.6.4881-e2e58f321
My client:
iPad 2018 (6th gen)
iOS 14.7.1
Plex for iOS 7.25.1615
Server logs:
Plex Media Server Logs_2021-12-14_18-15-53.zip (4.1 MB)
The most recent crash time (Dec 14 ~16:33) is in the Plex Media Server.1.log file at the end of it.
Server crash dumps including the last crash:
1.23.6.4881-e2e58f321.zip (3.4 MB)
Server kernel segfault with all PMS crashes (big timespan between crash in September and December is likely because I didn’t use Plex on iOS during that time):
kern.log (1.2 MB)
Bonus lookup of the segfault address will be at the bottom of the post (always points to the same server code segment).
iPad Plex Client logs:
PlexDebugInfo-7.25-1615 (2021-12-14 17.16.06 +0300).zip (545.8 KB)
I wrote about the crashes before I connected them with iOS, that topic was closed: Random crashes on DS216+II, DSM 6.2.3, PMS 1.22.3.4392 and earlier
Those crashes were seemingly random, sometimes it would crash when I opened my Plex iOS app, but not every time I opened it - I’m unable to reproduce it elaborately. Other times no one used the server and it still crashed. Both types of crashes had no obvious crash reasons in the logs. Both produced kernel segfaults with the same address.
Segfault crashes stopped right when I stopped using iOS clients completely. It started again when I resumed using them (mostly used to cast from them to PC clients). So I started to look for iOS-specific logs.
After inspecting the logs and app behavior described below, I’m starting to think: maybe all these crashes are related to something with the dashboard that happens both when I load the app and when those ghost logs happen.
I have Tautulli notification set up when the server is down (comes about 1 minute after the crash). Today I got a PMS down notification at 16:34 while my desktop Plex client was paused on some episode (desktop IP ends with 232). I haven’t used the iPad Plex at all today (IP ends with 203). I’m the only active user of my server (Ann, ID 16587110 in logs).
My iPad IP address suddenly appears in the server logs right before the crash time, and continues right until the end looking like this:
Dec 14, 2021 16:33:00.219 [0x7fa92c049b38] DEBUG - Auth: authenticated user 16587110 as Ann
Dec 14, 2021 16:33:00.219 [0x7fa929b93b38] DEBUG - Request: [192.168.1.203:56181 (Subnet)] GET / (11 live) TLS GZIP Signed-in Token (Ann)
Dec 14, 2021 16:33:00.220 [0x7fa929b93b38] DEBUG - (Capabilities) Platform 'iOS' not matched by plugin platform requirements
Dec 14, 2021 16:33:00.221 [0x7fa92c06cb38] DEBUG - Completed: [192.168.1.203:56181] 200 GET / (11 live) TLS GZIP 2ms 4358 bytes (pipelined: 1)
Dec 14, 2021 16:33:02.625 [0x7fa929b93b38] DEBUG - Request: [192.168.1.203:56195 (Subnet)] GET /hubs/continueWatching?contentDirectoryID=9%2C4%2C2%2C1%2Cplaylists&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1 (10 live) TLS GZIP Signed-in Token (Ann)
Dec 14, 2021 16:33:02.625 [0x7fa929b93b38] DEBUG - HubCache: Adding '16587110/continueWatching/1/hubs/continueWatching/enexternal-media/contentDirectoryID=9%2C4%2C2%2C1%2Cplaylists&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1' to the cache (16587110/continueWatching/1/hubs/continueWatching/enexternal-media/contentDirectoryID=9%2C4%2C2%2C1%2Cplaylists&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1).
Dec 14, 2021 16:33:02.629 [0x7fa92c049b38] DEBUG - Auth: authenticated user 16587110 as Ann
Dec 14, 2021 16:33:02.631 [0x7fa92ad2bb38] DEBUG - Request: [192.168.1.203:56202 (Subnet)] GET /hubs/promoted?contentDirectoryID=9&count=50&excludeContinueWatching=1&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1&pinnedContentDirectoryID=9%2C4%2C2%2C1%2Cplaylists (12 live) TLS GZIP Signed-in Token (Ann)
Dec 14, 2021 16:33:02.632 [0x7fa92ad2bb38] DEBUG - HubCache: Retrieving '16587110/home.continue/1/hubs/promoted/enexternal-media/contentDirectoryID=9&count=50&excludeContinueWatching=1&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1&pinnedContentDirectoryID=9%2C4%2C2%2C1%2Cplaylists' from the cache.
Dec 14, 2021 16:33:02.632 [0x7fa92ad2bb38] DEBUG - HubCache: 13 hubs cached, 73.9% hit ratio.
Dec 14, 2021 16:33:02.632 [0x7fa92ad2bb38] DEBUG - HubCache: Retrieving '16587110/home.ondeck/1/hubs/promoted/enexternal-media/contentDirectoryID=9&count=50&excludeContinueWatching=1&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1&pinnedContentDirectoryID=9%2C4%2C2%2C1%2Cplaylists' from the cache.
Dec 14, 2021 16:33:02.633 [0x7fa92ad2bb38] DEBUG - HubCache: Adding '16587110/home.movies.recent.9/1/hubs/promoted/enexternal-media/contentDirectoryID=9&count=50&excludeContinueWatching=1&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1&pinnedContentDirectoryID=9%2C4%2C2%2C1%2Cplaylists' to the cache (16587110/home.movies.recent.9/1/hubs/promoted/enexternal-media/contentDirectoryID=9&count=50&excludeContinueWatching=1&includeExternalMedia=1&includeGames=1&includeMeta=1&includeRecentChannels=1&pinnedContentDirectoryID=9%2C4%2C2%2C1%2Cplaylists).
Dec 14, 2021 16:33:02.634 [0x7fa92ad2bb38] DEBUG - Auth: Refreshing tokens inside the token-based authentication filter.
Right after receiving PMS down notification, I pressed the home button twice on my iPad, Plex iOS app was not launched there at the time of the crash and at the time of those logs with 203 IP. It wasn’t there, since some time ago I started force-closing Plex app on iPad when I don’t use it, cause I wanted to eliminate any background activity by explicitly closing it - but it didn’t affect the crashes, they kept happening all the same as before, when the app usually stayed open in the background.
Today, when I exported the client logs from iPad Plex (opened it to make the export around 17:15 timestamp, it was closed before), there was also activity at the time of the crash, just like in the server logs, and during the day too. How is that possible with the client app explicitly closed on iOS?
This is not the first time I noticed this discrepancy. All of my segfault PMS crashes happened either right when I explicitly loaded my Plex dashboard on iOS, or at some random times when I didn’t even used Plex. Both cases may be connected to some background dashboard activity that happens both when I load the app and when those ghost logs happen without any actual client usage.
I even removed the iPad TestFlight app, and installed the App Store app before I returned to the iOS app, but that didn’t help. I’m going to update the server version soon (I’m confused about some DSM 7 related things in the changelog, I hope it won’t affect my DSM 6 PMS install), but I updated the PMS frequently since the issue started around v1.21.1.3876 server update I did this spring (first segfault on March 26 2021, not sure if there a PMS version update log I could compare with, can’t name the previous version that didn’t have the issue), but PMS updates were never helpful before - issue always came back after some time.
Is it normal if both server and client logs contain activity from the iOS device that was force-closed? I know Apple is strict regarding the background activity, which makes those ghost connections seem even weirder.
If there’s nothing helpful in the logs, will it help to debug running PMS via a debugger until it crashes again? Supposedly there is a synogear package for DSM with gdb in it, but I’m not sure if I can use it with PMS and how.
PS: If I look up the address from segfault in kern.log (segfault line ip minus hex number from square brackets in the hex calculator)
<...>
2021-08-03T16:34:01+03:00 DiskStation kernel: [6649003.002244] Plex Media Serv[19030]: segfault at 0 ip 00007f2e73a31192 sp 00007f2e6c3b9ef0 error 4 in Plex Media Server[7f2e730e3000+99c000]
2021-08-03T17:01:44+03:00 DiskStation kernel: [6650665.347910] Plex Media Serv[20259]: segfault at 0 ip 00007fbcbc06e192 sp 00007fbcb7b7def0 error 4 in Plex Media Server[7fbcbb720000+99c000]
2021-08-03T18:55:33+03:00 DiskStation kernel: [6657494.590033] Plex Media Serv[21980]: segfault at 0 ip 00007ff0822a6192 sp 00007ff07dd03ef0 error 4 in Plex Media Server[7ff081958000+99c000]
2021-08-11T10:11:29+03:00 DiskStation kernel: [39761.755071] Plex Media Serv[1789]: segfault at 0 ip 00007f8967c78192 sp 00007f896158eef0 error 4 in Plex Media Server[7f896732a000+99c000]
2021-08-11T21:38:59+03:00 DiskStation kernel: [81011.266368] Plex Media Serv[30307]: segfault at 0 ip 00007f11ed80a192 sp 00007f11e70c6ef0 error 4 in Plex Media Server[7f11ecebc000+99c000]
2021-08-12T15:41:56+03:00 DiskStation kernel: [145986.906985] Plex Media Serv[11054]: segfault at 0 ip 00007f1502429192 sp 00007f14fbe39ef0 error 4 in Plex Media Server[7f1501adb000+99c000]
2021-08-12T20:48:16+03:00 DiskStation kernel: [164366.987014] Plex Media Serv[30608]: segfault at 0 ip 00007fa3dad53192 sp 00007fa3d3a3def0 error 4 in Plex Media Server[7fa3da405000+99c000]
2021-08-13T01:55:05+03:00 DiskStation kernel: [182775.610256] Plex Media Serv[24260]: segfault at 0 ip 00007fc3d3b59192 sp 00007fc3cca33ef0 error 4 in Plex Media Server[7fc3d320b000+99c000]
2021-08-13T13:32:41+03:00 DiskStation kernel: [224630.612346] Plex Media Serv[30729]: segfault at 0 ip 00007faaea490192 sp 00007faae422bef0 error 4 in Plex Media Server[7faae9b42000+99c000]
2021-08-13T22:35:40+03:00 DiskStation kernel: [257208.942965] Plex Media Serv[18711]: segfault at 0 ip 00007f6ea836e192 sp 00007f6ea16fdef0 error 4 in Plex Media Server[7f6ea7a20000+99c000]
2021-08-19T19:50:19+03:00 DiskStation kernel: [765679.905912] Plex Media Serv[27953]: segfault at 0 ip 00007f639569d192 sp 00007f638e38cef0 error 4 in Plex Media Server[7f6394d4f000+99c000]
2021-08-20T00:44:35+03:00 DiskStation kernel: [783336.221179] Plex Media Serv[10061]: segfault at 0 ip 00007f006c284192 sp 00007f0065315ef0 error 4 in Plex Media Server[7f006b936000+99c000]
2021-08-20T05:06:00+03:00 DiskStation kernel: [799020.311332] Plex Media Serv[28009]: segfault at 0 ip 00007f374a7da192 sp 00007f37432cfef0 error 4 in Plex Media Server[7f3749e8c000+99c000]
2021-08-20T14:06:24+03:00 DiskStation kernel: [831443.990681] Plex Media Serv[7490]: segfault at 0 ip 00007f41e1406192 sp 00007f41db05aef0 error 4 in Plex Media Server[7f41e0ab8000+99c000]
2021-08-22T19:06:56+03:00 DiskStation kernel: [1022273.155827] Plex Media Serv[19864]: segfault at 0 ip 00007ff4b0c77192 sp 00007ff4a953fef0 error 4 in Plex Media Server[7ff4b0329000+99c000]
2021-09-09T16:18:19+03:00 DiskStation kernel: [2567332.460595] Plex Media Serv[29191]: segfault at 0 ip 00007f04d4ad6192 sp 00007f04cc371ef0 error 4 in Plex Media Server[7f04d4188000+99c000]
2021-12-11T17:04:12+03:00 DiskStation kernel: [147864.474098] Plex Media Serv[25320]: segfault at 0 ip 00007fe3f830b192 sp 00007fe3f193eef0 error 4 in Plex Media Server[7fe3f79bd000+99c000]
2021-12-14T16:33:04+03:00 DiskStation kernel: [405188.440467] Plex Media Serv[2599]: segfault at 0 ip 00007fa9317c3192 sp 00007fa929b92ef0 error 4 in Plex Media Server[7fa930e75000+99c000]
2021-08-03T16:34:01
ip 00007f2e73a31192 - [7f2e730e3000] = 94E192
<...>
ip 00007fbcbc06e192 - [7fbcbb72000] = 94E192
ip 00007ff0822a6192 - [7ff081958000] = 94E192
<...>
2021-12-11
ip 00007fe3f830b192 - 7fe3f79bd000 = 94E192
2021-12-14
ip 00007fa9317c3192 - 7fa930e75000 = 94E192
addr2line -e "/volume1/@appstore/Plex Media Server/Plex Media Server" -fCi 0x94E192
void boost::asio::detail::epoll_reactor::schedule_timer<boost::asio::time_traits<boost::posix_time::ptime> >(boost::asio::detail::timer_queue<boost::asio::time_traits<boost::posix_time::ptime> >&, boost::asio::time_traits<boost::posix_time::ptime>::time_type const&, boost::asio::detail::timer_queue<boost::asio::time_traits<boost::posix_time::ptime> >::per_timer_data&, boost::asio::detail::wait_op*)
??:?
from some docs:
schedule_timer does schedule a new operation in the given timer queue to expire at the specified absolute time.
Is there a schedule_timer call that can be connected to the inconspicuous actions logged from the iPad just before the crash?
