If you have not already, we suggest setting your Plex username to something else rather than email which is displayed on your posts in forum. You can change the username at https://app.plex.tv/desktop#!/account
Welcome to our forums! Please take a few moments to read through our Community Guidelines (also conveniently linked in the header at the top of each page). There, you'll find guidelines on conduct, tips on getting the help you may be searching for, and more!

How to Debug the reason Plex Media Server seems "Stuck" on startup on cent 7 64 bit?

bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

Hi all.

Running cent 7 64bit in a vm and had a crash that caused some filesystem corruption on my application support folder. fsck repaired it, and the only files lost were 3 log files from the Logs dir.

Now, systemctl thinks that plex is running, but it is not in theory.

The process is indeed running: plex 982 0.2 0.3 337256 30972 ? Ssl 12:24 0:00 /usr/lib/plexmediaserver/Plex Media Server
but it does not seem to be doing anything. It does not update any logs in the application support directory, nor is it generating anything to journalctl or systemd.

This is a little frustrating as something is clearly wrong. If I try running plex media server manually with the proper environment vars exported, it just hangs. Is there a way to pass it into verbose mode or debug mode or something ?

The above process is the only plex process that starts instead of the usual group of them, and no logs are updated.

Best Answers

Answers

  • ChuckPAChuckPA Posts: 19,377Members, Plex Pass, Plex Ninja, Plex Team Member Plex Team Member

    Can you show me your environment? You should have the default service /lib/systemd/system/plexmediaserver.service and /var/lib/plexmediaserver/Library .

    What you're presenting sounds like a problem with PMS's home directory /var/lib/plexmediaserver

    What does the journalctl -xe | grep -i plex show? what errors?

    Please DISABLE Verbose logging until requested

    Please search before posting

    Primary support forums: Linux, Synology, and QNAP

    Please remember to report back. This benefits others.

    Useful links

     Installation and Basic Setup |  Media Preparation (How to name your media files)  |  Linux Permissions 

     Handling TV Specials | Handling Movie extras  |  Nas Compatibility List

     Reporting Plex Server issues | Plex Media Server FAQ | Linux Tips

     

    Other useful guides: Local Subtitles | The Plex "dance" | Synology FAQ | PMS Release Announcements

    Hosts: Fedora, QNAP, Synology, most Linux distros in VM

    No technical support via PM unless offered

    Please remember to mark the appropriate answer(s) which solved your issue.

     
  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

    Sure, I am running some non standard locations, but attached are the environments.

    Systemd rpm unit file:

    [Unit]
    Description=Plex Media Server for Linux
    After=network.target
    
    [Service]
    Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/var/lib/plexmediaserver/Library/Application Support"
    Environment=PLEX_MEDIA_SERVER_HOME=/usr/lib/plexmediaserver
    Environment=PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS=6
    Environment=PLEX_MEDIA_SERVER_TMPDIR=/tmp
    Environment=LC_ALL=en_US.UTF-8
    Environment=LANG=en_US.UTF-8
    ExecStartPre=/bin/sh -c '/usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}"'
    ExecStart=/bin/sh -c 'LD_LIBRARY_PATH=/usr/lib/plexmediaserver "/usr/lib/plexmediaserver/Plex Media Server"'
    Type=simple
    User=plex
    Group=plex
    Restart=on-failure
    RestartSec=5
    StartLimitInterval=60s
    StartLimitBurst=3
    
    [Install]
    WantedBy=multi-user.target
    

    My override systemd unit file to override some locations:

    # cat /etc/systemd/system/plexmediaserver.service.d/override.conf 
    [Service]
    Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/mnt/scratch/plexmediaserver/Library/Application Support"
    Environment=PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS=12
    Environment=PLEX_MEDIA_SERVER_TMPDIR=/mnt/scratch/tmp
    

    As one can see, all I am doing is moving the tmp dir and application support dir to an alternative mount point since / is a smaller filesystem. /mnt/scratch is the filesystem that had some corruption this AM but was fsck'd successfully.

    Output of journalctl -xe | grep -i plexmed as my system is named "plexxy" and thus that grep matches everything otherwise. :)

    # journalctl -xe | grep -i plexmed
    -- Subject: Unit plexmediaserver.service has begun start-up
    -- Unit plexmediaserver.service has begun starting up.
    -- Subject: Unit plexmediaserver.service has finished start-up
    -- Unit plexmediaserver.service has finished starting up.
    Jan 11 12:46:18 plexxy polkitd[707]: Operator of unix-process:3694:130706 successfully authenticated as unix-user:bezerker to gain ONE-SHOT authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.29 [systemctl stop plexmediaserver] (owned by unix-user:bezerker)
    -- Subject: Unit plexmediaserver.service has begun shutting down
    -- Unit plexmediaserver.service has begun shutting down.
    -- Subject: Unit plexmediaserver.service has finished shutting down
    -- Unit plexmediaserver.service has finished shutting down.
    -- Subject: Unit plexmediaserver.service has begun start-up
    -- Unit plexmediaserver.service has begun starting up.
    -- Subject: Unit plexmediaserver.service has finished start-up
    -- Unit plexmediaserver.service has finished starting up.
    

    An strace of the process shows the following. Again, no logs inside the Application support folder appear being updated and the journal isnt helpful either.

    stat("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0                                                                                    
    access("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server//plexmediaserver.pid", R_OK) = 0                                                                                                  
    stat("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server//plexmediaserver.pid", {st_mode=S_IFREG|0644, st_size=3, ...}) = 0                                                                  
    open("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server//plexmediaserver.pid", O_RDONLY) = 9                                                                                                
    read(9, "982", 8191)                    = 3          
    close(9)                                = 0          
    kill(982, SIG_0)                        = -1 ESRCH (No such process)                                      
    stat("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0                                                                                    
    stat("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0                                                                                    
    stat("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0                                                                                    
    open("/mnt/scratch/plexmediaserver/Library/Application Support/Plex Media Server//plexmediaserver.pid", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 9                                                                          
    write(9, "3721", 4)                     = 4          
    close(9)                                = 0          
    readlink("/proc/self/exe", "/usr/lib/plexmediaserver/Plex Me"..., 4096) = 42                              
    futex(0x7f516cf8e3c4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x7f516cf8e400, 2) = 1                    
    futex(0x7f516cf8e3c4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x7f516cf8e400, 4) = 1                    
    futex(0x7f516cf8e400, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)         
    futex(0x7f516cf8e400, FUTEX_WAKE_PRIVATE, 1) = 0     
    futex(0x7f516cf8e3c4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x7f516cf8e400, 6) = 1                    
    futex(0x7f516cf8e3c4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x7f516cf8e400, 8) = 1                    
    futex(0x7f516cf8e3c4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x7f516cf8e400, 10) = 1                   
    futex(0x7f516cf8e3c4, FUTEX_WAIT_PRIVATE, 12, NULL   
    

    As one can see, it sees the application support folder, writes the pid file, and then just chills out doing nothing.

    I agree something likely seems corrupt inside the App support dir but I have a lot of libraries etc configured and do not want to wipe it all unnecessarily. I wish this at least gave me a hint about what it's upset about. :)

  • ChuckPAChuckPA Posts: 19,377Members, Plex Pass, Plex Ninja, Plex Team Member Plex Team Member

    Thanks for that.

    You can't put the PMS Library on a network share which doesn't support local_lock=posix.
    To be clear, PMS doesn't officially support any network mounts for the PMS metadata because the database is there and it uses futex/mutex locks.

    In Linux Tips at the top of this forum, I show how to move the directory.

    The safest is to a physical drive on host, e.g /home/plexdir (which I show)
    The next safest, but not guaranteed, is local_lock=posix. I have done it with NFSv4 only. V3 and V2 do not support this.

    Please DISABLE Verbose logging until requested

    Please search before posting

    Primary support forums: Linux, Synology, and QNAP

    Please remember to report back. This benefits others.

    Useful links

     Installation and Basic Setup |  Media Preparation (How to name your media files)  |  Linux Permissions 

     Handling TV Specials | Handling Movie extras  |  Nas Compatibility List

     Reporting Plex Server issues | Plex Media Server FAQ | Linux Tips

     

    Other useful guides: Local Subtitles | The Plex "dance" | Synology FAQ | PMS Release Announcements

    Hosts: Fedora, QNAP, Synology, most Linux distros in VM

    No technical support via PM unless offered

    Please remember to mark the appropriate answer(s) which solved your issue.

     
  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

    Hrm, this is a standard ext4 device, it should support posix locks. Media is in an nfs mount though yes.

    I'll dig further. That's interesting to know. Thanks.

  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

    Yep. Just confirmed. My library/appsupport is NOT on a network share. (Rather, let me clarify, its a virtual disk in proxmox, and shows as a raw block device to the OS with an ext4 fs). In theory, this should have no issue with locks.

    Thus, I'm confused why we think it can't get a lock. The only thing that is mounted NFS directly is the actual directory the movies are in, which seems to be a supported configuration from what I see.

    Very bizarre. I'm going to try changing the directory to a "fresh" one just to see if it spins up with a default configuration, or not.

  • ChuckPAChuckPA Posts: 19,377Members, Plex Pass, Plex Ninja, Plex Team Member Plex Team Member

    That changes just about everything.

    Proxmox has VERY specific demands.

    Look at the top of the Linux forum. In Linux Tips you'll find:

    https://forums.plex.tv/discussion/301781/pms-installation-guide-when-using-a-proxmox-5-1-lxc-container

    Please DISABLE Verbose logging until requested

    Please search before posting

    Primary support forums: Linux, Synology, and QNAP

    Please remember to report back. This benefits others.

    Useful links

     Installation and Basic Setup |  Media Preparation (How to name your media files)  |  Linux Permissions 

     Handling TV Specials | Handling Movie extras  |  Nas Compatibility List

     Reporting Plex Server issues | Plex Media Server FAQ | Linux Tips

     

    Other useful guides: Local Subtitles | The Plex "dance" | Synology FAQ | PMS Release Announcements

    Hosts: Fedora, QNAP, Synology, most Linux distros in VM

    No technical support via PM unless offered

    Please remember to mark the appropriate answer(s) which solved your issue.

     
  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

    Interesting. It's not ina container but a kvm vm , but I'll poke. This has been functioning inside this virtual machine for almost a year now. Only issue was I had a power hit this morning and the fs became corrupted. I'll poke around though.

    Hmmm. Much appreciate the assistance by the way, hopefully I can find the root cause.

  • ChuckPAChuckPA Posts: 19,377Members, Plex Pass, Plex Ninja, Plex Team Member Plex Team Member

    As soon as you mentioned Proxmox, the rules still changed. :)

    Did you remember to allow the appropriate system calls / features to exit the VM? I know Proxmox is extremely secure and grants you such levels of control not normally found in the typical VM.

    Rephrased: Maybe it's a little too tight :)

    Please DISABLE Verbose logging until requested

    Please search before posting

    Primary support forums: Linux, Synology, and QNAP

    Please remember to report back. This benefits others.

    Useful links

     Installation and Basic Setup |  Media Preparation (How to name your media files)  |  Linux Permissions 

     Handling TV Specials | Handling Movie extras  |  Nas Compatibility List

     Reporting Plex Server issues | Plex Media Server FAQ | Linux Tips

     

    Other useful guides: Local Subtitles | The Plex "dance" | Synology FAQ | PMS Release Announcements

    Hosts: Fedora, QNAP, Synology, most Linux distros in VM

    No technical support via PM unless offered

    Please remember to mark the appropriate answer(s) which solved your issue.

     
  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

    Indeed, Plex and proxmox are fine. I confirmed it's something in the App support folder by renaming it temporarily and restarting. Works fine as a fresh clean install. Now I need to figure how to salvage what I can from the existing Library.

  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass
    Accepted Answer

    Looking more and more, this may be a corrupt Preferences.xml and other files. Going to dig deeper.

  • bezerkerbezerker Posts: 9Members, Plex Pass Plex Pass

    Was able to restore a proxmox backup of my vm I made from the 6th. Confirmed something got corrupted in preferences.xml (it was binary data). Replacing the file itself still didn't resolve it so I can only assume there was other corruption within the FS. Attempting to start fresh worked and plex started with no config, so as a result I decided that I'll just try a restore. Restored a backup of the block devices and all is working in my old configuration again.

    Much thanks for all of the help ChuckPA.

Sign In or Register to comment.