Conversion Failed Transcoder Crash

server-linux

#1

Hello,

Whenever i try to watch a video I am receiving the transcoder failure error.

Server: Feral Hosting
Plex Version: 1.9.1.4272
Videos Effected: All, error occurs faster on videos with subtitles but has also occurred without subs.

Solutions attempted: remove and recreate librarys, pkill, full reinstall

Logs attached (debug level, not Verbose)

Please advise.


#2

You are experiencing a common problem with Feral Hosting.

They build their own Linux kernel which conveniently disables ./ removes several of the needed system calls.

As such, PMS cannot send signals to its processes and processes do not exit cleanly.

This is shown in your logs where PMS is trying to stop processes but cannot

Your VPS has a configuration error which only they can resolve.

```
ep 25, 2017 10:10:32.713 [0x7fb37a458800] INFO - Killing process: Plex EAE Service (pid: 958)
Sep 25, 2017 10:10:32.713 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex DLNA Server (pid: 1856)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 1859)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.agents.localmedia
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex DLNA Server (pid: 6457)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 6459)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex DLNA Server (pid: 7286)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 7288)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 7323)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex DLNA Server (pid: 7363)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.plugins.WebTools
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex EAE Service (pid: 10783)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex DLNA Server (pid: 10866)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 10867)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex DLNA Server (pid: 11771)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 11772)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 12036)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Plug-in com.plexapp.system

-- list truncated --
```


#3

Hello @ChuckPA ,

Feral Support has agreed tt look into changing the Kernal settings however they are not 100% sure what dependencies are required, is there some kind of documentation or notes you've made about what they need to change to resolve the issue?

Kind regards,

James


#4

Tell them to implement Linux signals per Linux specification.

Do not mask when one process sends another a signal; specifically signals, 9, 11, and 15.

9 = kill
11 = induce a segmentation violation error
15 = terminate


#5

I'm another Feral user, and I am not sure I understand/agree with the claim that this is due to the lack of support for Linux signals. I use both kill, and kill -9 fairly frequently without any issues -- and in fact many of their support docs document using those commands to clean up processes, and I have never seen unexpected behavior with them before -- so they are at least partially supported.

If it was an issue of not being able to kill/restart processes, why does the transcoding issue occur immediately after a restart of Plex? There should be no stale processes 20 seconds after starting Plex, right?

Additionally, Feral Hosting does not offer VPS -- they offer Linux accounts on a shared server. They use network namespaces and port forwarding to give each user their own plex instance.

Looking at those log files, I see that while it is repeatedly trying to kill processes (immediately after starting), the PIDs of the processes change each time. If I had to guess, Plex is trying to nuke the processes of other users, and obviously that wont work:
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 7288)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 7323)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 10867)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 11772)
Sep 25, 2017 10:10:32.714 [0x7fb37a458800] INFO - Killing process: Plex Tuner Service (pid: 12036)

It's pretty clear that these are different PIDs -- so the signals are either working, or plex is trying to kill processes it has no right to kill (or less likely, Plex is repeatedly spawning new processes and then immediately trying to kill them in a suicidal rage).


#6

@ashmandias

They have modified the kernel. Do a uname -a. You will discover they have rolled their own kernel. Regardless whether a VPS or rented server, they do something which cannot be supported.


#7

Rolling your own kernel is not the same as removing support for basic Linux signals – and your explanation as to why it does not work does not match the evidence. What kernel features are needed that Feral does not currently use – and what evidence did you use to come to that conclusion?

The logs that you selected above simply appear to be the result of not following best practices on Plex’s part – you should not be trying to kill other user’s processes – and ‘DEBUG’ level logging should be collecting return codes/error messages from trying to kill the processes.

I’m not saying Feral is not doing something wrong, I am just saying that you seem to have leaped to a conclusion earlier, and are not able to back that up.

What exactly is Feral doing that is not supported? Rolling your own kernel (especially for odd hardware) is not uncommon, and at a bare minimum, the requirements for kernel support should be documented.


#8

@ashmandias

PMS, just like any other process, has the right and Linux-granted privilege to kill its child processes.
When the ABI presented to processes is non-standard and otherwise untested, the results are unpredictable.

The distributions PMS is supported on is so listed on the downloads page.

What distribution does Feral Hosting use? What Linux kernel core does Feral Hosting use as their baseline? What GLIBC version ?


#9

@ChuckPA said:
@ashmandias

PMS, just like any other process, has the right and Linux-granted privilege to kill its child processes.

You are 100% correct that PMS has the right to kill it’s child processes… The issue you are seeing in the logs is that Plex is trying to kill non-child processes. It is attempting to kill processes owned by other users. It has no right to kill processes running as other users.

When the ABI presented to processes is non-standard and otherwise untested, the results are unpredictable.

That is fine – however the thing you are claiming is ‘non-standard’ is not true. You claimed that Feral does not support the full suite of Linux process signals – and it does.

The distributions PMS is supported on is so listed on the downloads page.

What distribution does Feral Hosting use?

Debian Jessie (8.8) x64 – listed on the download page.

What Linux kernel core does Feral Hosting use as their baseline?

Linux 4.12.3-feral #2 SMP Sat Jul 22 14:53:32 BST 2017 x86_64 GNU/Linux

What GLIBC version ?

Debian GLIBC 2.19-18+deb8u10

There may be issues with Feral’s set up, I am just trying to point out that it is not the lack of signal support. Ultimately, I would love to find out what it is that Plex requires that Feral does not provide – and then work with the appropriate parties to get the issue fixed. So far, the only issue I am seeing is the lack of sanity checking prior to killing processes, which is a Plex issue, and not a Feral one.


#10

@ashmandias said:

@ChuckPA said:
@ashmandias

PMS, just like any other process, has the right and Linux-granted privilege to kill its child processes.

You are 100% correct that PMS has the right to kill it’s child processes… The issue you are seeing in the logs is that Plex is trying to kill non-child processes. It is attempting to kill processes owned by other users. It has no right to kill processes running as other users.

When the ABI presented to processes is non-standard and otherwise untested, the results are unpredictable.

That is fine – however the thing you are claiming is ‘non-standard’ is not true. You claimed that Feral does not support the full suite of Linux process signals – and it does.

The distributions PMS is supported on is so listed on the downloads page.

What distribution does Feral Hosting use?

Debian Jessie (8.8) x64 – listed on the download page.

What Linux kernel core does Feral Hosting use as their baseline?

Linux 4.12.3-feral #2 SMP Sat Jul 22 14:53:32 BST 2017 x86_64 GNU/Linux

What GLIBC version ?

Debian GLIBC 2.19-18+deb8u10

There may be issues with Feral’s set up, I am just trying to point out that it is not the lack of signal support. Ultimately, I would love to find out what it is that Plex requires that Feral does not provide – and then work with the appropriate parties to get the issue fixed. So far, the only issue I am seeing is the lack of sanity checking prior to killing processes, which is a Plex issue, and not a Feral one.

Prove to me PMS is attempting to kill processes of other users. It is only aware of processes it forks. Only Feral Hosting’s kernel has the problem you are seeing so it can not be a global Debian problem.

You show their uname entry which is clearly their own kernel build. Equally as clearly as this is Fedora’s own kernel build

[chuck@lizum ~.2]$ uname -a
Linux lizum.hessen.lan 4.13.4-200.fc26.x86_64 #1 SMP Thu Sep 28 20:46:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[chuck@lizum ~.3]$ 

Current GLIBC is:

[chuck@lizum ~.3]$ rpm -qa | grep glibc
glibc-common-2.25-10.fc26.x86_64
glibc-headers-2.25-10.fc26.x86_64
glibc-langpack-en-2.25-10.fc26.x86_64
glibc-2.25-10.fc26.i686
glibc-2.25-10.fc26.x86_64
glibc-devel-2.25-10.fc26.x86_64
glibc-all-langpacks-2.25-10.fc26.x86_64
[chuck@lizum ~.4]$ 

https://sourceware.org/glibc/wiki/Glibc%20Timeline

I have asked Engineering for finite version minimum on glibc.


#11

Prove to me PMS is attempting to kill processes of other users.

Look at the logs provided above. The PID being killed for the processes is changing on each attempt to kill a process. Why would Plex be re-spawning processes constantly?

That said, I’ll put up some actual information:

for PID in $(grep -i kill Library/Logs/Plex\ Media\ Server/Plex\ Media\ Server* | grep pid | cut -d: -f6 | tr -d ‘)’ ); do ps aux | grep -v grep | grep " $PID "; done
redacted2 20501 0.1 0.0 1868480 51892 pts/65 SNl+ Jul29 110:47 Plex Plug-in [com.plexapp.system] /media/sdm1/home/redacted2/private/plex/usr/lib/plexmediaserver/Resources/Plug-ins-5089475/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 0.9.16.6.1993-5089475 /media/sdm1/home/redacted2/private/plex/usr/lib/plexmediaserver/Resources/Plug-ins-5089475/System.bundle
redacted1 24913 0.1 0.0 1798112 48184 ? SNl Jul29 130:46 Plex Plug-in [com.plexapp.system] /media/sds1/home/redacted1/private/plex/usr/lib/plexmediaserver/Resources/Plug-ins-6f64a8d/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.1.3.2700-6f64a8d /media/sds1/home/redacted1/private/plex/usr/lib/plexmediaserver/Resources/Plug-ins-6f64a8d/System.bundle
redacted3+ 3906 0.0 0.0 21400 2512 pts/49 S+ Sep08 5:18 Plex EAE Service
redacted3+ 13404 0.1 0.0 1660400 64144 pts/49 SNl+ Aug31 75:17 Plex Plug-in [com.plexapp.system] /media/sdj1/home/redacted3ast/private/plex/usr/lib/plexmediaserver/Resources/Plug-ins-c789b3fbb/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.8.1.4139-c789b3fbb /media/sdj1/home/redacted3ast/private/plex/usr/lib/plexmediaserver/Resources/Plug-ins-c789b3fbb/System.bundle

I can provide more logs if needed – but clearly, it is exactly s stated – MY Plex logs log MY Plex trying to kill pids NOT owned by me.

It is only aware of processes it forks.

Clearly untrue. You can also verify that this is not true by reviewing the full logs provided above – in those logs the FIRST thing that Plex does is start killing processes after being started – there is NO log of those processes being forked.

Only Feral Hosting’s kernel has the problem you are seeing so it can not be a global Debian problem.

You seem obsessed with defending your claim that it is the Kernel – it is FAR more likely that THIS issue you are seeing in the logs is actually a result of Feral running more than one Plex install on a single server – that fits all the evidence (the log messages, the pids of other user’s processes, the multiple kills to processes with the same name but different pids, the fact that those signals you claim do not exist actually work when tested by hand, and the fact that it’s absurd to think that Plex would be the ONLY software to have an issue with breaking POSIX compliance in such a fundamental way)

You show their uname entry which is clearly their own kernel build. Equally as clearly as this is Fedora’s own kernel build

[chuck@lizum ~.2]$ uname -a
Linux lizum.hessen.lan 4.13.4-200.fc26.x86_64 #1 SMP Thu Sep 28 20:46:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[chuck@lizum ~.3]$ 

Yes, but that literally means nothing without further evidence. You claim that their kernel lacks basic POSIX support – and your only evidence is that they have a custom kernel. I have pointed out a much better hypothesis that fits all the evidence.


Current GLIBC is:

[chuck@lizum ~.3]$ rpm -qa | grep glibc
glibc-common-2.25-10.fc26.x86_64
glibc-headers-2.25-10.fc26.x86_64
glibc-langpack-en-2.25-10.fc26.x86_64
glibc-2.25-10.fc26.i686
glibc-2.25-10.fc26.x86_64
glibc-devel-2.25-10.fc26.x86_64
glibc-all-langpacks-2.25-10.fc26.x86_64
[chuck@lizum ~.4]$

https://sourceware.org/glibc/wiki/Glibc%20Timeline

I have asked Engineering for finite version minimum on `glibc`. 

That’s the current on your RH based distro. Debian tends to do far more back porting than Fedora does.

That said, if the issue IS an outdated glibc, that is progress. If Plex is going to drop support for Debian Jessie, that sucks, but at least it’s an answer.


#12

Putting it in someone’s home directory does NOT make it their process.

Redacting the usernames proves nothing.

There is no official support for Debian. How it was listed is unknown. It is being removed as I write here.


#13

@ChuckPA said:
Putting it in someone’s home directory does NOT make it their process.

No one said it did? The first column of ps aux is the username of the person owning the process. It also happens to be that the binaries are in their home dirs.

Redacting the usernames proves nothing.

Why would you assume that that was a claim I made? I redacted their usernames (with unique values) for privacy reasons.

There is no official support for Debian. How it was listed is unknown. It is being removed as I write here.

That’s a shame.


#14

Looks like Debian 8 is still listed…


#15

May be but it’s still not supported


#16

Could you clarify what you mean by that? Plex provides a binary download for Debian 8 on their main download site, but that somehow doesn’t imply Plex runs on Debian 8?


#17

@ChuckPA said:

There is no official support for Debian. How it was listed is unknown. It is being removed as I write here.


#18

Plex provides the .deb file for Ubuntu

It coincidentally runs on Debian 8+ but has not been tested and therefore is not supported.
It runs because a) Ubuntu uses Debian’s package manager (dpkg) and b) as of Debian 8, Debian uses systemd as the control program.

I’ve filed all the change paperwork to have it removed from the Downloads page.


#19

I, and most rational people, see that currently Plex provides a .deb for Ubuntu and Debian – the downloads page makes no differentiation between the support for Ubuntu/Debian and any other OS listed. What you are stating does not reflect the current condition of the download page. While it is sad to see you dropping support for Debian, at least the website will be accurate going forward.


#20

Most rational people would accept what I say and let it go.

It was never supported. A Ninja, out of the kindness of his heart, provided the repackaging for Debian 7. One cannot speak of something being dropped which was never supported and shouldn’t have been listed in the first place (Oh, to be human and make mistakes)