Transcoder fails when `/transcode` is on a network share

@kopfpilot said:
@sa2000: thank you :slight_smile:

Btw. the transcoding itself seem to happen in the background: the cpu load rises and the temporary EasyAudioEncode file keeps beeing replaced iterativly with a growing index number (caee72199bcdf59e-com-plexapp-android_834-1-n.ec3, where n is the index number), until playback is stopped.

Without a wav file appearing in the temp conversion directory ?

I can see PID 515 under user ‘abc’ running - but it is a plex transcoded job and not EasyAudioEncoder job. Probably both would be needed

Will have to wait for Linux experts - I have asked @chuckPa to step in

@sa2000 said:

@kopfpilot said:
@sa2000: thank you :slight_smile:

Btw. the transcoding itself seem to happen in the background: the cpu load rises and the temporary EasyAudioEncode file keeps beeing replaced iterativly with a growing index number (caee72199bcdf59e-com-plexapp-android_834-1-n.ec3, where n is the index number), until playback is stopped.

Without a wav file appearing in the temp conversion directory ?

At least no WAV file was listed when i executed ls. Though, it might have been created and renamed during my ls executions. Can’t tell if there is a race condition that i am just not able to see.

I can see PID 515 under user ‘abc’ running - but it is a plex transcoded job and not EasyAudioEncoder job. Probably both would be needed

Will have to wait for Linux experts - I have asked @chuckPa to step in

I would have expected to see it as well, but on the otherside files are continously written in the temp folder


@kopfpilot said:

@sa2000 said:

@kopfpilot said:
@sa2000: thank you :slight_smile:

Btw. the transcoding itself seem to happen in the background: the cpu load rises and the temporary EasyAudioEncode file keeps beeing replaced iterativly with a growing index number (caee72199bcdf59e-com-plexapp-android_834-1-n.ec3, where n is the index number), until playback is stopped.

Without a wav file appearing in the temp conversion directory ?

At least no WAV file was listed when i executed ls. Though, it might have been created and renamed during my ls executions. Can’t tell if there is a race condition that i am just not able to see.

I can see PID 515 under user ‘abc’ running - but it is a plex transcoded job and not EasyAudioEncoder job. Probably both would be needed

Will have to wait for Linux experts - I have asked @chuckPa to step in

I would have expected to see it as well, but on the otherside files are continously written in the temp folder


whilst waiting for @chuckpa, you could try this

Rename or move out of the directory this directory
/config/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-133-linux-ubuntu-x86_64

Then restart the server

Wait 5 minutes and start to stream a True HD or EAC3 media and when you get the failure, capture the logs and the list of codecs and tmp directories - if you can watch the codecs and tmp areas to see what gets created

hi all, am also on linux, was having same issues, seems like trouble with easyaudioencoder? also upgraded to latest plex 1.5.5 same issue. btw i have nothing on network shares, all on local machine.
See here for same problem on windows https://forums.plex.tv/discussion/267140/plex-wont-play-video-files-after-update
Downgrading to any version below 1.5 makes the transcoding work again.

I tried what you suggested, renamed the directory and then tried, i get this in the directories:

ls -al /tmp/plex-transcode-xijlsz0i2fkm46wl5ylw8iy7-3f347b2d-0db5-40f3-a0ca-440e9bb9947f/
total 0
drwxr-xr-x  2 plex plex  80 Apr 13 14:47 .
drwxrwxrwt 20 root root 460 Apr 13 14:47 ..
-rw-r--r--  1 plex plex   0 Apr 13 14:47 init-stream0.m4s
-rw-r--r--  1 plex plex   0 Apr 13 14:47 init-stream1.m4s

before for the same video file there were only the *.ac3 files and in the logs you see it can’t locate the *.wav files:

DEBUG - Job running: EAE_ROOT='/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder' FFMPEG_EXTERNAL_LIBS='/var/lib/plex/Plex\ Media\ Server/Codecs/57aa73f-1206-linux-ubuntu-x86_64/' XDG _CACHE_HOME='/var/lib/plex/Plex Media Server/Cache/' XDG_DATA_HOME='/opt/plexmediaserver/Resources/' X_PLEX_TOKEN='xxxxxxxxxxxxxxxxxxxxver/Plex Transcoder' '-codec:0' 'h264' '-codec:1' 'ac3_eae' '-eae_prefix:1' 'rfv9ke21l7it4523jtmt786y_' '-noaccurate_seek' '-i' '/Movies/Passengers.720p BRRip x264 AC3.mkv' '-map' '0:0' '-codec:0' 'copy' '-map' '0:1' '-codec:1' 'aac' '-ar:1' '48000' '-channel_layout:1' 'stereo' '-b:1' '258k' '-f' 'dash' '-use_timeline' '0' '-min_seg_duration' '5000000' '-skip_to_segment' '1' '-time_delta' '0.0625' '-avoid_negative_ts' 'disabled' '-map_metadata' '-1' '-map_chapters' '-1' '-movflags' '+faststart' 'dash' '-start_at_zero' '-copyts' '- vsync' 'cfr' '-y' '-nostats' '-loglevel' 'quiet' '-loglevel_plex' 'error' '-progressurl' 'http://127.0.0.1:32400/video/:/transcode/session/rfv9ke21l7it4523jtmt786y/progress' Apr 13, 2017 14:45:38.001 [0x7f05a9fff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder/Convert to WAV (to 8ch or l ess)/rfv9ke21l7it4523jtmt786y_22770-1-0.wav' Apr 13, 2017 14:45:38.003 [0x7f05a2ffe700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] error reading output Apr 13, 2017 14:45:38.005 [0x7f05b97ff700] ERROR - [Transcoder] Error while decoding stream #0:1: Input/output error Apr 13, 2017 14:45:41.002 [0x7f05b97ff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder/Convert to WAV (to 8ch or l ess)/rfv9ke21l7it4523jtmt786y_22770-1-1.wav' Apr 13, 2017 14:45:41.003 [0x7f05a9fff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] error reading output Apr 13, 2017 14:45:41.004 [0x7f05a2ffe700] ERROR - [Transcoder] Error while decoding stream #0:1: Input/output error Apr 13, 2017 14:45:44.001 [0x7f05a9fff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder/Convert to WAV (to 8ch or l ess)/rfv9ke21l7it4523jtmt786y_22770-1-2.wav'

@jody2 said:
hi all, am also on linux, was having same issues, seems like trouble with easyaudioencoder? also upgraded to latest plex 1.5.5 same issue. btw i have nothing on network shares, all on local machine.
See here for same problem on windows https://forums.plex.tv/discussion/267140/plex-wont-play-video-files-after-update
Downgrading to any version below 1.5 makes the transcoding work again.

I tried what you suggested, renamed the directory and then tried, i get this in the directories:

ls -al /tmp/plex-transcode-xijlsz0i2fkm46wl5ylw8iy7-3f347b2d-0db5-40f3-a0ca-440e9bb9947f/
total 0
drwxr-xr-x  2 plex plex  80 Apr 13 14:47 .
drwxrwxrwt 20 root root 460 Apr 13 14:47 ..
-rw-r--r--  1 plex plex   0 Apr 13 14:47 init-stream0.m4s
-rw-r--r--  1 plex plex   0 Apr 13 14:47 init-stream1.m4s

before for the same video file there were only the *.ac3 files and in the logs you see it can’t locate the *.wav files:

DEBUG - Job running: EAE_ROOT='/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder' FFMPEG_EXTERNAL_LIBS='/var/lib/plex/Plex\ Media\ Server/Codecs/57aa73f-1206-linux-ubuntu-x86_64/' XDG _CACHE_HOME='/var/lib/plex/Plex Media Server/Cache/' XDG_DATA_HOME='/opt/plexmediaserver/Resources/' X_PLEX_TOKEN='xxxxxxxxxxxxxxxxxxxxver/Plex Transcoder' '-codec:0' 'h264' '-codec:1' 'ac3_eae' '-eae_prefix:1' 'rfv9ke21l7it4523jtmt786y_' '-noaccurate_seek' '-i' '/Movies/Passengers.720p BRRip x264 AC3.mkv' '-map' '0:0' '-codec:0' 'copy' '-map' '0:1' '-codec:1' 'aac' '-ar:1' '48000' '-channel_layout:1' 'stereo' '-b:1' '258k' '-f' 'dash' '-use_timeline' '0' '-min_seg_duration' '5000000' '-skip_to_segment' '1' '-time_delta' '0.0625' '-avoid_negative_ts' 'disabled' '-map_metadata' '-1' '-map_chapters' '-1' '-movflags' '+faststart' 'dash' '-start_at_zero' '-copyts' '- vsync' 'cfr' '-y' '-nostats' '-loglevel' 'quiet' '-loglevel_plex' 'error' '-progressurl' 'http://127.0.0.1:32400/video/:/transcode/session/rfv9ke21l7it4523jtmt786y/progress' Apr 13, 2017 14:45:38.001 [0x7f05a9fff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder/Convert to WAV (to 8ch or l ess)/rfv9ke21l7it4523jtmt786y_22770-1-0.wav' Apr 13, 2017 14:45:38.003 [0x7f05a2ffe700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] error reading output Apr 13, 2017 14:45:38.005 [0x7f05b97ff700] ERROR - [Transcoder] Error while decoding stream #0:1: Input/output error Apr 13, 2017 14:45:41.002 [0x7f05b97ff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder/Convert to WAV (to 8ch or l ess)/rfv9ke21l7it4523jtmt786y_22770-1-1.wav' Apr 13, 2017 14:45:41.003 [0x7f05a9fff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] error reading output Apr 13, 2017 14:45:41.004 [0x7f05a2ffe700] ERROR - [Transcoder] Error while decoding stream #0:1: Input/output error Apr 13, 2017 14:45:44.001 [0x7f05a9fff700] ERROR - [Transcoder] [ac3_eae @ 0x1c3ce80] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942/EasyAudioEncoder/Convert to WAV (to 8ch or l ess)/rfv9ke21l7it4523jtmt786y_22770-1-2.wav'

The windows issue you linked to was specific to windows and relates to long user account names and was fixed in release 1.5.5. The symptom is the same but it is not the same problem

Could you see what directories and files get created within '/tmp/pms-ffc80b3d-a1cf-46f1-9640-f8fc3eff4942 and also do an ls -ail for this and all subdirectories within

Could you also do the ls -ail for the all the codecs directories and files within
'var/lib/plex/Plex Media Server/Codecs

I tried what you suggested, renamed the directory and then tried, i get this in the directories:

which directory did you rename ?
and what did you end up in the codecs and /tmp areas when streaming eac3 / true hd media

Here we go :slight_smile:

I thought my output would look identical to what jody2 postet, but the EAE temp files do have different file endings.

On some of the ls commands i piped the output directly into a file, so the command itself is mssing.
In the PMS-Logfile obtainging codecs can be found between lines 101 and 132.

The files ending with post_delete were made after delete of folders and restart of the container.
The files ending with post_transcode where made during playback.

Is it possible that the created m3u files contain entries with wrong file extensions? Where can i find these files?

There is no change in the process list during playback. A transcoder process is summoned, but the EAE process is still not listed.

In docker logs, the string " Dolby, Dolby Digital, Dolby Digital Plus, Dolby TrueHD and the double D symbol are trademarks of Dolby Laboratories." is logged for each transcoding playback once.

@kopfpilot said:
Here we go :slight_smile:

I thought my output would look identical to what jody2 postet, but the EAE temp files do have different file endings.

On some of the ls commands i piped the output directly into a file, so the command itself is mssing.
In the PMS-Logfile obtainging codecs can be found between lines 101 and 132.

The files ending with post_delete were made after delete of folders and restart of the container.
The files ending with post_transcode where made during playback.

Is it possible that the created m3u files contain entries with wrong file extensions? Where can i find these files?

There is no change in the process list during playback. A transcoder process is summoned, but the EAE process is still not listed.

In docker logs, the string " Dolby, Dolby Digital, Dolby Digital Plus, Dolby TrueHD and the double D symbol are trademarks of Dolby Laboratories." is logged for each transcoding playback once.

Thanks - according to the log the EasyAudioEncoder process was started at 11:16:04

Is there a way to establish from system logs if this did start - as you would have gathered I am not a Linux person. On windows for example I would be running all this with process monitor running in the background and I can then see what processes and threads start and exit. I do not know if you have equivalent on the NAS

Apr 14, 2017 11:16:04.102 [0x7f8af1fff700] DEBUG - Job running: '/config/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-133-linux-ubuntu-x86_64/EasyAudioEncoder/EasyAudioEncoder'

Was this on PMS 1.5.5 ?

Also all the docker scripts and documentation refer to user plex - yet yours is showing user abc - just wondering if this is a factor

@sa2000 said:

@kopfpilot said:
Here we go :slight_smile:

I thought my output would look identical to what jody2 postet, but the EAE temp files do have different file endings.

On some of the ls commands i piped the output directly into a file, so the command itself is mssing.
In the PMS-Logfile obtainging codecs can be found between lines 101 and 132.

The files ending with post_delete were made after delete of folders and restart of the container.
The files ending with post_transcode where made during playback.

Is it possible that the created m3u files contain entries with wrong file extensions? Where can i find these files?

There is no change in the process list during playback. A transcoder process is summoned, but the EAE process is still not listed.

In docker logs, the string " Dolby, Dolby Digital, Dolby Digital Plus, Dolby TrueHD and the double D symbol are trademarks of Dolby Laboratories." is logged for each transcoding playback once.

Thanks - according to the log the EasyAudioEncoder process was started at 11:16:04

Is there a way to establish from system logs if this did start - as you would have gathered I am not a Linux person. On windows for example I would be running all this with process monitor running in the background and I can then see what processes and threads start and exit. I do not know if you have equivalent on the NAS

Apr 14, 2017 11:16:04.102 [0x7f8af1fff700] DEBUG - Job running: '/config/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-133-linux-ubuntu-x86_64/EasyAudioEncoder/EasyAudioEncoder'

Was this on PMS 1.5.5 ?

yes, it was.

Also all the docker scripts and documentation refer to user plex - yet yours is showing user abc - just wondering if this is a factor
GitHub - plexinc/pms-docker: Plex Media Server Docker repo, for all your PMS docker needs.

I am using linuxserver/plex instead of plexinc/pms-docker. I would be surprised if that would have any effects, since both base on the same deb-binary. The difference is what happens during bootstrapping the application.

In germany we have a saying “
 i have seen horses vomit in front of a pharmacy” (which they actualy isn’t possible), which roughly translates to the meaning “everything is possible” :wink:

I am building the image now, but i had to remove the HEALTHCHECK line, because the used docker version on Synology is pretty outdated: Docker version 1.11.2, build 0ee7c49-synology. Let me see if there are any differences.

@kopfpilot said:

@sa2000 said:
Also all the docker scripts and documentation refer to user plex - yet yours is showing user abc - just wondering if this is a factor
GitHub - plexinc/pms-docker: Plex Media Server Docker repo, for all your PMS docker needs.

I am using linuxserver/plex instead of plexinc/pms-docker. I would be surprised if that would have any effects, since both base on the same deb-binary. The difference is what happens during bootstrapping the application.

You are in the Plex official docker forum. The assumption was that you were using the official Plex docker.

I would really like to see that this issue arises on the official Plex docker before taking it further. It may turn out to be a synology issue.

So please could you try out the official plex docker.

As a side, what is the reason for not using Plex Media Server on the synology directly rather than through docker ?

I am about to try the official docker version. But like i wrote: i doubt that it makes any difference. It is more likely that it’s a docker on synology problem, more specificly on how they handle the volume mapping. But then again, i am kind of lost to understand how all of this is related to expecting a different file ending than the actualy created files have.

I am using docker for everything rather then synology packages, i feel docker containers are clean compared to the mess synology packages create. And whenever i feel the need to move the container to another host, the folders i have to move are clear to me without any research.

@kopfpilot said:
I am about to try the official docker version. But like i wrote: i doubt that it makes any difference. It is more likely that it’s a docker on synology problem, more specificly on how they handle the volume mapping. But then again, i am kind of lost to understand how all of this is related to expecting a different file ending than the actualy created files have.

It is very likely that it is to do with synology. You may find it will be the same - but still I have to go through that step.

I am surprised the EasyAudioEncoder process does not get run when we do not appear to get an error on starting it

Early feedback:

Transcoding works with plexinc/pms-docker, if i start with an empty config folder!
My first idea was: shame on me it was the linuxserver/plex image all along
 but wait, what happes if i reuse my existing config folder? Well, transcoding does not work then. So the reason needs to be in the config folder and not in the used image.


 and i am glad that it is not a general Synology docker problem.

I am investigating the differences now.

Update:
I could isolte the problem area: the problem is caused by something in the Preferences.xml file!
Since there are lot of sensitive informations in the file i’d like to not post it here


It is working for me now :slight_smile:

Update2:
Believe it or not, but when i activate “Library/Update my library automatically” and restart the docker container, transcoding fails after the next start! After deactivation and restart of the container, it is working again! How do i make sense of this observation?!

The setting is found in Prefences.xml as attribute “FSEventLibraryUpdatesEnabled”

If we would know what feature it requires to effecticly work, we could figure out if adding a capability would fix the problem. Capabilities can be addeed via docker run --cap-add. See more about capabilites: http://man7.org/linux/man-pages/man7/capabilities.7.html

Starting the container in privileged mode does not change the behavior.

Btw, there is an additional “Plex EAE Service” in the process list now if FSEventLibraryUpdatesEnabled is disabled , even when no playback is in progress.

The EAE process relies on notifications which is why the area got moved to /tmp - a local directory

So which way round is it for that setting in preferences

FSEventLibraryUpdatesEnabled=“0” (“Update my library automatically” disabled in the ui) permits the EAE process.
FSEventLibraryUpdatesEnabled=“1” (“Update my library automatically” enabled in the ui) prevents the EAE process.

@kopfpilot - it could be that you have large libraries and reaching iNotify limit which is used by EasyAudioEncoder

See if increasing the limit makes a difference

Full logs after start up with the setting enabled would log every iNotify watch so we can count them

can you see if your logs have this
[Notify] Failed to add watch for

Always best to provide full logs from start of PMS so one gets the full picture

I’m quite lost at the logic for splitting the audio/video transcoding out here with this “easyaudioencode” which contrary to its name seems to have complicated the setup for many third party apps! I’m guessing it’s some parser which can help but not necessary as I see things running even when it doesn’t seem to get engaged properly.

Also, this application developer compulsion for using /tmp for anything at all they deem as temporary needs to stop or at least not be hardcoded so it can be changed. Gone are the days that /tmp was used for everything except very large files
now it’s hit n miss whether it’s memory or the root file system depending on distro n sometimes version. Root file system is not meant for application files and with PMS the total amount of space being used can be huge so neither memory or / are very clever places depending on the configuration. It comes down to the PMS devs not fully understanding Linux tbh, their use of startup/service mechanisms, variables, etc has been a constant battle fixing the mistakes & issues through the versions
their systemd setup is still bugged afaik and has been since they created it so not all the variables are used properly and it errors every time.

Sorry for the rant but it is appropriate here as they need to learn to work with the Linux OS rather than around it while making their configuration of the application more logical & flexible. In doing things correctly like that they wouldn’t hit problems like these, would mean the third party devs are able to create their enhancements without tearing their hair out and having to create constant work arounds. It could also allow for all the temporary areas to be easily configured by the users rather than creating performance or capacity issues all over the place (currently I’m aware of 4 different temporary locations for PMS jobs of which only 1 is configurable).

The third party devs help make PMS (an already great application) even better for so many and the users are Plex’s blood, life & soul
infuriating/alienating either or both should a massive red flag to the company but yet it continues when a lot is avoidable. Hopefully they can let us help them succeed as none of us are using this product and investing time & energy into logging faults or building addons because we want to fail now are we? Lol

Sorry, perhaps this was way more of a rant than it should have been pushing this from relevant to me looking like I want to flame for flaming sake but this issue has been just the latest painful one docker & nas plus The Plex Remote Transcoding project so I snapped a little. Hopefully it can be taken constructively as meant and someone internally can make good happen.

Anyway, sorry but not sorry (if you follow)

Liviy

@sa2000: well, the library is big enough to not consider it as small, I guess :slight_smile:

8192 matches [Notify] Now watching ... 3400 matches [Notify] Failed to add watch for ...

Seems like 8292 is the hard limit. Though a clean 3400 failed does look somehow suspicous as well.
Hontestly, as those entries have the log level DEBUNG and start with “Notifify”, i didn’t realy bothered making sense of those lines.

Seems like https://forums.plex.tv/discussion/comment/1409740/#Comment_1409740 is the fix.

update:
Following the suggested fix, did the trick :smiley:
Add fs.inotify.max_user_watches=100000 to /etc/sysctl.conf.
Reload config changes with sysctl --system

@sa2000: Thanks for pointing in the right direction!

@kopfpilot said:
@sa2000: well, the library is big enough to not consider it as small, I guess :slight_smile:

8192 matches [Notify] Now watching ... 3400 matches [Notify] Failed to add watch for ...

Seems like 8292 is the hard limit. Though a clean 3400 failed does look somehow suspicous as well.
Hontestly, as those entries have the log level DEBUG and start with “Notify”, i didn’t realy bothered making sense of those lines.

Seems like https://forums.plex.tv/discussion/comment/1409740/#Comment_1409740 is the fix.

update:
Following the suggested fix, did the trick :smiley:
Add fs.inotify.max_user_watches=100000 to /etc/sysctl.conf.
Reload config changes with sysctl --system

@sa2000: Thanks for poiting in the right direction!

Excellent - just need to be on the lookout for iNotify failures - unfortunately easyaudioencoder does not log to the Plex media server log and so such issues need to be spotted from the server’s own library iNotify add failure

Oh, i just realize the failed ones were marked as errors, but the message (28: No space left on device) made no sense after double checking it with df -h. Please make the developers replace the ambigous error message with an accurate one.

Tbh, I am pretty sure i won’t take a look at my logs for a loooong time after today
 until the next incident takes place


I am a developer myself and i know how important accurate log entries are. Though, sometimes breakage is data dependend and won’t become visible in any of our test stages, but hapens during smoktests of a productive deployment. I can asure you, writing last minute hotfixes, packaging and redeploying on short notice is another level of stress.

Nevertheless, understood your point. Thanks for the help :slight_smile: