@kopfpilot said: @sa2000: well, the library is big enough to not consider it as small, I guess
8192 matches [Notify] Now watching ... 3400 matches [Notify] Failed to add watch for ...
Seems like 8292 is the hard limit. Though a clean 3400 failed does look somehow suspicous as well.
Hontestly, as those entries have the log level DEBUNG and start with âNotififyâ, i didnât realy bothered making sense of those lines.
update:
Following the suggested fix, did the trick
Add fs.inotify.max_user_watches=100000 to /etc/sysctl.conf.
Reload config changes with sysctl --system
@sa2000: Thanks for pointing in the right direction!
I been experiencing same exact issue on CentOS 7.x with PMS 1.5.5. Your âupdateâ fixed the issue for me. Thank you for the help.
@kopfpilot said:
Oh, i just realize the failed ones were marked as errors, but the message (28: No space left on device) made no sense after double checking it with df -h. Please make the developers replace the ambigous error message with an accurate one.
If you let me have the full logs, I can put your feedback in context and raise it with the developers. Where was the error 28 logged
I attached redacted logs. They include everything from container start to the EAE playback issue.
The word âwatchingâ is overloaded and one asumes something different (as in related to playback) in a media server context.
How about using: [Notify] Added file watcher for xyz or [Notify] Now observing folder xyz istead of [Notify] Now watching xyz. [Notify] Failed to add file watcher for xyz or [Notify] Failed to observ folder xyz instead of [Notify] Failed to add watch for xyz
Just out of curriosity, I will downgrade to 1.5.3 later and see if adding the parameter --volume-driver=âlocalâ will fix EAE issues with docker volumes used as transcoding target.
Actually Now monitoring file change events on xyz and Failed monitoring file change events on xyz seems to be a more proper fit.
During startup PMS could query the inotify setting and use it during bootstraping to identify if the Failed monitoring file cahnge events on xyz error is caused by exceeding the inotify allowance.
@kopfpilot said:
I attached redacted logs. They include everything from container start to the EAE playback issue.
The word âwatchingâ is overloaded and one asumes something different (as in related to playback) in a media server context.
How about using: [Notify] Added file watcher for xyz or [Notify] Now observing folder xyz istead of [Notify] Now watching xyz. [Notify] Failed to add file watcher for xyz or [Notify] Failed to observ folder xyz instead of [Notify] Failed to add watch for xyz
Just out of curriosity, I will downgrade to 1.5.3 later and see if adding the parameter --volume-driver=âlocalâ will fix EAE issues with docker volumes used as transcoding target.
@kopfpilot said:
I attached redacted logs. They include everything from container start to the EAE playback issue.
The word âwatchingâ is overloaded and one asumes something different (as in related to playback) in a media server context.
How about using: [Notify] Added file watcher for xyz or [Notify] Now observing folder xyz istead of [Notify] Now watching xyz. [Notify] Failed to add file watcher for xyz or [Notify] Failed to observ folder xyz instead of [Notify] Failed to add watch for xyz
Just out of curriosity, I will downgrade to 1.5.3 later and see if adding the parameter --volume-driver=âlocalâ will fix EAE issues with docker volumes used as transcoding target.
I understand. Though, a less ambigous error message would have led to me including those parts of the logs earlier.
There would be no benefit in trying out 1.5.3. The Notify issue would affect 1.5.3 as well
My objective is the necessity of the hardcoded /tmp cache folder for EAE. In order to test it, i need to downgrade to a version without the hardcoded /tmp cache folder and that would be 1.5.3, wouldnât it?
I am confidendent that Docker options like --volume-driver=âlocalâ or adding a mode to the /transcode volume mapping would solve the problem that led to the hard coded /tmp cache folder workaround in first place.
@kopfpilot said:
I attached redacted logs. They include everything from container start to the EAE playback issue.
The word âwatchingâ is overloaded and one asumes something different (as in related to playback) in a media server context.
How about using: [Notify] Added file watcher for xyz or [Notify] Now observing folder xyz istead of [Notify] Now watching xyz. [Notify] Failed to add file watcher for xyz or [Notify] Failed to observ folder xyz instead of [Notify] Failed to add watch for xyz
Just out of curriosity, I will downgrade to 1.5.3 later and see if adding the parameter --volume-driver=âlocalâ will fix EAE issues with docker volumes used as transcoding target.
I understand. Though, a less ambigous error message would have led to me including those parts of the logs earlier.
There would be no benefit in trying out 1.5.3. The Notify issue would affect 1.5.3 as well
My objective is the necessity of the hardcoded /tmp cache folder for EAE. In order to test it, i need to downgrade to a version without the hardcoded /tmp cache folder and that would be 1.5.3, wouldnât it?
I am confidendent that Docker options like --volume-driver=âlocalâ or adding a mode to the /transcode volume mapping would solve the problem that led to the hard coded /tmp cache folder workaround in first place.
I would never recommend freezing your version of software as you would definitely miss out on new features and bug fixes
Apr 19, 2017 12:27:14.001 [0x7fcab3bff700] ERROR - [Transcoder] [ac3_eae @ 0x24165e0] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-37309c0b-e676-43ee-8cd8-8e1a96e388ce/EasyAudioEncoder/Convert to WAV (to 8ch or less)/ufrakbaneevgfljkepzsmslu_29803-1-10.wav'
This is on a normal x86 box with Docker 17.04.0-ce. Started after 1.5.5. Seems to be linked to library updates, since before that everything works but after running a scan these pop up.
@niko@salaliitto.com said:
FWIW iâm also seeing the same issue, eg.
Apr 19, 2017 12:27:14.001 [0x7fcab3bff700] ERROR - [Transcoder] [ac3_eae @ 0x24165e0] EAE timeout! EAE not running, or wrong folder? Could not read '/tmp/pms-37309c0b-e676-43ee-8cd8-8e1a96e388ce/EasyAudioEncoder/Convert to WAV (to 8ch or less)/ufrakbaneevgfljkepzsmslu_29803-1-10.wav'
This is on a normal x86 box with Docker 17.04.0-ce. Started after 1.5.5. Seems to be linked to library updates, since before that everything works but after running a scan these pop up.
And no erros about the disk space etc, just the EAE timeout and âcould not readâ
The audio transcoder process relies on inotify events and the symptoms you see would arise if the maximum of inotify watches is reached -and the transcoder fails to add an inotify watch. This could be what is happening if you have a very large set of plex media server libraries
You would need to increase the inotify maximum for the user account plex media server runs in
Sorry, no just mean a generic box (and not ARM based NAS or something like that). Ubuntu Trusty on 2x L5640 with Docker
@sa2000 said:
The audio transcoder process relies on inotify events and the symptoms you see would arise if the maximum of inotify watches is reached -and the transcoder fails to add an inotify watch. This could be what is happening if you have a very large set of plex media server libraries
You would need to increase the inotify maximum for the user account plex media server runs in
Hmm, even though I didnt see the same errors as other people 10xâing the inotify limit seems to have fixed it for me. I have 2000+ movies and 600+ series (20k+ episodes) so maybe thats it (although even the original amount is larger than the amount of files⊠and especially more than directories)
Sorry, no just mean a generic box (and not ARM based NAS or something like that). Ubuntu Trusty on 2x L5640 with Docker
@sa2000 said:
The audio transcoder process relies on inotify events and the symptoms you see would arise if the maximum of inotify watches is reached -and the transcoder fails to add an inotify watch. This could be what is happening if you have a very large set of plex media server libraries
You would need to increase the inotify maximum for the user account plex media server runs in
Hmm, even though I didnt see the same errors as other people 10xâing the inotify limit seems to have fixed it for me. I have 2000+ movies and 600+ series (20k+ episodes) so maybe thats it (although even the original amount is larger than the amount of files⊠and especially more than directories)
I understood this to be number of directories to watch for the user account - all processes. We could see how many iNotify watches the main Plex Media Server process is adding if you restart the server and capture all the server logs after 10 minutes.
@csutcliff said:
Just a quick +1 to the above, exactly the same issue here using the official docker image.
Restart the plex media server and wait 10 minutes and then get the plex media server.log and see if there are any iNotify adds failing
Nothing in the logs about iNotify failing. Oddly enough this only became a problem for me with 1.5.4 when the audio transcoder was switched to /tmp.
Iâve experimented today with passing in â-v /mnt/iscsi/transcode/tmp:/tmpâ to force my /tmp to go back to my iscsi mounted scratch area for transcoding. Although itâs still too early to say conclusively, it appears to have been fine all day.
Before I made that change it was breaking every time something was added to the library and a scan initiated, requiring a container restart to play again.
Another thing thatâs worth mentioning is Iâm using overlay2 as my storage driver for docker rather than aufs.
Would it be possible to make this a setting rather than hard-coded? I canât imagine Iâm the only person who doesnât want/need it to be in /tmp?
Btw. i recently installed Corral to understand how Docker is working there. In Corral, Docker Hosts are installed inside dedicated and jailed boot2docker VMs. Thatâs why pointing the temp folder to âVMâ did result in having the temp folder inside the boot2docker vm and not directly on a Corral volume. I highly doubt that folders on a volume are mounted inside the boot2docker vm as network shares. The original problem must be related on how the used bHyve hypervisor mapps folders from a volume inside the boot2docker vm.
@csutcliff
How did you provide the settings for the volume-driver?
@kopfpilot said:
I highly doubt that folders on a volume are mounted inside the boot2docker vm as network shares. The original problem must be related on how the used bHyve hypervisor mapps folders from a volume inside the boot2docker vm.
The guest OS (boot2docker) sees the volumes mounted from the host as 9PFS which is a network share as far as it is concerned. Whether that data actually goes through the network or is handled internally by the hypervisor is immaterial. What matters here is how the guest OS sees it.
This is an issue running on 1.6.1.3722 as well (I just updated). The following solved it for me:
sudo nano /etc/sysctl.conf (requires sudo yum -y install nano or use vi, or even echo the below)
add this to the bottom of the file (append) fs.inotify.max_user_watches=25000 (any number, really)
then reload the config, as said above sudo sysctl --system
No need to restart plex, just make sure you restart the streams trying to use EAE.
As a note to the devs:
Can you please, if you read this, allow us a simple way to configure all these paths ourselves?
If you add them as options in Plex Web itâs a lot easier. But a plex.conf file with all configurations would be awesome.
Even how and which transcoders we would like to use, and maybe some integral cli commands would be cool to for stuff like db opt, bundles and more. I would love to be able to schedule these things, and leave plex out of it.
I just try to understand if docker volumes can be paramterized to actualy behave like the general authfs overlay for the container.