Turn VERBOSE OFF please… they are of no help; only hindrance. DEBUG is more than enough
Chuck, so, I upgraded the plex server to 1.30. I then replicated the issue. While the playback was hung, I downloaded a set of logs. I then hit stop play and downloaded a second set of logs. Am attaching both.
Plex Media Server Logs_2022-12-09_12-46-26.zip (3.7 MB)
Plex Media Server Logs_2022-12-09_12-46-14.zip (3.7 MB)
Let me know if you would like more or something different.
If this is all you need, please let me know, so I can roll back to functional 1.29. Thanks
I see the error loud and clear . WOW!
How much space is on /tmp ?? (df -h)
This smells like /tmp is full (again)
Chuck, don’t think so.
xxxxxxxxxxx:/home/nas# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p2 938G 283G 607G 32% /
Dec 09, 2022 12:46:04.128 [0x7fb9494f4b38] ERROR - [Req#c8df/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.129 [0x7fb94bfc1b38] ERROR - [Req#c8e0/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.129 [0x7fb9496f7b38] ERROR - [Req#c8e1/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.130 [0x7fb9492f1b38] ERROR - [Req#c8e2/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.131 [0x7fb9444c6b38] ERROR - [Req#c8e3/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.131 [0x7fb947687b38] ERROR - [Req#c8e4/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.132 [0x7fb94788ab38] ERROR - [Req#c8e5/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.133 [0x7fb943a26b38] ERROR - [Req#c8e6/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.133 [0x7fb947acdb38] ERROR - [Req#c8e7/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.134 [0x7fb9494f4b38] ERROR - [Req#c8e8/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.135 [0x7fb94bfc1b38] ERROR - [Req#c8e9/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:46:04.135 [0x7fb9496f7b38] ERROR - [Req#c8ea/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Something is full. 
lol
How many media items do you have? More than 65536 directories being monitored ?
I wonder if inotify table full is being reported weird because this is dolby audio ??
supplemental – TrueHD
Dec 09, 2022 12:45:47.256 [0x7fb947acdb38] ERROR - [Req#68a3/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] [truehd_eae @ 0x7fbe516a8080] does not start with major sync!
Dec 09, 2022 12:45:47.257 [0x7fb94788ab38] ERROR - [Req#68a4/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: Invalid data found when processing input
Dec 09, 2022 12:45:47.258 [0x7fb9494f4b38] ERROR - [Req#68a5/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] [truehd_eae @ 0x7fbe516a8080] does not start with major sync!
Dec 09, 2022 12:45:47.259 [0x7fb94bfc1b38] ERROR - [Req#68a6/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: Invalid data found when processing input
Dec 09, 2022 12:45:47.260 [0x7fb9496f7b38] ERROR - [Req#68a7/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] [truehd_eae @ 0x7fbe516a8080] does not start with major sync!
Dec 09, 2022 12:45:47.261 [0x7fb947687b38] ERROR - [Req#68a8/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: Invalid data found when processing input
Dec 09, 2022 12:45:47.261 [0x7fb947acdb38] ERROR - [Req#68a9/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] [truehd_eae @ 0x7fbe516a8080] does not start with major sync!
Dec 09, 2022 12:45:47.262 [0x7fb94788ab38] ERROR - [Req#68aa/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: Invalid data found when processing input
Dec 09, 2022 12:45:47.263 [0x7fb9494f4b38] ERROR - [Req#68ab/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] [truehd_eae @ 0x7fbe516a8080] does not start with major sync!
Dec 09, 2022 12:45:47.264 [0x7fb94bfc1b38] ERROR - [Req#68ac/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: Invalid data found when processing input
Dec 09, 2022 12:45:47.265 [0x7fb9496f7b38] ERROR - [Req#68ad/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] [truehd_eae @ 0x7fbe516a8080] does not start with major sync!
Dec 09, 2022 12:45:47.266 [0x7fb947687b38] ERROR - [Req#68ae/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: Invalid data found when processing input
Dec 09, 2022 12:45:47.267 [0x7fb947acdb38] ERROR - [Req#68af/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
Dec 09, 2022 12:45:47.268 [0x7fb94788ab38] ERROR - [Req#68b0/Transcode/r0i5gw8ehynryd7c4vfey0vx/32a33ff0-3fff-4ae6-9e47-5e7794d0e9c0] Error while decoding stream #0:1: No space left on device
What would you like me to pull next? Here’s a full df listing.
Filesystem Size Used Avail Use% Mounted on
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 32G 0 32G 0% /dev
devpts 0 0 0 - /dev/pts
tmpfs 6.3G 3.9M 6.3G 1% /run
/dev/nvme0n1p2 938G 283G 607G 32% /
securityfs 0 0 0 - /sys/kernel/security
tmpfs 32G 4.0K 32G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
cgroup2 0 0 0 - /sys/fs/cgroup
pstore 0 0 0 - /sys/fs/pstore
efivarfs 0 0 0 - /sys/firmware/efi/efivars
bpf 0 0 0 - /sys/fs/bpf
systemd-1 - - - - /proc/sys/fs/binfmt_misc
hugetlbfs 0 0 0 - /dev/hugepages
mqueue 0 0 0 - /dev/mqueue
debugfs 0 0 0 - /sys/kernel/debug
tracefs 0 0 0 - /sys/kernel/tracing
fusectl 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /run/credentials/systemd-sysusers.service
configfs 0 0 0 - /sys/kernel/config
nfsd 0 0 0 - /proc/fs/nfsd
/dev/loop0 163M 163M 0 100% /snap/gnome-3-28-1804/145
/dev/nvme0n1p1 511M 5.3M 506M 2% /boot/efi
movies 93T 87T 6.5T 94% /video/movies
tv 138T 126T 12T 92% /video/tv
/dev/loop1 128K 128K 0 100% /snap/bare/5
/dev/loop2 117M 117M 0 100% /snap/core/14399
/dev/loop3 56M 56M 0 100% /snap/core18/2620
/dev/loop4 56M 56M 0 100% /snap/core18/2632
/dev/loop6 64M 64M 0 100% /snap/core20/1695
/dev/loop7 165M 165M 0 100% /snap/gnome-3-28-1804/161
/dev/loop9 219M 219M 0 100% /snap/gnome-3-34-1804/77
/dev/loop8 219M 219M 0 100% /snap/gnome-3-34-1804/72
/dev/loop11 347M 347M 0 100% /snap/gnome-3-38-2004/119
/dev/loop10 347M 347M 0 100% /snap/gnome-3-38-2004/115
/dev/loop12 82M 82M 0 100% /snap/gtk-common-themes/1534
/dev/loop13 92M 92M 0 100% /snap/gtk-common-themes/1535
/dev/loop14 46M 46M 0 100% /snap/snap-store/599
/dev/loop15 46M 46M 0 100% /snap/snap-store/638
/dev/loop16 384K 384K 0 100% /snap/snapd-desktop-integration/14
/dev/loop17 384K 384K 0 100% /snap/snapd-desktop-integration/43
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
sunrpc 0 0 0 - /run/rpc_pipefs
hpsrv:/data 9.1T 5.5T 3.2T 64% /data
tmpfs 6.3G 76K 6.3G 1% /run/user/124
portal 0.0K 0.0K 0.0K - /run/user/124/doc
gvfsd-fuse 0.0K 0.0K 0.0K - /run/user/124/gvfs
tmpfs 6.3G 3.9M 6.3G 1% /run/snapd/ns
tmpfs 6.3G 68K 6.3G 1% /run/user/1000
gvfsd-fuse 0.0K 0.0K 0.0K - /run/user/1000/gvfs
portal 0.0K 0.0K 0.0K - /run/user/1000/doc
/dev/loop18 64M 64M 0 100% /snap/core20/1738
nsfs 0 0 0 - /run/snapd/ns/snapd-desktop-integration.mnt
root@nasserver:/home/nas#
I have 4 libraries - /video/tv /video/movies /data/music and /data/audiobooks
/video/tv - 1957 dir
/video/movies - 7397
/data/music - 1336
/data/audibooks - 4714
so, nope not even close to 65k
I have transcodes using /dev/shm, but have also replicated with that pointing to the /var/lib directory. Also, why would it happen with 1.30 but not 1.29 if we were running out of space on my server side?
what about supplemental truehd?
I’m starting to suspect; Update Dolby core (EasyAccessEncoder) and introduce regression.
TrueHD requires inotify slots.
I’ve reported other TrueHD errors.
Do you have any other videos which aren’t trueHD which transcode OK on 1.30 ?
One other test (if you’re able)
- With 1.30.0 installed and showing No Space error
- Restart host
- Play same video again
See if “no space” error is present or not.
I have numerous titles without TrueHD. I believe they will play fine, but without watching one all the way through, I cannot say with 100% certainty. When the problem was discovered, I was out of town. I know we watched other shows with no problems. When I got back and you said to rollback, I have been on 1.29 ever since.
OK… stay on 1.29.2 then.
I’ll lump this in with the other 1.29.2 TrueHD issue I currently have open.
Thanks very much for your help !!
So, do you still want me to reboot the box and try it again? If so, could you elaborate? I saw the no space issue in the logs, but my tests are to see if the movie hangs. I most certainly could tail a file and monitor for the space lines to appear. Is that what you are looking for?
If you have the time, and only if you have the time, it would help provide a predictable datapoint.
The reason for this is: Your machine reliably exhibits the problem.
If we can show the fault is removed by restarting the host then it’s clear “no space left on device” is because inotify slot locks are not being freed after each dolby transcode session.
Chuck, no joy. I replicated the issue, rebooted, and before anyone else jumped on, I attempted to play the video. Still hung in EXACTLY the same place. Since you continued to mention inotify locks, I also grabbed the inotify settings from the /proc filesystem. The numbers were initially lower, but I increased the values, but again to no joy. I have included them below for your appraisal. If you feel I should bump them again, please let me know.
root@nasserver:/proc/sys/fs/inotify# for i in *
do
hold=cat $i
echo “$i - $hold”
done
max_queued_events - 16384
max_user_instances - 8192
max_user_watches - 262144
I also checked inode usage and don’t see anything above 5% usage.
I was also wondering how the video which hangs under 1.30 can run perfectly fine under 1.29 by literally just downgrading the installation. No reboot or anything. If I’m actually running out of room, I would expect 1.29 to exhibit the issue as well.
1 Like
I was also having this issue on 1.30. Before I found this thread, I tracked the issue down to transcoding Dolby Atmos (truehd 7.1) audio. I thought I messed something up so I even restored from backup. I also tested increasing my inotify watches and instances. Rebooted many times. I deleted logs, then played Paranorman in 4k. Video was playing direct but audio was being transcoded. Client was TCL Roku TV. Server is Ubuntu 16.04 VM with 53GB of space available. Media is on a server running TrueNAS and connected directly via 10Gb NIC.
Hope this helps. I downgraded to 1.29 so we could watch videos.
Plex Media Server Logs_2022-12-13_22-35-17.zip (506.1 KB)
Stupac62, appreciate your confirmation on this. Our setups are similar. I am on Ubuntu 22.04 and utilize a local zfs filesystem. It was confirmed that the culprit was transcode of truehd, or at least that is what I understood from Chuckpa. He sent something on to the devs, but there has now been a number of releases since this was reported and it is still happening with plexmediaserver_1.30.1.6497-5fc2e0894_amd64.deb. I just hope that they have everything to replicate and resolve this issue.
It’s interesting your copy of Paranorman has truehd, I checked mine immediately upon reading your post and mine has DTS-HD.
ChuckPa, do you have any updates on this, or is there anything else we can do to help?
Shout Factory just released a 4k remaster of Paranorman and Coraline. I had it first fail on my copy of The Grinch which has truehd 7.1 as well.
As the industry moves forward, this will become a more common issue as releases are made with better audio. Again, I hope the plex developers address this soon.
