Unifi - 40 GB/day excessive bandwidth

Server Version#: 1.18.0.1913
Player Version#: N/A

For at least 2 months my ISP has complained that I was using too much bandwidth.
I have traced this back to my PMS setup (shutting it down stopped the bandwidth usage).
I have limited to none streaming to WAN so this makes no sense.
Initially I thought the music library scanner kept refreshing metadata, but it continued after I deleted that library.

How do I interpret the logs (attached) to resolve this?

Those 4 log files don’t provide enough information.
Your logs provide activity coverage for about 2 hours of elapsed time, during which PMS was completely idle.

The entire Logs directory is needed.

I would also check the Scheduled Tasks options to see if metadata is being refreshed nightly.

If you have media stored in a cloud (common now), PMS will read those file when refreshing what it knows of your media at night during maintenance.

All logs here: [moderator edit: link to logs erased]

PMS and the storage is colocated on the same machine

btw the bandwidth usage seems to be a continuous / periodic burst - eg every 5 minutes i’d see a spike of 10sec sustained of 1 MB/s (so not only at night / during the scheduled period)

Thank you. Now I can see much better what’s happening.

PMS is still trying to scan and match media then complete obtaining metadata for it. Those bursts you see are PMS slowly trying to make inquiries to complete your library.

  1. You have permission problems and PMS is not able to read some of your media (Permission denied) . User plex can’t read those files.
Oct 13, 2019 13:23:25.255 [0x7fd366a73740] DEBUG - [MI] Opening input file: "/volume1/Music/MP3 New Archive/Volume 7 (aug10-jul11)/Clubland 19 (2011)/CD1/(04) [Jennifer Lopez] On The Floor.mp3"
Oct 13, 2019 13:23:25.255 [0x7fd366a73740] DEBUG - [FFMPEG] - Opening '/volume1/Music/MP3 New Archive/Volume 7 (aug10-jul11)/Clubland 19 (2011)/CD1/(04) [Jennifer Lopez] On The Floor.mp3' for reading
Oct 13, 2019 13:23:25.255 [0x7fd366a73740] DEBUG - [FFMPEG] - Setting default whitelist 'file,crypto'
Oct 13, 2019 13:23:25.255 [0x7fd366a73740] ERROR - Match: Caught exception opening file: Could not parse /volume1/Music/MP3 New Archive/Volume 7 (aug10-jul11)/Clubland 19 (2011)/CD1/(04) [Jennifer Lopez] On The Floor.mp3 (error=-13): Permission denied
Oct 13, 2019 13:23:25.255 [0x7fd366a73740] DEBUG - [MI] Opening input file: "/volume1/Music/MP3 New Archive/Volume 7 (aug10-jul11)/Clubland 19 (2011)/CD1/(05) [Jessie J] Price Tag (Lockout's Club Remix).mp3"
Oct 13, 2019 13:23:25.255 [0x7fd366a73740] DEBUG - [FFMPEG] - Opening '/volume1/Music/MP3 New Archive/Volume 7 (aug10-jul11)/Clubland 19 (2011)/CD1/(05) [Jessie J] Price Tag (Lockout's Club Remix).mp3' for reading
  1. It is continuing to retry obtaining metadata for media which it can’t get a proper match for.
Oct 16, 2019 10:50:12.659 [0x7fa61e09e740] DEBUG - Scanner: Processing directory /volume1/Video/Snoes (2 - Series)/Pingu (parent: yes)
Oct 16, 2019 10:50:12.706 [0x7fa61e09e740] DEBUG - Directory had 157 files, database had 156 files, can't skip.
Oct 16, 2019 10:50:12.973 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e01.pingu.is.introduced.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.974 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413933
Oct 16, 2019 10:50:12.975 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e02.pingu.helps.to.deliver.the.mail.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.976 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413934
Oct 16, 2019 10:50:12.976 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e03.pingu.helps.with.incubating.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.977 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413935
Oct 16, 2019 10:50:12.978 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e04.pingu.is.born.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.979 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413936
Oct 16, 2019 10:50:12.979 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e05.pingu.goes.fishing.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.980 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413937
Oct 16, 2019 10:50:12.980 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e06.pingu.is.jealous.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.981 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413938
Oct 16, 2019 10:50:12.982 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e07.pingu.and.the.snowball.fight.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.983 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413939
Oct 16, 2019 10:50:12.983 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e08.pingu.and.the.avalanche.incident.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.984 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413940
Oct 16, 2019 10:50:12.985 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e09.pingu.plays.fish.tennis.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.985 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413941
Oct 16, 2019 10:50:12.986 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e10.pingu.on.makeshift.skis.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.987 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413942
Oct 16, 2019 10:50:12.987 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e11.pingus.tobogganing.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.988 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413943
Oct 16, 2019 10:50:12.989 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e12.pingu.is.left.out.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.990 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413944
Oct 16, 2019 10:50:12.990 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e13.pingu.plays.ice.hockey.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.991 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413945
Oct 16, 2019 10:50:12.991 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e14.pingu.runs.away.from.home.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.992 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413946
Oct 16, 2019 10:50:12.993 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e15.pingu.builds.an.igloo.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.994 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413947
Oct 16, 2019 10:50:12.994 [0x7fa61e09e740] DEBUG - Looking for path match for [/volume1/Video/Snoes (2 - Series)/Pingu/aaf-pingu.s01e16.pingu.and.pinga.dont.want.to.go.to.bed.dvdrip.xvid.avi]
Oct 16, 2019 10:50:12.995 [0x7fa61e09e740] DEBUG - Path matched, we're reusing media item 413948
  1. Here, it shows it knows this media is ok and is skipping over it.
Oct 16, 2019 10:50:13.747 [0x7fa61e09e740] DEBUG - Skipping over directory 'Brandweerman Sam/Season 5', as nothing has changed; removing 5 media items from map.
Oct 16, 2019 10:50:13.747 [0x7fa60d921700] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/updateProgressMessage?message=Scanning%20Brandweerman%20Sam%2FSeason%205
Oct 16, 2019 10:50:13.747 [0x7fa61e09e740] DEBUG - HTTP requesting PUT http://127.0.0.1:32400/activities/98265aa6-bb25-4db6-a0e1-0593a739f426?percentComplete=%2B3%2E57142854&subtitle=Season%205
Oct 16, 2019 10:50:13.749 [0x7fa60d921700] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/updateProgressMessage?message=Scanning%20Brandweerman%20Sam%2FSeason%205
Oct 16, 2019 10:50:13.750 [0x7fa61e09e740] DEBUG - HTTP 200 response from PUT http://127.0.0.1:32400/activities/98265aa6-bb25-4db6-a0e1-0593a739f426?percentComplete=%2B3%2E57142854&subtitle=Season%205
Oct 16, 2019 10:50:13.750 [0x7fa61e09e740] DEBUG - Scanner: Processing directory /volume1/Video/Snoes (2 - Series)/Kleine Rode Tractor (parent: yes)
Oct 16, 2019 10:50:13.751 [0x7fa61e09e740] DEBUG - Skipping over directory 'Kleine Rode Tractor', as nothing has changed; removing 0 media items from map.
Oct 16, 2019 10:50:13.752 [0x7fa60d921700] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/updateProgressMessage?message=Scanning%20Kleine%20

While not trying to tell you how to name your media, some of it is not named per Plex standard. Television series . The same is true of your music curation .

If I may recommend:

Thanks.

The music library has been deleted since Oct14 - but that was not resolving the bandwidth usage.

I will rename the episodes, but will that solve the bandwidth usage by itself? if so, How can I prevent that if i add other badly named files this will again cause runaway bandwidth consumption? (trying to understand how badly named files cause bandwidth, esp since it seems to be querying the localhost)

The utility FileBot is very popular. It takes your media filenames as you have them, performs the internet lookup, then helps automate the renaming of your files into perfect name and structure.

If your files are perfectly named, PMS will never miss. Once matched, it will never look again unless you ask it to keep checking. (Settings - Server - Libarary). Turn off “Periodically scan”

I use the options:

  1. Automatically detect
  2. Run a partial scan.

I have over 13,000 movies+episodes indexed and never have an issue. I also live in a rural area where internet is premium. PMS runs here and my total monthly usage is from what I download / browse purposefully. Daily usuage for me, including all I do for tech support, is about 20 GB.

Is your excessive usage from download or upload? This would be interesting to learn.
I ask because streaming a single movie from Netflix can easily pull 20+ GB.

Sorry FWIW this was not caused by Plex rather by my Unifi controller speedtest.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.