[Bug] Trying to change lib folder structure, losing affected metadata as a result

Plex Media Server Metadata & Adding Files
1

Server Version#: 1.25.6.5577
Player Version#: 3.104.2

TL;DR Bug replicating steps

There is a library that has folder structure as in the following example:

/mnt/disk1
/mnt/disk2
...

I need to exclude a few files from this library, so I moved them in a separate dir ‘B’ on disk root, while the rest of the files remain in dir ‘A’. Now, trying to modify lib paths to the following:

/mnt/disk1/A
/mnt/disk2/A
...

All goes well, rescan (scanner = “Plex Video Files”) works as expected, all the files that are located in dir ‘A’ are still present, ‘available’ and playing no problem.
Even if I manually initiate lib scan, no more changes done anymore to the lib.

But then I hit “Empty trash”, it deletes all the files, including the ones from dir ‘A’ that are not even marked as trash.
After emptying, if I issue a manual rescan it finds all the same files from dir ‘A’, but the metadata is screwed now (date added, play position, manually filled description fields, etc).

Good thing I have a 2d old backup of database and able to restore no problem.
But no matter what I do, I can not perform desired change, and files keep being deleted in the same fashion. I tried (after changing path) to refresh metadata, analyze, optimize database, change scanner to “Plex Video Files Scanner”, adding redundant path ‘/mnt/diskN/A’ before deleting original ‘/mnt/diskN’… Nothing worked.

What am I doing wrong?

Edits:

  1. append [bug] to title
  2. adjust thread tag
  3. add link on top for steps to reproduce
2

Have you given this a read?

3

Thank you for your reply.

Yes, I did all the same steps.
Problem is, in my case after adding new path while keeping the old one, it rescans, but only shows one path in “Media Info” tab for each item. And I can not tell if it is the old one or new, because new is a subfolder of old one, so the path never change, which makes sense.

I don’t know if it is related, but the lib that I am trying to modify is the ‘All media’ library, while there are like 10 more auxiliary libs, that are different subsets of the same media, based on own selection criteria and paths are subfolders of different depth.
Only the lib I am trying to modify has custom metadata, though, so it is the valuable one.

The weird thing is, that I used to toss media around all the time including making minor changes to paths, and do not recall having matching issues. Never changed root paths though that affect so many items.

4

Ok, got some results.
This must be a bug in scanner algorithm (or whatever component(s) mark/delete lib items as trash).

Even the fact that items that are not marked as trash getting deleted is an inconsistency in management logic. And I’ve double checked that fact, because applying a custom filter of “Trash” = “is true” reported a 100 or so items that were actually intended for deletion by lib path changes and were marked correctly, while the count of missing items after emptying trash was more than 3500 (should be all of them located on disks that I’ve changed root locations).

I’m not sure how it is implemented behind the curtains, but there should be a check that does not allow under any circumstances any deletions to items that are not marked as trash.

Additionally, there must be an counter of items to be deleted shown to user in a confirmation dialog and ideally there should really be a configurable threshold setting that requires to enter that counter as a confirmation if trash counter exceeds it. The latter might seem overkill, but is really not considering how often I see folks (myself including) using PMS as more of a collection manager than a glorified media beautifier.

The workaround was to:

  1. alter new root dir name in the actual filesystem (so that PMS had the chance to register location change)
mv /mnt/disk1/A /mnt/disk1/C
mv /mnt/disk2/A /mnt/disk2/C
...
  1. point PMS library to new root folders /mnt/diskN/C
  2. after a rescan, it should be properly empty trash, and delete only files that are located in /mnt/diskN and subfolders other than C.
    Note: one should issue it only for specific lib in question and not the whole server in case other lib still contain the same location, but were not corrected to renamed dir C on previous step
  3. rename dir back to what it was:
mv /mnt/disk1/C /mnt/disk1/A
mv /mnt/disk2/C /mnt/disk2/A
...
  1. point library to a new root with old name /mnt/diskN/A.
  2. after above changes rescan followed by “empty trash” should make zero deletions, ensure it is not otherwise in Settings - Status - Alerts.
5

@ChuckPa Should I report it someplace special?

6

Please show me your structure?

I think you have a mount underneath another mount ??

If so, you must be careful about the mount order.

Show me?

Do you have something like –

  /disk/diskA
    /disk/diskA/diskB

??

7

No no no,
I read about that, no nested mounts involved.

Each disk is an NFS mount from a single disk pool from TrueNAS.
Each in it’s own /mnt/diskN mount point.

# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>

10.10.10.10:/mnt/triplet/aden-db /mnt/aden_db nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/triplet/aden-meta /mnt/aden_meta nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0

10.10.10.10:/mnt/triplet/PRIVATE/ADEN /mnt/aden_triplet nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/node0/PRIVATE/ADEN /mnt/aden_node0 nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/node1/PRIVATE/ADEN /mnt/aden_node1 nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/node2/PRIVATE/ADEN /mnt/aden_node2 nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/node3/PRIVATE/ADEN /mnt/aden_node3 nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0

10.10.10.10:/mnt/triplet/dion-db /mnt/dion_db nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/triplet/dion-meta /mnt/dion_meta nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0

10.10.10.10:/mnt/triplet/PRIVATE/DION /mnt/dion_triplet nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/node1/PRIVATE/DION /mnt/dion_node1 nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0
10.10.10.10:/mnt/node4/PRIVATE/DION /mnt/dion_node4 nfs4 auto,rw,proto=tcp,hard,vers=4.1 0 0

aden, dion are separate PMS servers sharing this VM.

8

I will try to recreate this locally.

When I have it setup , I’ll show you what I have and you can advise.

Sound like a plan?

9

A. Library created (180 series)

Screenshot from 2022-03-02 21-21-25
Screenshot from 2022-03-02 21-21-25730×524 27.2 KB

B. Add new directory to library /vie/tl/new

Screenshot from 2022-03-02 21-24-18
Screenshot from 2022-03-02 21-24-18734×523 30.5 KB

C. Turn off “Scan my library automatically”
D. Empty Trash remains enabled.
E. Directory contents:

[chuck@lizum tl.2005]$ ls *
a:
$#! My Dad Says/                    Ah! My Goddess/         Angel Beats!/
07-Ghost/                           Air/                    Anger Management/
11.22.63/                           Air Gear/               An Idiot Abroad/
11eyes/                             Aishiteruze Baby/       Animaniacs/
12 Monkeys/                         Akame ga Kill!/         anohana- The Flower We Saw That Day/
24/                                 Alcatraz/               Another/
2 Broke Girls/                      Aldnoah.Zero/           Another Life (2019)/
30 Rock/                            ALF/                    Anthony Bourdain- No Reservations/
3rd Rock from the Sun/              Alias/                  Aqua Teen Hunger Force Forever/
666 Park Avenue/                    Alle Mot Alle/          Archer (2009)/
800 Words/                          'Allo 'Allo!/           Arrested Development/
90210/                              Ally McBeal/            Arrow/
90 Day Fiancé: Before the 90 Days/  Almost Human/           Ascension/
Absolutely Fabulous/                Alone/                  Ashes to Ashes/
Accel World/                        Alphas/                 Ash vs Evil Dead/
According to Jim/                   American Dad!/          As Time Goes By/
A Certain Magical Index/            American Horror Story/  Attack on Titan/
A Certain Scientific Railgun/       American Idol/          Avatar- The Last Airbender/
Adventure Time/                     Ancient Aliens (2010)/  Avenue 5/
Afro Samurai/                       Andromeda/              Awake/
Agatha Christie's Poirot/           Angel/                  Awkward/

b:
Babylon 5/                             Beauty and the Beast (2012)/         Black Lagoon/           Bones/
Baccano/                               Beavis and Butt-Head/                Black Mirror/           Bored to Death/
Bakuman/                               Beck- Mongolian Chop Squad/          Black Rock Shooter/     Bosch/
Band of Brothers/                      Beelzebub/                           Black Sails/            Boss (2011)/
Banshee/                               Being Erica/                         Blast of Tempest/       Boston Legal/
Barakamon/                             Being Human/                         Bleach/                 Bottom/
Bates Motel/                           Being Human (US)/                    Blindspot/              Breaking Bad/
Batman (1966)/                         Believe/                             Blood+/                 Breaking In/
Batman Beyond/                         Berserk/                             Blood-C/                Breakout Kings/
Batman- The Animated Series/           Better Call Saul/                    Blood & Treasure/       Broadchurch/
Batman- The Brave and the Bold/        Better Off Ted/                      Blue Bloods/            Brooklyn Ninety-Nine/
Battlestar Galactica/                  B Gata H Kei - Yamada's First Time/  Blue Exorcist/          Brothers & Sisters/
Battlestar Galactica (1978)/           Big Hero 6 The Series/               Blue Mountain State/    Btooom!/
Battlestar Galactica (1980)/           Big Love/                            Blue Planet II (2017)/  Buffy the Vampire Slayer/
Battlestar Galactica (2003)/           Bionic Woman/                        Boardwalk Empire/       Bugs Bunny (1938-2004)/
Battlestar Galactica- Blood & Chrome/  Bitten/                              Bob's Burgers/          Bunny Drop/
Baywatch/                              Blackadder/                          Body of Proof/          Burn Notice/
BBC Documentaries/                     Black Books/                         Bokurano/

c:
Californication/           Chicago Fire/     Cobra (2020)/                          Cosmos- A Spacetime Odyssey/
Camelot/                   Chicago Med/      Code Black/                            Cougar Town/
Canaan/                    Chicago P.D/      Code Geass- Lelouch of the Rebellion/  Coupling/
Caprica/                   Chihayafuru/      Cold Case/                             Covert Affairs/
Cardcaptor Sakura/         Childhood's End/  Columbo/                               Cowboy Bebop/
Carnivàle/                 Chobits/          Community/                             Criminal Minds/
Castle (2009)/             Chrono Crusade/   Conan (2010)/                          Crossing Lines/
Catch 22/                  Chuck/            Constantine/                           Crusade/
Chappelle's Show/          City Hunter/      Continuum/                             CSI- Crime Scene Investigation/
Charger Girl Ju-den Chan/  Clannad/          Cops/                                  CSI- Cyber/
Charmed/                   Claymore/         Cosmos/                                CSI- Miami/
Cheers/                    Cobra/            Cosmos (2014)/                         CSI- NY/

new:
[chuck@lizum tl.2006]$

F. Move contents of a → new

a:

b:
Babylon 5/                       Battlestar Galactica- Blood & Chrome/  Big Hero 6 The Series/  Blood & Treasure/       Breaking Bad/
Baccano/                         Baywatch/                              Big Love/               Blue Bloods/            Breaking In/
Bakuman/                         BBC Documentaries/                     Bionic Woman/           Blue Exorcist/          Breakout Kings/
Band of Brothers/                Beauty and the Beast (2012)/           Bitten/                 Blue Mountain State/    Broadchurch/
Banshee/                         Beavis and Butt-Head/                  Blackadder/             Blue Planet II (2017)/  Brooklyn Ninety-Nine/
Barakamon/                       Beck- Mongolian Chop Squad/            Black Books/            Boardwalk Empire/       Brothers & Sisters/
Bates Motel/                     Beelzebub/                             Black Lagoon/           Bob's Burgers/          Btooom!/
Batman (1966)/                   Being Erica/                           Black Mirror/           Body of Proof/          Buffy the Vampire Slayer/
Batman Beyond/                   Being Human/                           Black Rock Shooter/     Bokurano/               Bugs Bunny (1938-2004)/
Batman- The Animated Series/     Being Human (US)/                      Black Sails/            Bones/                  Bunny Drop/
Batman- The Brave and the Bold/  Believe/                               Blast of Tempest/       Bored to Death/         Burn Notice/
Battlestar Galactica/            Berserk/                               Bleach/                 Bosch/
Battlestar Galactica (1978)/     Better Call Saul/                      Blindspot/              Boss (2011)/
Battlestar Galactica (1980)/     Better Off Ted/                        Blood+/                 Boston Legal/
Battlestar Galactica (2003)/     B Gata H Kei - Yamada's First Time/    Blood-C/                Bottom/

c:
Californication/    Chappelle's Show/          Childhood's End/  Cobra (2020)/                          Continuum/                    Cowboy Bebop/
Camelot/            Charger Girl Ju-den Chan/  Chobits/          Code Black/                            Cops/                         Criminal Minds/
Canaan/             Charmed/                   Chrono Crusade/   Code Geass- Lelouch of the Rebellion/  Cosmos/                       Crossing Lines/
Caprica/            Cheers/                    Chuck/            Cold Case/                             Cosmos (2014)/                Crusade/
Cardcaptor Sakura/  Chicago Fire/              City Hunter/      Columbo/                               Cosmos- A Spacetime Odyssey/  CSI- Crime Scene Investigation/
Carnivàle/          Chicago Med/               Clannad/          Community/                             Cougar Town/                  CSI- Cyber/
Castle (2009)/      Chicago P.D/               Claymore/         Conan (2010)/                          Coupling/                     CSI- Miami/
Catch 22/           Chihayafuru/               Cobra/            Constantine/                           Covert Affairs/               CSI- NY/

new:
$#! My Dad Says/        90210/                              Air/               Almost Human/           An Idiot Abroad/                      Ashes to Ashes/
07-Ghost/               90 Day Fiancé: Before the 90 Days/  Air Gear/          Alone/                  Animaniacs/                           Ash vs Evil Dead/
11.22.63/               Absolutely Fabulous/                Aishiteruze Baby/  Alphas/                 anohana- The Flower We Saw That Day/  As Time Goes By/
11eyes/                 Accel World/                        Akame ga Kill!/    American Dad!/          Another/                              Attack on Titan/
12 Monkeys/             According to Jim/                   Alcatraz/          American Horror Story/  Another Life (2019)/                  Avatar- The Last Airbender/
24/                     A Certain Magical Index/            Aldnoah.Zero/      American Idol/          Anthony Bourdain- No Reservations/    Avenue 5/
2 Broke Girls/          A Certain Scientific Railgun/       ALF/               Ancient Aliens (2010)/  Aqua Teen Hunger Force Forever/       Awake/
30 Rock/                Adventure Time/                     Alias/             Andromeda/              Archer (2009)/                        Awkward/
3rd Rock from the Sun/  Afro Samurai/                       Alle Mot Alle/     Angel/                  Arrested Development/
666 Park Avenue/        Agatha Christie's Poirot/           'Allo 'Allo!/      Angel Beats!/           Arrow/
800 Words/              Ah! My Goddess/                     Ally McBeal/       Anger Management/       Ascension/
[chuck@lizum tl.2008]$

G. Restructure complete, update PMS

  1. Scan files manually
  2. Empty Trash
  3. Clean bundles
  4. No files or metadata lost
    Screenshot from 2022-03-02 21-32-14
    Screenshot from 2022-03-02 21-32-14781×709 73.1 KB

As expected, the media has been restructured and no metadata lost.

Please advise. What is different?

10

Sure, glad to help.

So, your root dir is /vie/tl, you crated new subdir and moved files there. This is different indeed, it works and helped me to avoid my metadata loss.

The difference is that my folder structure is based on MergerFS logic without actual fuse FS, I’m doing naming manually.
Every disk in my setup (for the Plex part) has the same (or very close) folder tree structure with different files.

HOW IT WORKS

mergerfs logically merges multiple paths together. Think a union of sets.

A         +      B        =       C
/disk1           /disk2           /merged
|                |                |
+-- /dir1        +-- /dir1        +-- /dir1
|   |            |   |            |   |
|   +-- file1    |   +-- file2    |   +-- file1
|                |   +-- file3    |   +-- file2
+-- /dir2        |                |   +-- file3
|   |            +-- /dir3        |
|   +-- file4        |            +-- /dir2
|                     +-- file5   |   |
+-- file6                         |   +-- file4
                                  |
                                  +-- /dir3
                                  |   |
                                  |   +-- file5
                                  |
                                  +-- file6

Plex, while presenting lib items, actually does the merger itself beautifully, it looks just like an above example if I switch library to folder view instead of ‘grid’ or ‘list’.

So you actually need one directory for test vie/tl/a, but make a subdir vie/tl/a/sub, put all the files in it and go from there.
Then in TLDR lib properties change vie/tl/a to vie/tl/a/sub.
When auto-rescan after saving properties finishes, files still play and not marked as trash.
Then, hit “Empty trash” and you will see in server “Status” page as PMS deletes all the files.
Next, hit “Scan” manually, and all files are added back as if they were new.

11

Info here:

  1. /vie is my NAS (‘vienna’) and the main grouping directory (as top level)
  2. tl for TL;DR :rofl: and the mount point
  3. then a, b, c, and new as the participant directories on the NAS
  4. Copied files into a, b, and c.
  5. For purpose of testing, moved contents of a → new

You would like me to change the path for one of them (let’s use b),
to be something like /vie/tl/b/bb , with all content resulting in /vie/tl/b/bb/_stuff_ ?

Side note: MergerFS has caused people issues. I know it does work but you must obey its rules as well as PMS’ rules.

12

No, what you are creating was my end goal.

  1. Assume that there was only one b to begin with all the files in several levels of depth.
    And point TL;DR to /vie/tl. One path is sufficient for the test. This is what I was starting with.
  2. When all files are in b and library is fully scanned perform the change, pointing TL;DR to /vie/tl/b (without moving any files, because it is already there).
  3. then it will find all the same files, but fails to recognize them as valid and deletes for some reason when “empty trash” is pressed.
13

Did you add the new location directory and “scan files” before starting the moving around?

If you move the files before you add the new location directory, I can see how it will delete them. (the new location directory is added AFTER it’s successfully able to scan that directory one time)

14

:man_facepalming:

Think I see it right here.

Preparation
1. Turn off “Empty Trash after Library Scan”
2. Turn off “Automatically Update my Library”
*  MISSING STEP TO ADD NEW LOCATIONS TO PLEX
3 .Make all the moves
15

I figured that might happen and never tried to use it with plex. All is done manually. It is just convenient for tiered storage and since PMS does the the magic itself. I am just following MergerFS naming principles.

16

I apologize for the typo in the TL;DR How-To.

I’m fixing it now.

The documented way works but is messy by comparison.

17

I was never moving files that were getting deleted. All I done is change the path in library properties to a subdirectory.
The goal was to move files that I needed to exclude from library to a new subfolder that won’t be included in new path.

18

Did you EDIT the path?
(Remove one and add another at the same time) ??

19

I tested both. Even if you do it at the same time, even if you keep /vie/tl and add /vie/tl/b, rescan, then edit again to delete /vie/tl, result is the same.

I was stopping server and rolling SQL database back.

20

That’s how I’ve always done it:

  1. EDIT section - Add, SAVE … and let it scan
  2. EDIT section again - REMOVE old path, SAVE … and let it scan.

It never gave me any grief.

You were playing with the DB?
Did you get both blobs and plexapp databases ?

1 Like