Technique for merging servers

Server Version#:1.17.0.1709 (FREENAS)
Player Version#: (not sure how to find, not sure if it applies)

Unimportant (?) background: Recently I posted a question about reviving a server that was way back-level (UBUNTU 14.04LTS), and the suggestion was to upgrade that to 20.04. While I agree with the approach, the move is problematic for reasons that seem to be related to the server hardware - it’s getting old. Now I want to bite the bullet and remove that old technology from the list of critical devices. This server holds my home movies and a few other select TV shows and videos, a couple hundred or so that have nice descriptions and so on. It also contains thousands of music files (MP4 and mostly MP3) that have been meticulously categorized and so on.

My new goal is to merge the movies and_their_metadata from the expiring system into my existing FREENAS server, which has plenty of storage to accommodate the library contained on the old. But it would be a real pain to have to retype all that metadata, set the sort values again, and so on. I can easily copy the MP4s from the old server to the new, and I am looking for a way to merge the existing library into the new server. Some, but not all, of the OLD categories exist also on the new. MUSIC, for example, exists on both but there are no duplicates. HOME MOVIES exists only on the old server.

Is there a way for me to do such a merge operation and not have to redo all that metadata work that represents many hundreds of hours of effort? I will provide any other information needed. I’m coming from UBUNTU 14 (if version number is on the filesystem, please tell me where and I will provide it)… and moving to TrueNAS 11.3-U2.1, Plex Server Version#:1.17.0.1709

You can’t “merge” HOWEVER :smiling_imp:

If you

  1. Know what the new absolute media pathnames will be
  2. Can completely emulate those pathnames
  3. Add the “new, emulated” diretories as media sources
  4. Allow Plex to think it sees duplicates

and

can create a tar of the Linux Plex Media Server directory and all below it to capture the metadata and database

then

you’ll have a mirror CLONE you can take over to the new machine and lay ON TOP OF the new installation.

See what I’m thinking?

Wow, thanks so much for the time and thought! I think we’re closing in on it here…

Not sure I follow all of this, but I think I follow most. I can work any file-related magic necessary on the servers (once I understand what needs to happen), and can create a tar or zip folder on one system and lay it down on the other. No issues there at all. but not certain about some details.

The old instance, which cannot be brought back online due to installation issues, had libraries at these locations:

/svr/Media/HomeMovies/
/svr/Media/IFYL Podcasts/
/svr/Media/Movies/
/svr/Media/Music/
/svr/Media/Music Videos/
/svr/Media/Music-Christmas/
/svr/Media/TV/
/svr/Media/Trailers/
/svr/Media/UHD 4K/

The current locations on the FREENAS server are (from Plex’ point of view, which has its root at /mnt/Main-HDDs/iocage/jails/plex-nas1/root/):

/Media/Converted-DVDs
/Media/Favorites-to-Share
/Media/Movies
/Media/Music
/Media/MusicChristmas
/Media/MusicPlex
/Media/MyVideos
/Media/NewStuff
/Media/TV

So I think you’re saying to:

  1. create folders named “svr/Media/HomeMovies” and so on under the /mnt/Main-HDDs/iocage/jails/plex-nas1/root/ folder of Plex-NAS1
  2. Add these folders as media libraries to Plex-NAS1 and then shut it down
  3. tar-backup the old server metadata hierarchy at:
    /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata
  4. untar that backup onto Plex-NAS1 at
    (/mnt/Main-HDDs/iocage/jails/plex-nas1/root)/usr/local/plexdata-plexpass/Plex Media Server
  5. Populate the new folders on Plex-NAS1 and bring the server up.

Is that about right?

I didn’t follow what you meant about allowing Plex to think it sees duplicates. I am also not sure if I need to (nor if it would be bad to) go one folder “up” from the Metadata folder… Gut tells me that would not be good…

In general, the above sounds good to me (one who doesn’t understand the nuances of Plex under the covers)

I don’t know how “jails” work.
If those are containers – fine.

What is important is:

If your media, how it will look to Plex once installed, is:

/Media/Converted-DVDs
/Media/Movies
/Media/Music
/Media/TV
etc
etc

Then that’s the exact directory structure we create on the existing Ubuntu 14 system.

What directory structure are you currently using on the Ubuntu system?

I ask this so we can take the existing media and relocate every piece (1:1) into the new structure.

If you will provide a few of these pieces (e.g. Movies, Music, and TV), I’ll show you how we take what exists in 14 and migrate the existing Ubuntu 14 PMS into what will be the data structure once it’s on the FreeNAS.

Thanks, ChuckPa, for all your attention! I really appreciate it!

I sat on my hands for a few hours, because I don’t want to make a rash response. Will this actually work, though? I am pretty sure I need to emulate the UBUNTU path on FREENAS, and not the other way around since that’s the one that’s in trouble. For example, this random file content:

> cat Application\ Support/Plex\ Media\ Server/Media/localhost/9/42cadce65a167f8daef1a70418593d671669031.bundle/Contents/Subtitles.xml
<?xml version='1.0' encoding='utf8'?>
<Subtitles>
  <Language code="xx"/>
  <Language code="en">
    <Subtitle codec="srt" name="The Best of Me (2014).en.srt" file="/svr/Media/Movies/The Best of Me (2014)/The Best of Me (2014).en.srt" format="srt"/>
  </Language>
</Subtitles>
dennis@hpmicro1:/oldboot/var/lib/plexmediaserver/Library   

Since the Metadata we will be copying over, unalterably (?) refers to /svr/Media as the top of its path, don’t we need to make that a valid statement on the FREENAS system? It’s a simple thing for me: create a /svr/Media structure on the FREENAS system… But not sure what steps you might be thinking that are not occurring to me…

First answer: Which is the source? Which is the destination?

Next: Emulate DESTINATION on SOURCE
Next: Migrate Existing Source structure to Emulated Destination structure.

  1. move the actual mount point to the emulated point
  2. Create a mount --bind point to make the now-relocated media look like it’s still where it was.

Last: CLONE “./Library” and all below it (tar works perfect for this) and transport to Destination.

When Destination starts, it will find all of the real media right where it already expects it.

Before we move on… what’s going to happen when I overlay Library/Application Support/Plex Media Server/Plug-in Support/Databases on the only good server I have?

Never overlay it that deep. All or nothing.

Here is the mount - bind method we used to use for moving /var/lib/plexmediaserver before systemd overrides were stabilized.

  1. mkdir /home/plexdir; chown plex:plex /home/plexdir
  2. stop plex
  3. cd /var/lib/plexmedia
  4. tar cf - . | (cd /home/plexdir ; tar xf -)
  5. rm -rf ./Library (removes /var/lib/plexmediaserver/Library)
  6. mount --bind /home/plexdir /var/lib/plexmediaserver
  7. start plex

Here’s how to play with this

mkdir /tmp/a /tmp/b
mkdir /tmp/a/dummy
touch /tmp/a/dummy/foo
mount --bind /tmp/a /tmp/b
ls -laR /tmp/b

This is the mechanism I’m proposing to use

By the way: Source = UBUNTU (now broken), with media files at /svr/Media/*
Destination = FREENAS, with media files at /Media/*

OK, I follow that much of what you’re doing. But, although this approach doesn’t cause the destination’s original db and so on to be permanently lost (they’re just not visible after mount --bind), I am missing how this will end up merging the libraries…

You’ve been patient and kind, and all I’ve been able to do is to question your approach. At this point I wonder if i’m being more nuisance than necessary. I can see this merge business is going to stretch my understanding, which is not a bad thing, but it puts someone into teacher mode.

Another approach might be to bring up a second PLEX instance on the FREENAS server, keeping them separate, as before - even though they’re on the same physical disks. Pretty sure I can pull that off, and if not then it’s nothing lost. I do really like the idea of reducing to one via some merge approach - and I’m sure I’m not the first to want to bring servers together. But in the end, that could be done later. And I don’t want to be the difficult part of anyone’s day…

Let me ask your thoughts on that. Might it be advantageous to have both servers operational and then figure out how to merge them while there’s some chance of manipulation? Also, might it be more advantageous to just have two and be done with it?

  1. Me? Teacher mode? :rofl: (It’s pretty much my job description and I gladly do it)
  2. Now that I’m FINALLY :roll_eyes: clear about source/dest and structure (can you tell it’s Friday?)

I have one question:

  • Does Ubuntu work well enough to administer it at the Plex/Web console? If so, that’s all we need it to do. The other functionality will work itself out after being relocated.

Nope. See " 1.19.3.2740-add6f438d on UBUNTU 14.04 LTS keeps crashing at startup 1.19.3.2740-add6f438d on UBUNTU 14.04 LTS keeps crashing at startup"

Thanks.

  1. You’ll need to emulate the existing Ubuntu directory structure on the new system using mount --bind from the command line ONE TIME. (Don’t reboot the new system until all work complete)
  2. cd into /var/lib/plexmediaserver/Library/Application Support/Plex Media Server
  3. tar cf /some/place/safe/PlexMediaServer.tar . Trailing dot required.

On the new system

  1. Install and start once then stop
  2. Get in at that command line and into its Plex Media Server directory
  3. Remove everything there
  4. untar the saved Ubuntu here.
cd /path/to/Application Support/Plex Media Server   (on new system)
rm -rf *
tar xf  /path/to/PlexMediaServer.tar
chown -R plex:plex .       ( supersede  "plex:plex"  or correct  or skip as appropriate )
  1. Start PMS
  2. Now, one section at a time, migrate from the Emulated Ubuntu directory names to their new real directory names. (aka. Add real, scan, See duplicate, Remove old, scan again process) for each library section until done

Looking into it… Thanks for your help.

Took a long time to get all the data copied over to the (new) FREENAS server from the UBUNTU system. But I have completed the process.

Provisioned and initialized plex2-nas1
Shut down plex2-nas1 service (service plexmediaserver stop)
Created a tarball of UBUNTU data from
/var/lib/plexmediaserver/Library/Application Support/Plex Media Server
Copied all media files from UBUNTU /svr/Media/* to /svr/Media
on plex2-nas1:

cd "/Plex Media Server "# (Yes, that's at root level - installation default)
tar cf /PMS-original.tar .
rm -rf *
tar xf UBUNTU-metadata-tar
service plexmediaserver start
root@plex2-nas1:~ # ps aux
USER    PID %CPU %MEM    VSZ   RSS TT  STAT STARTED    TIME COMMAND
root  14370  3.3  0.0   7488  3812  0  SJ   09:26   0:00.01 -csh (csh)
root  14369  2.8  0.0   6948  2856  0  SJ   09:26   0:00.00 login [pam] (login)
root   6655  0.0  0.0   6544  2352  -  IsJ  08:42   0:00.00 dhclient: epair0b [priv] (dhclient)
_dhcp  6701  0.0  0.0   6544  2464  -  ICsJ 08:42   0:00.00 dhclient: epair0b (dhclient)
root   6780  0.0  0.0   6428  2468  -  SsJ  08:42   0:00.01 /usr/sbin/syslogd -c -ss
root   6845  0.0  0.0   6460  2344  -  IsJ  08:42   0:00.01 /usr/sbin/cron -J 15 -s
plex  12356  0.0  0.0   6384  2028  -  IsJ  09:12   0:00.00 daemon: /usr/local/share/plexmediaserve
plex  12385  0.0  0.1 189016 77552  -  IJ   09:13   0:03.66 /usr/local/share/plexmediaserver/Plex_M
plex  12386  0.0  0.1 112280 63256  -  INJ  09:13   0:04.41 Plex Plug-in [com.plexapp.system] /usr/
plex  12391  0.0  0.0  32012 20592  -  IJ   09:13   0:00.05 /usr/local/share/plexmediaserver/Plex T
plex  12392  0.0  0.1  66456 38988  -  IJ   09:13   0:01.38 Plex Plug-in [tv.plex.agents.music] /us
plex  12393  0.0  0.1  73880 47956  -  IJ   09:13   0:01.87 Plex Plug-in [com.plexapp.agents.thetvd
plex  12419  0.0  0.1  65688 37684  -  IJ   09:13   0:01.27 Plex Plug-in [com.plexapp.agents.plexth
plex  12424  0.0  0.1  72088 45376  -  IJ   09:13   0:01.74 Plex Plug-in [com.plexapp.agents.themov
plex  12454  0.0  0.1  65688 37676  -  IJ   09:13   0:01.26 Plex Plug-in [org.musicbrainz.agents.mu
plex  12480  0.0  0.1  76824 50060  -  IJ   09:13   0:02.02 Plex Plug-in [com.plexapp.agents.imdb]
root  14379  0.0  0.0   7000  2824  0  R+J  09:26   0:00.00 ps aux

But when I try to connect to Plex2-nas1, I get this page:

Log is attached.PlexMediaServerLog.zip (19.7 KB)

I’d appreciate your advice when you get the chance.

  1. You’re signed into Plex/Web . Need to be signed out for first time contact and may still need reset ownership in the Preferences.xml

  2. DO NOT use FQDN for claiming. NEVER allowed. :slight_smile:

  3. Be prepared to SSH-Tunnel

But it had been claimed, and the metadata restored.

Even so, I will document as I go along:


Got it…

Next…

Next…

Done…

Above was after I once again removed the stuff under /Plex Media Server and restored it again…

At that screen shown above, CTRL-F5 and make it do it again.

Now go grab the log files (manually).

Let’s see what it isn’t liking.

I hope you meant F5 since CTRL+F5 didn’t work for me (Windoze, Chrome). Anyway, here are the logs:
plex-logs.zip (79.8 KB)

Ctrl+F5 skips the cache.