Music metadata

[UKenGB]

Server Version#: 1.18.3.2156

Although having had a Plex Pass for several years now, I’ve held off from using Plex much as I was having a hard time getting it to work how I wanted. I would now like to persevere with this however and although it does get used for stored video recordings, I want to get music working correctly.

I have been working with music data for about 40 years and some of that time in the computer and music industries, so it is not new to me. But, I have specific requirements. Nothing irregular, but I do need it to work correctly.

First of all, I am using iTunes to curate the media as it is an excellent tool for this. However it does mean that the file and folder structure and naming is out of my control (but iTunes uses a structure that Plex CAN deal with). I can however state that for the purposes of this ‘discussion’ my music files are all correctly tagged. The metadata tags are THE definitive data for the music. But when creating a Plex library, I am still unable to achieve the simple requirement of having Plex actually use all that meticulously crafted metadata.

Having trouble with troubleshooting a large music library, so I created 2 new libraries with a very small subset of my music. Small enough that changes and/or entire rebuilds of the libraries takes hardly any time. So…

These 2 test libraries were set up almost identically and both with ‘Prefer local metadata’ set/checked. Artist bios also and embedded genres as well. The fundamental difference I wanted to compare was using the new ‘Plex Music’ scanner and agent compared to ‘Plex Music Scanner’ and ‘Personal Media Artists’. These produced very different results.

The first (Plex Music) took my Sort Artist tag, matched the artist and all their albums. But the actual artist name was changed. Only the case/capitalisation of the text, but it was clear that Plex was using the Name from wherever it was doing the matching for additional data. Well mostly. A few were not actually changed which is even harder to understand. How can it be inconsistent like that?

The second library (using Plex Music Scanner etc) did NOT change the artist’s name, but duplicated that into the Sort Artist field (so artist list sorting is, well, wrong). No other data seems to have been retrieved for the artist, nor their albums. No artwork whatsoever is displayed. So basically, no match of anything and just seems to have used the filenames and not any of those files’ metadata.

Neither of these is correct or acceptable. What I want is not complicated. Everything I NEED is contained in each file’s metadata tags. All the rest that Plex is able to download is very nice, but it must NOT, under any circumstances overwrite what already exists in the embedded tags. That is sacrosanct. Other data can be added, but data from the embedded tags MUST be used. I’m not being weird here. It’s just the logical way for any music software to operate, unless it is specifically configured to ignore embedded tags and fetch everything from on-line. I understand there are some who may wish to do that, but it’s no good for what I (and many others it would appear) need.

Can anyone advise me of the correct configuration to ensure that Plex honours ALL my embedded tag data. It’s just no good if Plex makes it’s own choice about what data it plucks from where. I need to tell it to use my data and not mess with any of it, only adding other data not available from the file itself.

I have some other issues with Plex, but want to stick to this very important problem here. Using the ‘Plex Music’ combo gives the best result, but how can I stop it from overwriting my Artist names? Yes I could edit them, but don’t fancy doing that for 800 artists. Certainly not when there’s a very distinct possibility of having to do it again sometime. I want to be able to set up a library, let it work away and for everything to then be correct. This is NOT too much to expect. I can start a new iTunes library file and throw everything at that and everything, I mean EVERYTHING will be as it is in the files. I have never come across any other music software that is so determined to throw away customer data in favour of what its developers (who have no idea what users actually want) thought when writing the code.

I ask again. How can I create a library and have all my metadata used, unmolested?

[UKenGB]

In fact it’s a little more inconsistent than I first thought.

Most artists have been set with my embedded Sort Artist tag with their Artist name a capitalised (i.e. upper case first letter of each word, rest lower case) version of my Artist tag. But…

In some cases, the Artist name has not been changed from my embedded tag yet the Sort Artist HAS been overwriten from my tag. In one case I then ‘refreshed metadata’ and it was modified to the more standard result of capitalised Artist name and my unmodified Sort Artist tag. So in that case there was a matching error in the original ‘import’. I might have an idea of why in that case which I have asked about in another thread.

The major problem remains. When I set ‘Prefer local metadata’ (which is specified to include embedded tags), that leads me to expect that any data from embedded tags will NOT be overwritten and anything found on-line will merely supplement that data extracted from the file. This is not a wild assumption. The UI clearly states that this is the case, but it is not. The Artist Name and Sort Name may both be overwritten, which is exactly the opposite of what that setting is apparently supposed to do.

I don’t really want to start a semantic argument about the word ‘bug’, but when software does not do what it clearly states it will do, well, that’s exactly what it is. A bug.

If the import process intends to overwrite embedded tag data, then the UI is lying. But that’s still a bug. Albeit a different one.

Anyway, semantics aside, it is imperative that I find a way to overcome this problem. I need to find a way to ensure that when setting up a library and importing my music files, the embedded tag data is NOT futzed with. Why do developers think they know better. I have spent countless hours getting all my music data correct. Who are they to arbitrarily decide to overwrite it. Especially when having applied the setting to NOT do this.

These are the sort of shortcomings that have caused me to push Plex aside and use something better that respects my wishes in this regard, but I now want to get this sorted. Someone please enlighten me. How can I ensure that Plex honours my embedded tags? All of them.

[certuna]

You’re correct, no matter what you specify in the settings, Plex does not treat embedded metadata as sacrosanct and will always try to match and ‘correct’ artists against some online database (Musicbrainz probably). Which you can then correct back by doing individual manual edits in Plex.

It’s one of the many bugs in the new “Plex Music” scanner, you’re not alone, the Metadata subforum is awash with people who are just as annoyed as you and me.

The old “Plex Music Scanner” is legacy and will probably be dropped soon from PMS.

At the moment you cannot work around this problem, you’ll just have to live with it. Just keep pushing and hoping they’ll fix it some day.

[UKenGB]

It does look to me as if the new Plex Music combo does an almost very good job whereas the older Plex Music Scanner currently is hopeless, so they might as well get rid of that.

In fact, after some more trawling through the entries in my music library that had been treated correctly, I realised that AlbumArtist seems to be critical. I know it can be important, but it shouldn’t be if it is in fact the same as the Artist tag. iTunes is pretty smart in this repect and if AlbumArtist is empty, then Artist is used instead (just like Sort tags that if set are used in preference to the regular version of the tag). But Plex is not that smart.

For tracks that had their AlbumArtist and SortAlbumArtist tags set, Plex uses both those for the Artist record that it creates. This is great and ensures that the Artist is now represented and sorted perfectly, as I want and as specified in the file’s tags. But, if those tags were not set, Plex takes the Artist tag, matches against that and writes the tag according to what was returned from the search. If the SortArtist tag was set, that is imported by Plex, but if not set, Plex simply copies it from however Artist has now been set during the matching, so these may end up different from the embedded tags.

As I said, none of these empty tags matter in iTunes. That is how iTunes functions so it doesn’t need any tags to be duplicated. But Plex does. So…

I am now working through all my music to ensure that the [Sort]AlbumArtist tags are set, i.e. simply duplicated from thier non Album versions, rather than left empty as iTunes would normally.

Copying those tags in all those files is proving problematic as iTunes is simply choking on any scripts that try to update nearly 20k tracks (it’s definitely getting worse in this regard), but I’ll get them all set eventually and then they should all be imported correctly by Plex. We’ll see, but I am hopeful.

I’ll update here once I’ve got the tracks updated and set up the library (again, sigh).

[certuna]

Empty Album Artist tags for compilations are actually how it should be, otherwise you break localisation (ie, hardcoded “Various Artists” will be wrong for any other language).

Plex now actually works with Various Artists compilations (ie no Album Artist) but it will tend to pick the wrong album artist from its database. You can change this manually with a SQL editor in the Plex db, which is way faster than editing all tags manually. It’s a one time thing, I’ve detailed it here: How do I prevent Plex agent from renaming AlbumArtist 'Various Artists' into 'Various'?

If you still want to do this workaround in iTunes, you don’t need scripts, make a smart playlist of tracks where “part of compilation” is true, and AA is empty. Select all, mass set desired tags, save. If you happen to have thousands of compilations, go to sleep because it’ll take hours.

[UKenGB]

If I need anything modified, I can use either AppleScript or JavaScript in iTunes or just a shell script, depends on how best to achieve what I want. So that’s not a problem.

I realise that I could just write some SQL to correct these issues in the Plex SQLite database, but that’s not ideal as it would mean having to do so for any subsequent rebuild/reimport. I want to get the data correct so Plex simply gets it right. Every time.

The real problem is to understand what Plex is doing when importing music media and what it does with the data it finds in the embedded tags. Once this is fully understood, then the metadata can be suitably constructed so all is correct once in Plex.

Compilations are an issue. First of all, what are they? When iTunes introduced the ‘Compilation’ tag, it was a way to avoid all the individual artists being listed and hence was useful for all those albums of tracks by different artists. But in truth, a compilation album is also an album by one artist that comprises independent tracks (probably from other albums or singles) that are grouped together for this release, as typified by ‘Best of…’ albums. These are not the same as the aforementioned albums by multiple artists.

However, the Album Artist tag was then available and that really solved the problem. For albums by multiple artists, set the AlbumArtist tag to something suitable, like Various Artists (although I use VARIOUS ARTISTS). All these are then grouped under that one artist. Leaving the ‘Compilation’ tag available for better use to identify an artist’s compilation albums from the rest of their regular albums.

I don’t see any evidence that Plex uses the ‘Compilation’ tag, which is fine with me. All my albums by multiple artists use ‘VARIOUS ARTISTS’ for their AlbumArtist tag and without any actual artist match, that is copied to the SortArtist once in Plex. Viewing any such album lists the tracks along with the actual track artist. So, as far as I’m concerned, Plex is doing all that 100% correct. I have since duplicated the AlbumArtist tag to SortAlbumArtist, just to be sure and it keeps it all consistent with all 4 [Sort][Album]Artist tags complete and correct for all tracks, even though it means much duplication. Worth it if it ensures Plex imports correctly.

Speaking of which, my testing indicates that as it scans the music files, it will create an Artist from the AlbumArtist (and apparently match online data from that), but if it later imports a track whose AlbumArtist is empty, even though Artist is set the same as the existing Artist, Plex will match it ok, but then simply overwrite the existing Artist name in the database with that matched Artist name.

I’ve watched this occur. As it begins to import that artist, the database record is created and displayed from the correct AlbumArtist tag, but on later finding tracks by that Artist, but with no AlbumArtist, it matches and overwrites the database with the matched name and you can watch as the displayed Artist’s name changes on screen during the import. In my case that meant seeing ‘Eric CLAPTON’ created, but then some seconds later, it changed on screen to ‘Eric Clapton’.

However, once I discovered that errant album with missing AlbumArtist and corrected it, the artist was correctly imported and correctly displayed as ‘Eric CLAPTON’ with ‘CLAPTON•Eric’ as the SortArtist.

There is one outstanding problem. On all Eric CLAPTON albums, when viewing the track, they are each listed with ‘Eric CLAPTON’ as the Artist (correct as those are the embedded tags) except for ‘Crossroads’. This is a double album and none of the tracks list an artist name - because Plex’s Artist field is empty and I cannot yet see why it gets all the other tracks (over 20 album’s worth) correct, except for this album.

Can anyone suggest why Plex would choose just this album to ignore the correctly set embedded Artist tags?

[UKenGB]

Aha, I know why it gets Crossroads wrong. Those tracks are MP3, whereas all the others are M4A. This should not matter, but…

MP3 and MP4 tags use some different actual tag names/indexes by which they are extracted from a file. It’s easy to get caught out by this and it looks like Plex has been caught out. So when importing an MP3 track, it fails to obtain the actual Artist data. Seems to work for the other MP3 tags and all MP4 tags, but it’s failing to get an MP3 Artist/Performer tag data.

I also just checked the Alice Cooper imports as they are all MP3 and sure enough, none of the tracks have an artist.

Ooops. Well this has to be a bug, so I need to figure out how best to report this.

[certuna]

The big problem with “Various Artists” as hardcoded Album Artist is that it breaks localization, and makes it a lot harder for a library to have a separate Compilations section (since having compilations wedged into the album list under “V” is just messy).

This is why V/A compilations always have an empty Album Artist field (and “Part of Compilation” = TRUE, which takes the guesswork out of it on the library side). Plex doesn’t support that tag unfortunately, it would solve a lot of problems if it did.

Another problem in Plex are albums with songs from multiple years (typically artist or label anthologies) and/or multiple genres. Currently there’s no way to make smart playlists/filters using that metadata.

If you look into the SQLite database you can see that Plex can store year and genre by track, the fields are there, but the scanner just doesn’t populate them (and the clients don’t use track-level info in search/filter). Some things are missing (like a Composer field) but all in all The more I look at it, the database design isn’t the major problem, although I’d still prefer groupings like Albums to be created on-the-fly, not as static tables.

I’d love to have a look at the “Plex Music” scanner source code since some of the bugs seem trivial to fix (like Plex picking the wrong Album Artist for compilations, preferring Musicbrainz info over embedded tags, not populating track-level year & genre), but unfortunately it seems to be closed.

[UKenGB]

I don’t agree about AlbumArtist being empty meaning a Various Artists type album. In conjunction with the Compilation flag, yes it could work, but in fact an empty AlbumArtist tag actually just means use the Artist tag instead, although that could be the other way around. Leaving it empty to indicate various artists is just a barfy kludge.

As you say however, it would be good if Plex used the Compilation tag, but that still leaves the question of how such albums would be displayed. iTunes includes such tracks in an artist’s list, but also can include a ‘Compilations’ entry in the artists list which is far worse that having ‘Various Artists’ listed. I don’t see any problem with that in any case, various artists is EXACTLY what it is and makes total sense to include that in the artists list. Surely any text could be used instead. How would Plex react if all such albums had ‘*’ in the AlbumArtist tag? or what about an emoji/symbol like ? Wouldn’t that be ‘international’? I don’t know if Plex requires an exact phrase to treat a various artist album correctly. If it does, hardly surprising they used an english phrase as Plex is in the US and they hardly know the rest of the world exists.

[certuna]

I set the Sort Artist for “Various Artists” in Plex to the last letter of the Japanese Unicode alphabet make sure that V/A compilations end up at the bottom of the Album list, it’s a hack but until Plex supports a Compilations flag (either using the tag, or just as an internal flag in the database) it will have to do.

[UKenGB]

Not a hack as I see it. How else would you view these albums. They have to be listed somewhere in the artists list and using the Sort tag to define the sort order is EXACTLY why the Sort tags were introduced. So users could define how they wanted their lists sorted.

I’ve been looking into using the emoji as emojis do seem to work ok. However, does Plex rely on that actual ‘Various Artists’ string in any way? What happens on import if you set both AlbumArtist and SortAlbumArtist to the same single character (or anything other than ‘Various Artists’)?

[certuna]

Well ideally compilations have their own section.

Plex does use the “Various Artists” tag for any subsequently added compilation, there’s a bug though where it picks the wrong album artist (all my compilations suddenly had The Muppets” as album artist) but I’ll have to dive a bit deeper to find out what the “Plex Music” scanner does exactly.

[UKenGB]

As far as I can tell, Plex understands the actual string ‘Various Artists’ and doesn’t try to match that and obviously get it wrong as it’s not a real artist. But, we have a problem with “Various Artists” simply being shown in the artists list. So, here’s my suggestion…

A new configuration option to specify the actual text that we want Plex to understand as meaning Various Artists. So users can simply enter any text they like as the AlbumArtist tag for all tracks of VA type albums and Plex will understand that to mean Various Artists and to then import as it does now for files tagged as ‘Various Artists’ in the AlbumArtist tag. We can then use whatever desired as the SortAlbumArtist tag to ensure this entry sorts where we want it in the artists list.

In that case I would use for both the AlbumArtist and SortAlbumArtist tags. I like that such an icon/emoji represents quite well what it is - internationally, it stands out from the other artists in the list and it sorts at the end anyway. The problem is that Plex will not understand that as meaning Various Artists. For this we need that new preference setting. Are they listening to this?

As certuna says though, we also need to be able to see VA content within an artist’s page along with their own albums. I think this should be in a separate section of the page and not just mixed in with the rest as that’s just messy. On an artist’s page I want to see their own albums (sorted in the way I want) and also their other tracks from VA albums. Plus, other tracks on which they have participated. This latter is however tricky as it would require the ability to assign multiple artists to a track. Not impossible and really is something Plex should be working to incorporate, but it’s not as simple as simply displaying an artist’s tracks from VA albums, on their own page. iTunes does this, but they’re mixed in together which is not the best way to handle it. In my opinion. So here’s another config option we need. A ‘Display VA tracks in separate section’ checkbox. Even better, 2 options. ‘Show VA tracks’ and when that is selected, the aforementioned checkbox that allows the user to configure whether they are shown ‘in-line’ or in their own section on the page. I would simply select both checkboxes, but all users could then set it how they best wanted it.

I also have a problem with Singles. I have quite a few, some of which do have album information, but the majority do not. They are NOT the same as VA albums. In most cases there is NO album, but Plex keeps trying to match. I didn’t realise there were so many groups called ‘The Singles’ or similar. Matching to these is obviously hopelessly wrong and I’ve not figured a way to tell Plex to NOT match that artist to anything. Just list the artist with that name. It’s like how it treats the Various Artists AlbumArtist, but using different text.

I’ve thought a lot about this and they just cannot be incorporated into the VA group. If the import worked as it should, there’d be some VA albums with just a single track and a huge section under Unknown Album. I wouldn’t mind the latter, but how to then associate those tracks with the other single track albums. There is nothing that links these together so I cannot simply play all the singles. There has to be a separate section for these ‘tracks by various artists’ as that is not the same as ‘various artists albums’.

I’m struggling to figure out how to achieve this in Plex. Currently in iTunes I set the AlbumArtist as ‘Singles’ (which works perfectly), but when I import them, Plex matches that ‘virtual’ artist to the artist of one of the imported tracks, which is obviously inappropriate. How can I simply tell Plex to ‘NOT match’ that artist with anything, just like for VA? I made various attempts, but when the result was no good, I couldn’t go back. The only option I can see is to delete the library and start again. Not a big deal now as it’s just a small test library, but I want to get this right before I set up the real library with a LOT more content.

Anyone comment on how to deal with singles?

My best suggestion is a modification of the configurable VA string/text I mentioned earlier. What we need is to be able to configure multiple strings for this purpose. Enter any number of strings, each of which will be used by Plex to mean ‘DO NOT TRY TO MATCH’. So we can have multiple virtual artists that will behave like VA, but which will be listed separately in the artists list and sorted with the text we specify - all in the embedded tags.

There you go Plex. Designed for you. Now you just have to implement it (which wouldn’t be hard, really it wouldn’t).

[certuna]

Basically, that’s implementing a “Part Of Compilation” flag, but in a roundabout way

But really, at this point I’m happy with anything. I would still strongly advise the Plex devs to:

a) add a “part of compilation” column to the metadata_items table in the Plex internal SQLite database, that way all subsequent client (app/web) code you’ll ever develop in later years can pull out, process and group compilation tracks and compilation albums with a simple sql query without much difficulty.

and

b) implement the reading of “TCMP” (ID3v2), “cpil” (mp4/m4a) and “COMPILATION” (FLAC/Vorbis Comments) tags in the “Plex Music” scanner, as it is so, so much easier to just simply read this info directly from an existing tag and dump that in the database versus writing all kinds of convoluted code trying to figure out if something is a compilation or not based on the exact album artist tag, folder name, etc.

Lastly, even Musicbrainz (the company Plex partners with for online metadata scraping) uses the “part of compilation” tag, see their own documentation: https://picard.musicbrainz.org/docs/tags/ and Release Group / Type - MusicBrainz

Come on Plex, this change has to come - don’t fight it

The issue of grouping Albums, Singles, EP’s, Live releases under separate headings is another one, and unfortunately that’s not easily done with just embedded tags, as unlike with Compilations, there’s no “Release Type” tag in ID3/MP4/FLAC. This is actually where online matching can fill in the gaps, as Musicbrainz does store this info (see their documentation: Release Group - MusicBrainz ).

Now that Plex has joined up with Musicbrainz, I’m perfectly fine with Plex conforming to their metadata structure (it’s quite well thought out, and handles compilations, singles etc well), but embracing the Musicbrainz way of things does mean significant changes to the current Plex database structure, which in turn means changes downstream in all the client code.

[UKenGB]

The trouble with MusicBrainz ‘Release Groups’ is that there’s no way to flag a music file as being in any particular Release Group. There is the ‘Grouping’ tag, but that pre-exists for other purposes.

In any case, their analysis still looks to me to be an attempted lash-up of trying to fix what was wrong the first time around. When I worked in the music industry I had a lot to do with MCPS (UK rights organisation) and they had fully and correctly analysed how to store music data. It had to be PERFECT as it is used to cut the checks to the contributors. They are the only organisation that ever actually got it right.

However, my idea for Plex to be configured with multiple text strings that would be recognised as meaning ‘don’t match’ (hence the same as Various Artists) requires no additional tags. Just set the AlbumArtist tag to any string you want, tell Plex and that music would be imported just like VA, but each in their own Artist grouping within the library. There are several advantages to this:-

• It requires no additional tags to be ‘built-in’ and used.
• It’s totally flexible and under user control.
• It’s scalable, so you could just have Various Artists, or as many other groupings as desired.
• Very simple for Plex to implement.
• It’s international. Every user can use whatever strings they want to list and sort these entries in the artist list.
• Regarding the previous point, this means even Various Artists could be replaced by any localised version that means the same thing. Isn’t this what I’ve seen several users demanding?
• The ‘Compilation’ tag provides almost none of the above and as has been pointed out is little used outside iTunes. It was in fact a kludge that was only needed until the AlbumArtist tag was introduced and provided better functionality. Since then, the ‘Compilation’ tag really is redundant and not worth wasting any time on.