This question is for the Plex Ninjas or Devs who were responsible for creating and/or working on Sonic Analysis.
Can you please provide a basic explanation as to how this new process works?
According to the blog post, the following is stated:
> Once you enable Sonic analysis on a music library, all tracks, albums, and artists are processed in order to place them in a Musical Universe (math nerd? This musical universe consists of points in N-dimensional space). This allows us to see how “close” anything in your library is from anything else, where distance is based on a large number of sonic elements in the audio.
Can this be explained in a way that makes sense to users? Where is the metadata source being used to create this? MusicBrainz? Other new source? How can Plex really do it with just code or a source?
Feedback is welcome to make sense of this as it becomes available.
The analysis isn’t using metadata… it’s scanning the actual media files for various characteristics and uses machine learning to place each song / album on a multi-dimensional matrix w/ all those characteristics. Imagine that like coordinates on a map where each point can be addressed with 3 parameters → east/west orientation (longitude), north/south orientation (latitude) and distance above/below ocean level (altitude). Plex is doing the same with the music files… but instead of longitude/latitude/altitude it’s using elements like loudness, tonality… (don’t know the full list; I believe I’ve been reading it’s more than 10 different factors). The closer different songs are in that multidimensional grid, the higher their sonic similarity.
The only part using metadata is the album/release categorization (live albums, compilations, singles/eps, regular albums…) – this information is indeed coming from Musicbrainz.
Pretty much. It’s like BeaTunes as well would work by analyzing music by color to get a relationship between songs.
Doesn’t just play music, also listens
beaTunes employs sophisticated algorithms to analyze your music for metadata like tempo (BPM), key, color, segments, similarities, loudness, and acoustical fingerprints. It lets you know, what’s in all those files!
Analysis is a stable foundation for great sounding playlists as well as tag lookup and acoustical duplicate detection.
Lol
how can a song have emotions…and different people respond differently to hearing the same thing.
The most successful music grading system IMO used to be the early pandora who had The Music Genome Project. Each song was actually listened to and analysed amd graded by musicians on hundreds of points.
I’m not sure if Pandora stll uses this as we cant get it here in NZ. Pandora was a music discovery thing way before it was a streaming service. It was much more accurate that the crappy crowd sourced recommedations which is more like ‘people who like xxx also like xxx’…its lowest common denominator. This is entirely why an algorythm putting a line through a graph and putting something on either side of the line is wholy inept at this task.
So Sonic analysis.
I’ve been using it since launch…minus the 3 days it took to analyse.
Let me say I’m all for a sonic analysis over those who liked…also like stuff as mentioned above. Hats off to those who did this potentially ground breaking way of connecting songs.
However so far the sonicly related music has not really been that meaningfull to me…yet. It often finds music of the same era, thats understandable and sometimes complete off the wall disimilar in every way (at least to me) tracks.
I’ve been wondering if in the future as more is learned about the results we are getting, that this will grow into something bigger, being merged with other data to produce maybe more usefull.
Apparently it’s transcoding each music file into a single channel 16khz wav file and then using that to get whatever data it uses to put into the database.
Sep 11, 2021 20:39:13.396 [12124] Debug — [MusicAnalysis/JobRunner] Job running: set "FFMPEG_EXTERNAL_LIBS=\\\\?\\C\:\\Users\\carbo\\AppData\\Local\\Plex\ Media\ Server\\Codecs\\be22e26-4019-windows-x86\\" & "C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe" -i "D:\MUSIC\Music\Golden Earring\Just Earrings (1965)\01 - Nobody but You.mp3" -ac 1 -ar 16000 -acodec pcm_s16le track-0000.wav
Sep 11, 2021 20:39:13.398 [12124] Debug — [MusicAnalysis/JobRunner] Jobs: Starting child process with pid 10752
Sep 11, 2021 20:39:13.398 [8860] Debug — [MusicAnalysis/JobRunner] Job running: set "FFMPEG_EXTERNAL_LIBS=\\\\?\\C\:\\Users\\carbo\\AppData\\Local\\Plex\ Media\ Server\\Codecs\\be22e26-4019-windows-x86\\" & "C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe" -i "D:\MUSIC\Music\Golden Earring\Just Earrings (1965)\02 - I Hate Saying These Words.mp3" -ac 1 -ar 16000 -acodec pcm_s16le track-0001.wav
Sep 11, 2021 20:39:13.399 [8860] Debug — [MusicAnalysis/JobRunner] Jobs: Starting child process with pid 2752
Sep 11, 2021 20:39:13.955 [11584] Debug — Jobs: 'C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe' exit code for process 2752 is 0 (success)
Sep 11, 2021 20:39:13.955 [8860] Debug — [MusicAnalysis/JobRunner] Job running: set "FFMPEG_EXTERNAL_LIBS=\\\\?\\C\:\\Users\\carbo\\AppData\\Local\\Plex\ Media\ Server\\Codecs\\be22e26-4019-windows-x86\\" & "C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe" -i "D:\MUSIC\Music\Golden Earring\Just Earrings (1965)\03 - She May Be.mp3" -ac 1 -ar 16000 -acodec pcm_s16le track-0002.wav
Sep 11, 2021 20:39:13.957 [8860] Debug — [MusicAnalysis/JobRunner] Jobs: Starting child process with pid 11028
Sep 11, 2021 20:39:13.983 [11584] Debug — Jobs: 'C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe' exit code for process 10752 is 0 (success)
Sep 11, 2021 20:39:13.983 [12124] Debug — [MusicAnalysis/JobRunner] Job running: set "FFMPEG_EXTERNAL_LIBS=\\\\?\\C\:\\Users\\carbo\\AppData\\Local\\Plex\ Media\ Server\\Codecs\\be22e26-4019-windows-x86\\" & "C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe" -i "D:\MUSIC\Music\Golden Earring\Just Earrings (1965)\04 - Holy Witness.mp3" -ac 1 -ar 16000 -acodec pcm_s16le track-0003.wav
I too am interested in what kind of analysis the sonic analysis is doing.
Obviously it’s not just analyzing the mood of a track, but also the timbre / instrumentation of it, since the similar tracks tend to come from the same genre or involve the same instruments. Is that right, though? If so, would one of the Plex engineers be willing to say more about what it does?
I don’t have specifics, but can tell you it takes into account over 50 metrics, and plots them in n-dimensional space to find items that are closest together.
Timbre/instrumentation is definitely part of this, as you rightly guessed. In addition to that, tempo, rhythm, texture, tone, vocalisation, volume and more are considered.
For an example of what sort of data can be extracted in a similar process, check out any AcousticBrainz entry:
To clarify: Plex does not use AcousticBrainz or the Essentia engine for Sonic Analysis; this serves only as an example of a similar service for anyone that might be curious
On top of this, the radios that make use of Sonic Analysis also go through the Plex smart shuffle process, which ‘dithers’ items to result in less repetition (true shuffle doesn’t feel random enough, see below), and favours popular (from last.fm) or highly rated tracks (your rating tags) that you haven’t heard recently by positioning them higher in a given queue.
There are definitely further secret herbs and spices that make up the final recipe, but hopefully this can satiate some of the curious cats in this topic
The article from Spotify’s engineering team about randomness and shuffling is a very interesting read, and I highly recommend checking out the resources linked to at the bottom of that post if the intersect between psychology, maths, and music interests you.
Right, that will give you a random assortment of tracks, but our brains are really good at pattern recognition, and for 99.99% of people, the end result just wouldn’t feel right, even if mathematically the shuffle was entirely random.
The same artist playing back to back 5 times in a row is just as likely to happen as any other combination of 5 artists, and while truly random, will feel very much not random. Spotify, Apple, and most other streaming services came to the same conclusion when users complained that shuffle wasn’t “shuffling properly”. Even Plex faced the same complaints before Smart Shuffle (and a few bug fixes)
Personally I care less about artists reoccurring, than repeating of the same songs, from not iterating the resulting shuffled list (however it is derived) sequentially.
In other words, if I shuffle a list of 10 tracks, I want them to play sequentially in a random order, not bounce around the list in a random order, potentially repeating themselves.
I’m not saying that plex is or isn’t doing that, only that is my own preferred method.
The original Winamp provided both operating modes, shuffling the now playing queue, or randomly bouncing to different queue positions.
The only artist that makes any sense would be Justin Bieber. Why do I have Latin artists here when none of the music on this album matches against those artists? These artists have reggaeton sounds and last I check Maroon 5 ain’t reggaeton.
Is the point of Sonic Analysis is to have music that sounds a like so you have a better listening experience than what other machine learning methods can do?
And for anyone gonna say, well you don’t have enough hence not enough data, I have 28, 139 tracks!
Why can’t Plex just go back to use a company like Gracenote that works with music labels to get the best info instead of relying on the public to do so? I get it; the public, free domain might be appealing because you have music fans participating in the information of music; but the reality is, there is way too much music. We do need help.
Spotify gets this and this is why they are #1. They have editors who overlook playlists, but it’s the machine learning that empowers their business.
If you don’t wanna use a company like Gracenote; use TuneBat (www.tunebat.com) which empowers information about the music to help users listen to music. Personally, I feel the whole approach to Plex Music using MusicBrainz and AllMusic is really limited because these companies especially MusicBrainz relys on the public to provide the data. If the data is wrong, editors on the site have to review and then fix, but it is not entirely perfect. Whereas Gracenote leaves and breaths this approach. I’d pay a bit more for Plex if Plex invented in better metadata to empower the service.
There are definitely outlier tracks/albums/artists where the sonic analysis doesn’t seem to generate great results, although sometimes that’s more about my assumptions of what a “great result” actually is.
Completely different genres sometimes have albums I wouldn’t associate in my head, but listening critically, I can see how the analysis came to that conclusion.
There isn’t really a critical point where a library becomes “big enough” for sonic analysis to work well, it’s more that as a rule of thumb, the bigger your library, the better your results are. I’ve got about 90k tracks and it’s been nothing short of amazing. That said, I know people with much smaller libraries that are also very happy with it.
Where I’ve seen it fall down most often is with artists/albums/tracks that are outliers in your library. If I had a library of 30,000 tracks, but it contained nothing but mid-90s SKA, and then I added Mozart’s Symphony No. 40, the results would obviously be less than ideal!
In my experience the new metadata engine performs better for my library than Gracenote ever did. Tivo/Allmusic album reviews and artist bios are amazing, the similar artists/albums from them is usually very good too, and having it all augmented by last.fm for fallback data and popularity, MusicBrainz for metadata, SuperSonic for ML analysis (particularly for obscure artists without AllMusic/Tivo data), and all the other good stuff has resulted in (personally) much broader metadata coverage.
The other benefit (personally) of MusicBrainz is that I can clean up poor data and add missing artists/albums myself, rather than having to wait for a provider to get to it for me (which for obscure stuff will never happen).
My library isn’t based on mostly similar music. It’s all different kinds and there is a good amount to build a great pattern. All I would like is for Plex to explain sonic analysis better than they have now.
While I understand for obscure stuff, if Plex is going all in with this, use more tools like Acousticbrainz and build from this. Plex is a great position to have its users contribute metadata about recordings to help bolster Acousticbrainz so it’s better and universal to use like MusicBrainz. Just food for thought.
