Shortcut: The core feature suggestion is in paragraph #3s for Audio and Subtitles, respectively.
Audio/Spoken Language
Detection of Audio Language
Options are:
from file name: IMHO, this is not sensible, as video files can contain multiple audio language tracks. There might be use cases, where versions and editions could be used. I wanted to mention this to pre-empt suggestions to do this via naming conventions.
It could work for audio books, but at the moment, there is only limited support for these.
from file tags and meta data: This is already evaluated by Plex. Unfortunately, in the underlying data there are
numerous flavors of unknown or even
incorrect tagging where the tool at the time (TATT™) defaulted to the system language, regardless of the actual audio language.
There is also the potential confusion between the language tag of the audio streams and that of the video stream that needs to be accounted for.
This brings me to the actual request:
by analysis: This is what I’d like Plex to do, either (a) generally for unknown audio languages, (b) on demand for cases where I disagree with the audio language derived from the tags/meta data, or (c) generally for all files, ignoring tags/meta data.
People already created tools/libraries to identify a language from spoken text. Here are some I found:
The challenges are likely around the size of these models and if they could be run on the local server (instead of an web API). A potential solution could be fingerprinting which has already been discussed here for the purpose of identifying titles. Fingerprinting audio locally is relatively light weight. Matching a fingerprint to a language would require a suitable database, though (perhaps community generated?)
Update of Audio Language Information
Once the spoken language has been identified, how does Plex use it?
updating file tags is hopefully something we will never see with Plex. This should be reserved to external tools operating directly on the user’s files. That said, if someone knows a tool that digs through all audio streams in a library, identifies the respective spoken language and offers to bulk update the language tags in the video files - please drop a comment. The emphasis here is on bulk update - doing it for individual files is not too complicated but not practical.
updating of Plex Metadata/Library: this is really what should happen, if spoken language detection is implemented within Plex. I would like to see functions like analysis for individual overwrites and bulk updates with preview of before/detected/(editable) after.
Subtitle Language
To do the same recognition for subtitles (i.e. written text) is conceptually very similar, but from an implementation perspective a different (presumably easier) challenge. Once support for ebooks is added these would equally benefit from language recognition.
Detection of Subtitle Language
Detection from file name works slightly better than for audio languages when the subtitles are in separate files. Although, it faces the same challenges for subtitles embedded in files. That said, if someone knows a tool/script that digs through *.srt, *.sub, *.ssa, *.smi, *.vtt, etc. files, identifies the file’s language and offers bulk updates of file names according to Plex naming conventions - please drop a comment.
Detection from file tags and meta data: As for audio, Plex already evaluates this and faces the same issues of missing, unknown or incorrectly set tags.
Detection of subtile language by analysis is then the actual feature request here. The technical overhead is much simpler than language detection from spoken language. There are readily available tools like these, e.g.
- LangDetect
- Spacy-langdetect
- fastText
- gcld3
Update of Audio Language
updating tags in files or file names, is something Plex should leave to the user or external tools. So please do not.
updating of Plex Metadata/Library would be the preferred way to retain the identified subtitle languages. As before, I would like to see functions like individual overwrites, bulk updates with preview of before/detected/(editable) after.
Conclusion
More accurate and complete language recognition allows better filtering and compliance with user’s language preferences. There are plenty requests already that would benefit from this. E.g.
This request is not about how the language data is used. It is about getting the underlying data in ship shape.
N.B. The only other language detection feature requested by @Astrid-Desmet I found is 3 years old and mostly neglected. I hope to change that with a more general request, some more context and offering pointers to potential solutions.
Wow @tomenendal color me impressed! This is a nice post. While there is a lot here, we are wanting to do a deep dive into subtitles next year and while I won’t commit to all of the ideas here, I hope that some of that work would at least cover some of these ideas. The main reason I wanted to jump in here is to give a big thank you for a well reasoned and written post!
regarding sub-titles… especially forced ones. In sub-title tracks with “forced translations” of signs, there may or may not be few lines of text - maybe too little for analysis.
music scores in movies (can be somewhere in the first 10 minutes of the movie): if there is singing involved, then this could lead to false analysis results
musicals: you often have singing in these movies and this could very well be in the original language while the rest is synchronized to a foreign language - also a potential trap for analysis
for dubbed movies: these may or may not have complete passages of “original language” if containing scenes that had not been synchro’ed but cut out instead. There may or may not b sub-title tracks pointing towards such scenes.
For me, it would be great to be able to edit track info from within Plex - and have a button to check an audio track when doing so (play & fast-forward/-backward audio in the Edit dialog pop-up window or review/scroll through sub-title track texts from the edit window.
This would be great in order to not having to leave that edit window and be sure to edit the correct track (I remember movies having six different tracks and it is hard to edit them if not being completely sure which “English” audio track you want to re-label.
I concur with the concerns you mentioned and think they’d require dedicated measures to handle them correctly. Some additional comments:
For dubbed movies, your concern is the behavior I’d be expecting: That it shows the detected language. So when you have multiple audio tracks and/or subtitles, plex can show the actual language as detected (instead of some variant of unknown).
As for editing the audio track/subtitle meta data, I am a strong believer of the hands-off principle that Plex seems to adhere to. I.e. plex does not edit or modify any of the files in the library. Instead, Plex maintains a separate meta-data layer in a data base. Most of this is auto populated by agents and this is where I’d see the detected languages as well. And this is where I’d suggest the editing/overwriting you mention should go into.
Some of the concerns you mention are also hinting at potential enhancements on the detection. In addition to detecting the language, also the type of subtitle might be detected and logged in the meta data. Here is a breakdown that Gemini LLM came up with to classify subtitles:
By Display Option:
Closed Captions (CC): These are the most common subtitles you’ll see on DVDs and streaming services. They appear as text at the bottom of the screen and can be turned on or off by the viewer.
Open Captions (OC): In contrast to closed captions, these subtitles are permanently burned into the video file and can’t be turned off. They’re less common but you might find them on some older videos or videos shared online. [I left this in for completeness - would be hard to detect and would be a meta data annotation of the video stream]
By Function:
Translated Subtitles: These are the standard subtitles used to translate dialogue from a foreign language movie into the viewer’s language.
Forced Subtitles: A type of closed caption, these subtitles appear only when there’s foreign language dialogue, sound effects, or on-screen text important to the plot. They are there to ensure viewers don’t miss any crucial information.
Descriptive Subtitles (SDH): Also known as subtitles for the deaf and hard of hearing (SDH), these go beyond just dialogue and describe sounds, music, and other auditory information for viewers who are deaf or hard of hearing.
For Audio Tracks, I’d add Commentary to the list, where people (typically the director) talk over the actual movie.
And perhaps there are more. But auto-detecting these is likely a harder challenge than detecting the language and likely more error prone. But still preferable to unknown which is the current state
Your mention of dubbed movies, also brings up the concept of the original language. At least MKV has a flag for audio and subtitle streams to indicate which of them are the original language. This would be even harder to detect. However, this could be something that could be looked up in online databases. E.g. IMDB typically shows this information.
I will keep watching this thread to nudge it along. With only three upvotes at the time I write this, I keep my expectations in check, though.
I love the idea, but my votes are all blocked by other suggestions.
Apart from being able to delete media files (if option is enabled), that is.
Well, there are plenty of data fields that are editable. I agree that ATM the language of (audio and video) tracks is not.
Plex always tells people to go and edit the files themselves instead.
And usually, for only that many files, this is the preferred way to do it.
What I wanted to say is that a complex analysis method is quite hard to implement and that - as a start - the data fields could become editable. If Plex gives false results for analysis, you’d be happy to be able to correct this.
I usually modify the name of subtitle files when saving them, adding a language suffix. However, there are situations where it’s not convenient to add them during storage, and it’s cumbersome to modify the filenames later.
As for the language of audio files, it can’t be modified externally, and Plex doesn’t seem to have a way to intervene internally either. Currently, I have many files that don’t have language tags or have incorrect language tags embedded.
It would be great if Plex could automatically recognize the language of audio and subtitle tracks. Additionally, having the ability to manually correct language tags within Plex would also be helpful.
renaming the subtitle files does work. It relies on them being separate files, though. E.g. in Matroska Video Containers (MKV) you can have an arbitrary number of subtitles embedded. The same goes for audio tracks, where it is even more common to have them embedded in the video file.
The other drawback (over automatic language recognition) is that you have to look at and touch each file manually. As a movie library grows, this eventually becomes tedious; same goes for TV shows where some have easily more than 100 episodes.
There is a tool I found very helpful to look at subtitles and audio tracks: MKVToolNix GUI. You can drop in many kinds of video, audio and subtitle files and it shows you the meta data that comes with the file. Even when you are not using it to remux the parts into an MKV file, it tells you, what is already tagged within the files. (unfortunately, most files I come across are missing any tags). If you do use it to rebuild your video files as MKV, it does so without any quality degradation as it keeps the original streams. And you can also tag language information onto audio tracks and subtitles manually. Plex does recognize and show these tags.
This is what I am currently doing. It is manually and tedious - hence the feature request
please drop a comment. The emphasis here is on bulk update - doing it for individual files is not too complicated but not practical.
and this could very well be in the original language while the rest is synchronized to a foreign language - also a potential trap for analysis
