Play external audio files in sync with video

General Discussions
61

I know that post was in “jest”, to be sarcastic but it is true it would take an extraordinary effort to do and would require certain existing functionality in Plex to be “violated”. As an example you would never be able to direct play these files so they would have to go through the transcoder in order to be remuxed properly. What does that do to all clients and server functions currently in place that know and understand what it means to play the “original” file?

Every client would basically need a way to adjust the delay of the sound which brings up it’s own can or worms since this involves a lot of new programming plus requires that anyone using the client learn a new function. I don’t know about you guys but many of my family members are just above the pay scale of playing video via Plex and struggle with changing the volume. Adding a subtitle to the video using the menu is beyond their ability. I just couldn’t imaging trying to show them or trying to give them directions to sync audio.

All of this to me (my personal opinion) is trying to come up with a solution to a problem that doesn’t exist. In a nutshell we’re asking for Plex to now take multiple parts of a file and combine them back together in some magic way and to handle the proper delays needed in some magic fashion. Not only that be we are asking them to do this every time the file is played and to do it on every client available.

Why not just fix the source file one time and be done with it? The admin can control the whole process and no the job is done correct and then none of this functionality is needed?

1 Like
62

@cayars said:
I know that post was in “jest”, to be sarcastic…

Sorta… sometimes truth is stranger than sarcasm.

1 Like
63

I still think your assessment of the problem is off. I think you’re just dismissing the idea out of hand and not putting any critical thought into understanding the suggestion, and as a result you are arguing against a strawman.

@cayars said:
As an example you would never be able to direct play these files so they would have to go through the transcoder in order to be remuxed properly. What does that do to all clients and server functions currently in place that know and understand what it means to play the “original” file?
…
Not only that be we are asking them to do this every time the file is played and to do it on every client available.

Yes and no. In my example usage scenario, I have a BDrip with h.264 video and AC3/5.1 audio. These are muxed in one file that can be ‘directly played’. Browsers / web clients do not support AC3/5.1 audio, though. To prevent a transcode, I’d normally create a “plex verion”, but that would be incredibly wasteful. The source video is already compatible and doesn’t need a “plex version”, yet I’m wasting time and SIGNIFICANT disk space to duplicate that video. I would like to just take the AC3/5.1 audio track and create a “plex version”, externally, that is compatible with whatever browsers support today.

As I understand it, plex will prioritize whatever “plex versions” with codecs which can be direct played- the remux wouldn’t need happen for clients which support the source file.

Why not just fix the source file one time and be done with it?

There is nothing wrong with the source file. Why modify my archive to be compatible with contemporary limitations of today? Browsers could gain support for AC3/5.1 in a few years. The only thing that needs “fixing” here is plex, or web client codec support.

Every client would basically need a way to adjust the delay of the sound…

I agree, that would be absolutely insane to expect of users, and although it would be nice to have that functionality in plexmediaplayer (where it already exists in mpv and just needs to be exposed), I also agree it wouldn’t be worth the time to implement that in every plex client.

But!

Maybe the technical details of this are above me, but I really do not understand why you think any solution to this problem would involve the user manually syncing audio. How can mpv, vlc, mpc-hc keep an external audio track in sync? How does youtube keep its very many DASH streams in sync? Heck, how does the plex transcoder keep transcoded audio + video streams in sync? Why is it you think it can’t be done? Why is a rather old feature of mpv/vlc/mpc-hc/youtube “magic” to you?

Before we can continue this discussion, we need to separate the terms “keeping in sync” and “delay/offset”. I think up until this point, you have been using them interchangeably.

A delay/offset would only be necessary if the audio file does not match the video. Again, this is not a plex problem, it is up to the user to hardcode the delay into the file themselves. This problem will not arise if the audio file is literally a transcoded version of the source file.

Keeping in sync is, as I understand it, the player adjusting timings on the fly to correct de-synchronizations introduced by the computer itself. This is… a basic component of a video player and nothing is “magic” about it.

BTW, does anyone know of any streaming software similar to Plex that does anything like this?

YouTube’s DASH streaming system.

By all means DO put in a formal request for this feature it in the proper section of the website (request forum) and see if you can get support for it from the users. That’s the proper way to request features and to see how important features are to users so Plex knows where to spend it resources on new features.

Sorry, I thought that’s what this was (I didn’t start this topic). Also, and this is a genuine comment, I’m not being snide: I’m just used to participating in projects where features are implemented because they make sense, not by popular vote :slight_smile:

3 Likes
64

@LinkSatonaka said:
I’m just used to participating in projects where features are implemented because they make sense, not by popular vote :slight_smile:

You’re new here.

Common Horse Sense, Logic OR popular vote play no part in the development process. Plex develops as they see fit regardless of user wishes or collective riot. Employees would have you believe that mysterious ‘voting system’ actually means something, but it does not.

If Plex takes a liking to a ‘feature’ they haven’t already got on the white board they ‘may’ work it in to the grand plan, but no amount of ‘votes’ are going to make one bit of difference in this process. If they did, there are feature requests with hundreds of votes that are languishing right now drawing flies serenaded by crickets (or so I’m told) that would be getting some attention, but are not. I can’t officially make, view or vote on a feature request, but I’m told PPers are ignored with the same steadfast commitment ‘regular’ users are.

What’s more - if Otto appears with a user ‘work-around’ - you’d better embrace it, learn it, do it, 'cause if Otto’s doing it it’s very likely that’s the only way it’s ever going to get done.

Case in point - The Plex Dance®. There’s a user work-around to a bug so old and so prolific, it’s one of the few user work-arounds with a registered trademark.

You never know though. Make that ‘Feature Request’ or vote for one in progress. Nobody knows what Plex is going to do. They might like it. They spend most of their time developing things I don’t need - or want, for that matter - so run it up the flag pole and see if Plex salutes.

2 Likes
65

@LinkSatonaka said:
I still think your assessment of the problem is off. I think you’re just dismissing the idea out of hand and not putting any critical thought into understanding the suggestion, and as a result you are arguing against a strawman.

I really don’t think so, but I’m certainly open to suggestions.

@cayars said:
As an example you would never be able to direct play these files so they would have to go through the transcoder in order to be remuxed properly. What does that do to all clients and server functions currently in place that know and understand what it means to play the “original” file?
…
Not only that be we are asking them to do this every time the file is played and to do it on every client available.

Yes and no. In my example usage scenario, I have a BDrip with h.264 video and AC3/5.1 audio. These are muxed in one file that can be ‘directly played’. Browsers / web clients do not support AC3/5.1 audio, though. To prevent a transcode, I’d normally create a “plex verion”, but that would be incredibly wasteful. The source video is already compatible and doesn’t need a “plex version”, yet I’m wasting time and SIGNIFICANT disk space to duplicate that video. I would like to just take the AC3/5.1 audio track and create a “plex version”, externally, that is compatible with whatever browsers support today.

This is a completely different objective than what I thought this thread was about. Plex itself can transcode only the audio portion of the file while not touching the video which is very low in the CPU department. Alternately check out the thread in my sig for the file optimization scripts. They will do exactly as you just described hands off. It will take your file with the AC3/5.1 audio and create a stereo 2 track audio channel and make it the default first channel while coping in as is the AC3/5.1 audio track. You end up with a file that will play perfectly on any client in direct play and you will have a choice of 2 channel or 5.1 surround sound.

So 1 file is all that’s needed and the size increased only be the addition of the added 2 channel audio track.

What you just described is not a current problem with Plex and I don’t think is what this thread was asking for…

As I understand it, plex will prioritize whatever “plex versions” with codecs which can be direct played- the remux wouldn’t need happen for clients which support the source file.

Why not just fix the source file one time and be done with it?

There is nothing wrong with the source file. Why modify my archive to be compatible with contemporary limitations of today? Browsers could gain support for AC3/5.1 in a few years. The only thing that needs “fixing” here is plex, or web client codec support.

Well in the case you just gave there is a problem with the source. You have a file that doesn’t have an appropriate audio track in it for the player or hardware playing it back. Only two ways to fix it and that is to transcode the audio in realtime every time the file is played or fix the problem before adding it to Plex in the first place. Both situations are already covered.

Every client would basically need a way to adjust the delay of the sound…

I agree, that would be absolutely insane to expect of users, and although it would be nice to have that functionality in plexmediaplayer (where it already exists in mpv and just needs to be exposed), I also agree it wouldn’t be worth the time to implement that in every plex client.

But!

Maybe the technical details of this are above me, but I really do not understand why you think any solution to this problem would involve the user manually syncing audio. How can mpv, vlc, mpc-hc keep an external audio track in sync? How does youtube keep its very many DASH streams in sync? Heck, how does the plex transcoder keep transcoded audio + video streams in sync? Why is it you think it can’t be done? Why is a rather old feature of mpv/vlc/mpc-hc/youtube “magic” to you?

Before we can continue this discussion, we need to separate the terms “keeping in sync” and “delay/offset”. I think up until this point, you have been using them interchangeably.

A delay/offset would only be necessary if the audio file does not match the video. Again, this is not a plex problem, it is up to the user to hardcode the delay into the file themselves. This problem will not arise if the audio file is literally a transcoded version of the source file.

I used the terms interchangeably. As in, you have to keep the audio in sync with video by adding a delay so that for example you hear the words when the lips move.

Keeping in sync is, as I understand it, the player adjusting timings on the fly to correct de-synchronizations introduced by the computer itself. This is… a basic component of a video player and nothing is “magic” about it.

BTW, does anyone know of any streaming software similar to Plex that does anything like this?

YouTube’s DASH streaming system.

Not even close. DASH and HLS are just segmented video or chunked video. You still have to start with the proper media in a format where the audio and video are in sync with each other.

By all means DO put in a formal request for this feature it in the proper section of the website (request forum) and see if you can get support for it from the users. That’s the proper way to request features and to see how important features are to users so Plex knows where to spend it resources on new features.

Sorry, I thought that’s what this was (I didn’t start this topic). Also, and this is a genuine comment, I’m not being snide: I’m just used to participating in projects where features are implemented because they make sense, not by popular vote :slight_smile:

Just to add a bit more information and to explain why something like this is so hard to do. The hardware device, web browser etc would need to know how to side or to combine an audio and a video track. At that point you don’t have an MKV, MP4, HLS, DASH or any other established protocol. This type of thing really just doesn’t exist.

Now if we want to ignore streaming video/audio and just concentrate on local lan or file system based files then sure this type of thing could be done and has/is done. But this is apples and oranges.

So before Plex could do anything like this in Direct Play mode the devices & browsers would have to support this via some kind of industrial spec (which currently doesn’t exist).

66

@cayars said:
To the best of my knowledge the video and audio codecs (tracks) all need to be in the master media file. Which brings us back to the need to remux the file outside of Plex to achieve this. That’s not hard but we need to know what the “delay” is of the additional audio track so it can be synced correctly.

Figure out a reliable way to do that and we have something to work with.

I know how to add it to the master file, the problem is I am afraid it will take forever for me to do that. My computer hasn’t been upgraded in a long while and doing such things seems to take longer than it used to but not only that… I have over 1,500 files. x.x The source I used to get my first dub file didn’t seem to need syncing not sure if it’s just pure luck or similarities with the original file. o.o I honestly don’t really watch that many things that would need a dub but some family members prefer to watch movies or programing in our native language rather than in english or the program’s native language where I usually don’t have said issues or need. .-. so I was trying to see if there was a way for me to add a secondary audio track externally so that they could enjoy movies or programming better without me having to ‘remux’ every single file that doesn’t already have dual audio which not very many do anyway. I hope this is made a thing going forward.

67

Try downloading the scripts linked in the thread in my sig. Get it working and I bet you never look back. :slight_smile:

They can do all kinds of things to make the source file the best it can be for direct play purposes. From taking MPEG2, AVI or HEVC codecs and converting them to h.264 to taking audio track in fringe formats and converting them to a more common format that will be direct playable. It can take multi-channel audio and correctly convert it to 2 channel without throwing out channels (as some programs do) but by combining them so you don’t loose parts of the sound. It can even normalize you audio for you so one file isn’t loud and another soft. Same with in file audio. You won’t need to turn up the sound in quiet parts of the movie to have to turn down the sound during high action/explosion scenes. It can do all kinds of things in an automated fashion based on the source file itself. It’s all control by ini settings.

If you have a file that already has h.264 video in in and AC3 audio it will take about 2 minutes to convert give or take and is basically about as fast as reading the file and writing it. On the other hand if it needs to convert the video then it can take a while. But in any case the result will be much better than what Plex can do simply because offline processing isn’t time dependent where real-time processing has to be done in X amount of time while it’s streaming.

BTW, it can also strip things from the file as well. So for example if you rip your BluRay disk and it has 9 different languages in it for audio but you and your family only need English or Spanish then it can keep only the two languages you specify in it’s settings. It will discard the other langues audio tracks. Same for subtitles. So it can also help you to end up with smaller files that don’t need quite as much bitrate to stream as well.

68

@cayars said:
So 1 file is all that’s needed and the size increased only be the addition of the added 2 channel audio track.
…
You have a file that doesn’t have an appropriate audio track in it for the player or hardware playing it back. Only two ways to fix it and that is to transcode the audio in realtime every time the file is played or fix the problem before adding it to Plex in the first place. Both situations are already covered.

“The only two ways” are not satisfactory, hence, the feature request :wink:

I still think there is a misunderstanding of intentions. I know how to use ffmpeg and mkvmerge. I do not want to. This feature request is not borne of me not knowing how to do it, this feature request is about making plex more useful. You do not need to provide me with alternative ways to acheive what you think I’m after.

I’m storing my media in its native format. I will not be altering it- particularly not for contemporary, transient, temporary limitations. I do not want to pollute my media archive with modifications I need today but not tomorrow. To me, this is the role plex serves- to act as the middleman between my archive, and the limitations/features of the hardware I have today. It’s not fitting of a middleman to request that I make changes on the originating side.

The plex transcoder does a great job at being a middleman. Plex version are also a great idea. My suggestion is simply to extend the concept of a “plex version” to be more granular. Per-track.

I’m aware audio transcodes are “not that resource intense”. That’s not the point. The point is that we can make it more efficient through a logical extension of an existing feature.

I used the terms interchangeably. As in, you have to keep the audio in sync with video by adding a delay so that for example you hear the words when the lips move.
…

YouTube’s DASH streaming system.

Not even close. DASH and HLS are just segmented video or chunked video. You still have to start with the proper media in a format where the audio and video are in sync with each other.

DASH presents multiple SEPARATE video and audio streams that are played by the client. The video and audio do not come together. And it’s not just youtube’s client that can keep these in sync, but mpv and ffmpeg also have no problem doing it. But, I’ve identified a misunderstanding: I am not suggesting that we somehow shoehorn Direct Play into this…

Just to add a bit more information and to explain why something like this is so hard to do. The hardware device, web browser etc would need to know how to side or to combine an audio and a video track. At that point you don’t have an MKV, MP4, HLS, DASH or any other established protocol. This type of thing really just doesn’t exist.
…
So before Plex could do anything like this in Direct Play mode the devices & browsers would have to support this via some kind of industrial spec (which currently doesn’t exist).

And this is why I think you’re strawmanning: there is a simple solution here, but you are so focused on telling me why your understanding of my suggestion can’t be done- you aren’t even trying to meet me halfway! Something like- “It can’t be done the way you describe, but it can be done via [xyz]”

The server remuxes the video and the external audio track for the client. It was always my intention that the plex media server would be remuxing the streams together for the client.

69

LinkSatonaka,

Trust me I understand DASH and what it can and can’t do. I also understand that for something like this to work it has to be supported by all the different Clients. How do you handle this on clients that don’t support DASH or only support HLS? How do you support this in web browsers who don’t support DASH natively? How do you support this when not all DASH players fully support multiple language tracks or the role feature?

Let’s take a step back for a second and think this through. In order to come up with the additional track you want to be external you almost surely have to pull an existing track out of your media file and convert it. If you don’t do this you would have the pain staking issues we talked about earlier with syncing issues.

So knowing that the source track has to come from the original in the first place and knowing that there must be some type of transcoding going on during the playback of the media to create the dash stream what do you gain by trying to make this a manual process. The manual process I speak of is that you have some manual step you need to do to extract a track and convert it to the external file.

Since this will NEVER be direct playable (no streaming players can handle this) the Plex transcoder then has to reassemble the file back to an MP4, MKV or to HLS or DASH.

Why do any of this? Why not let Plex ask the client what audio codecs it supports and if your untouched original files doesn’t have an appropriate audio track then let Plex create it on the fly? It’s going to be doing 90% of the same work anyway. Doing an audio conversion vs remuxing in a 2nd file is almost no CPU difference.

So again, I’m asking what is the purpose of doing this? What do you possibly gain from doing it vs pre processing your files automatically and added the track so it can direct play OR letting Plex handle this all on it’s own?

That is the heart of the question. Why would Plex want to to try and re-invent a new way to handle something it already does wonderfully for all clients?

I totally understand you personally don’t want to touch your original files but are willing to put time in to do all the other work to make the external audio track, but what is gained? You say you understand ffmpeg and mkvmerge and I think implied you don’t want to use them. What/how do you plan to create these external audio tracks even if you can find a purpose for them?

You say you’re trying to come up with another way to make it more efficient but you’re not. You’re way would cause the need for the transcoder to be involved in every playback and mine would almost eliminate the need.

70

@cayars said:
Why do any of this? Why not let Plex ask the client what audio codecs it supports and if your untouched original files doesn’t have an appropriate audio track then let Plex create it on the fly? It’s going to be doing 90% of the same work anyway. Doing an audio conversion vs remuxing in a 2nd file is almost no CPU difference.

The hunk of junk I use for my plex server begs to differ. Remuxing takes no effort, but transcoding the audio AND remuxing on top of that is often too much. As I understand it, it’s not “audio conversion vs remux”, it’s “audio conversion AND remux vs remux”.

I totally understand you personally don’t want to touch your original files but are willing to put time in to do all the other work to make the external audio track, but what is gained? You say you understand ffmpeg and mkvmerge and I think implied you don’t want to use them. What/how do you plan to create these external audio tracks even if you can find a purpose for them?

Sorry I wasn’t clear. I’d use those tools to create my external tracks, yes. I only intended to say that I do know how to use those tools, but I want to keep “plex versions/tracks” separated from the archive.

Although if this feature were imlemented, it would also make sense for plex itself to have the functionality to create a plex version of tracks.

You say you’re trying to come up with another way to make it more efficient but you’re not. You’re way would cause the need for the transcoder to be involved in every playback and mine would almost eliminate the need.

No. In contexts where the original/internal AC3/5.1 (or whatever it happens to be) track is supported, direct play would be available and preferrable.

In contexts where the internal audio track isn’t supported, then only a remux would happen, as opposed to the transcode+remux.

Remuxing a new audio track into the original file would work better for plex’s feature set as it is now, sure. But I’d sooner upgrade my plex-server box to something more capable than tamper with my archive.

71

@cayars said:
BTW, does anyone know of any streaming software similar to Plex that does anything like this?

VLC does that perfectly. Well, maybe it’s not very similar to Plex, but it can stream

72

LinkSatonaka It really sounds like you need to upgrade your “hunk of junk” more than anything else. :slight_smile:

If you did this your problems would go away for the most part. With any decent CPU running on a PC you should be able to have several audio transcodes at the same time since they don’t impact the CPU much at all. With any modern CPU you should be able to have anywhere from a couple to several full (audio and video) transcodes going at the same time.

What you asked for isn’t impossible to do but would require changes across the board. From the scanner, to the database, to the MDE (media decision engine) to the transcoder to the clients and many parts in between.

At the end of the day someone at Plex would need to be able to justify the request and the man hours needed for all the programming and testing. They would need to justify the new feature and answer the simple question what does it add to the Plex eco system that it doesn’t already do.

Do yourself a favor and make a nice little write up for an OP post and go post it in the Features Request section of the forum https://forums.plex.tv/categories/feature-requests . Users who agree with you and your request will “like” your post which drives up the count. Plex uses this count to see what feature requests are popular and worthy of including in Plex.

So don’t let me or anyone else talk you out of a feature request just because we don’t agree with the idea!

73

@cayars said:
So don’t let me or anyone else talk you out of a feature request just because we don’t agree with the idea!

I’ll second that one.

74

I’d like to list a few use cases that may demonstrate why external audio tracks are useful and why remuxing might not always be practical.

  • Consider a rip of BD that has no commentary track. I also have a DVD of the same movie with a commentary track. I don’t wan to keep two rips. I’d rather have BD rip with external commentary track from DVD. I don’t want to remux my BD rip because there won’t be a way to validate it against the disc.

  • I’m sure many people watch shows in languages that are not native to them. For many foreign shows enthusiast groups release a translated voiceover track. The main file (video) and an external audio track come from different sources and at different times. It’s not that hard to automate remuxing but it doesn’t seem practical to do it for every single piece of periodic media one consumes.

    This goes for the bigger part of the world watching Australia/Canada/NZ/UK/USA shows, pretty much everyone outside of Japan watching anime and doramas, everyone outside of Korea watching their shows, etc. For any given language there are more people not speaking it than native speakers. Demand is enormous in this department.

1 Like
75

Looking to see if Plex had added this 8 years later, glad to see discussion is going well. To answer the question of

So again, I’m asking what is the purpose of doing this? What do you possibly gain from doing it vs pre processing your files automatically and added the track so it can direct play OR letting Plex handle this all on it’s own?

The initial ask around rifftrax, directors commentary, and external audio that does not require frame-accurate sync is what I would hope to support. I have visions of being able to support a community of fan-created commentary tracks, were the multi-platform player support to exist.

It sounds like that would easily be conflated with external language files that would require frame accurate sync and be untenable, so 8 years later, I guess: never mind.

76

So we still dont have a solution for this problem? Geez, plex is really catching on slow.

77

+1 on requesting this feature in Plex. Atleast be able to play external files that are already in sync, adding resyncing options would just be a bonus!

Remuxing the track into the vide file isn’t an elegant solution for many situations.

For my example, I have a number of movies which I’d like to keep in their original form, yet add an alternative language track to for my kids. I’d like to simply have the audio track as a seperate file in the same folder and play it, like I can in most any other media player out there. I was shocked when I realized Plex doesn’t support this.

78

+1 in 2018

I especially want the feature in Plex Cloud because the remuxing solution is a huge pain there as opposed to just uploading an additional audio track.

79

Just want to link to the feature request and encourage people to click “Like” to vote for it:
forums.plex.tv/discussion/46122/alternate-external-audio-tracks

80

Voted + commented