Cayars - Setup walk through and some tips and tricks

General Discussions
401

@kieranc Mike already answer the majority of your questions. The only thing I’ll add is this:
MP4 is a good streaming container (well it works).
MKV is a great storage container but a lousy streaming container.

We care about streaming with Plex so that makes MP4 better to us.

You are in full control of what subtitles are kept or thrown out. By default it will rip ENGLISH subtitles (ini setting) out of the MKV and create a seperate .SRT file. So they aren’t lost, just transformed into something more usable.

At present it does not use Quick Sync and I’m not sure it should. Sure it’s faster but still doesn’t match the quality of pure software encodes (very close on the latest GPUs). I personally have a different philosophy when it comes to QS. I WANT IT BAD for real-time encoding but for encodes that are meant to be permanent I want the absolute best I can get for the time it takes to do. With that said I’ll probably add this soon. :slight_smile:

@don.alcombright
Yes you DO WANT relocate_moov = False
I create the moov atom a different way than the original script did. It’s faster and has less issues with compatibility.

Carlo

402

@cayars said:
@kieranc Mike already answer the majority of your questions. The only thing I’ll add is this:
MP4 is a good streaming container (well it works).
MKV is a great storage container but a lousy streaming container.

We care about streaming with Plex so that makes MP4 better to us.

Understood. So far streaming is not my priority, though given Plex’s ability to do it (I’ve never had this ability before with Kodi) I may start doing it more. This would only be when traveling though; I don’t expect to share my collection with dozens of friends like you guys.

I don’t understand why one container is better than another for streaming (or vice-versa for local playback). Why is mkv lousy for streaming? I would think if the target device/client can play with mkv (which Plex can), and the bitrate of the encode (unrelated to container) is well below the available network bandwidth limits, and the encode/bitrate is within the decoding capabilities of the target/client’s cpu, then mkv would be fine…

Is it because people tend to package too much “stuff” in mkv’s (e.g. multiple languages, track/chapter stops, multiple subtitles, etc.?)

Lots of questions… Maybe there’s a “streaming faq” somewhere I can read? :slight_smile:

Thanks.

403

If you use Plex you ARE streaming. You are streaming to clients in your house. Now granted because the clients are in your house running over (assumed) fast connections you won’t notice some of the downsides of MKV as much as you would streaming over the Internet.

I wouldn’t say MKV is lousy for streaming but it was designed as a versatile storage container. I’ll try and explain why MP4 is better but you need a bit of technical knowledge first so here goes:

In an MPEG-4–compliant container, every movie contains a moov atom. Normally, a movie atom contains a movie header atom (an mvhd atom) that defines the timescale and duration information for the entire movie, as well as its display characteristics. The movie atom also contains one track atom (a trak atom) for each track in the movie. Each track atom contains one or more media atoms (an mdia atom) along with other atoms that define other track and movie characteristics.

In this tree-like hierarchy, the moov atom acts an index of the video data. It is here that the MPEG-4 muxer stores information about the file to enable the viewer to play and scrub the file. The file will not start to play until the player can access this index.

Unless specified otherwise, the moov atom is normally stored at the end of the file in on-demand content, after all of the information describing the file has been generated. Depending on the type of on demand delivery method selected—progressive download, streaming, or local playback—the location will need to move either to the end or to the beginning of the file.

OK so now you know a bit about the atoms in a file. Typically in MKV files these atoms follow all other track info and you really don’t have a “goto” spot in the files you can go to quickly read this info.

On the other hand in an MP4 file you can have the moov atom at the end or the BEGINNING of the file. Running the scripts I provide the moov atom will always be at the start of the file. So essentially this enables the file to start to play much faster as you eliminate the “lookup” routines that have to be done with an MKV file. In a local network this isn’t as noticeable but it becomes more and more noticeable as the connection speed gets slower.

So an MP4 is a bit better suited to streaming than an MKV file. MKV format is kind of an anything goes storage container which is both good and bad. You can stick all kinds of things in an MKV but the downside is that these “extra” items can cause fits for some client devices. Also why many devices don’t support this format. MP4 on the other hand is a standard with specific features that are allowed to be in the file. This makes it easier for people to build MP4 compliant devices and test them properly.

So that’s a high level view of why MKV may be a superior container but why MP4 is a better container for streaming.

Carlo

404

Learning something new everyday…

Now… if I can just recall any of this when needed… that’s another thing. :wink:

405
> If you use Plex you ARE streaming. You are streaming to clients in your house. Now granted because the clients are in your house running over (assumed) fast connections you won't notice some of the downsides of MKV as much as you would streaming over the Internet
Would you then consider using Kodi to play content on the same local NAS to be streaming as well? Where do you draw the line of streaming vs. not streaming? Just curious. In my mind, it's more of a bandwidth thing; not that I'm correct, it's just how I thought of it. I suppose if that were true the arrival of 100+mbps internet service would mean the end of "streaming"...
> > I wouldn't say MKV is lousy for streaming
LOL ... except that you did! :)
but it was designed as a versatile storage container. I'll try and explain why MP4 is better but you need a bit of technical knowledge first so here goes:
Snip...
> So that's a high level view of why MKV may be a superior container but why MP4 is a better container for streaming. >
Thanks for that excellent explanation. So if one doesn't mind the initial start-up delay, are there other aspects of one container vs the other that would affect playback quality/performance/etc.?
406

@kieranc said:
Would you then consider using Kodi to play content on the same local NAS to be streaming as well? Where do you draw the line of streaming vs. not streaming? Just curious. In my mind, it’s more of a bandwidth thing; not that I’m correct, it’s just how I thought of it. I suppose if that were true the arrival of 100+mbps internet service would mean the end of “streaming”… >

I might be foggy in my Kodi experience but Kodi are all local servers, no matter the device. So your Kodi Fire TV application for example is the Server directly linking to the files. Plex has a central server (your PC/Mac/NAS) that your clients (Fire TV, iOS, Android) connect to and “stream” from. 2 different points on this, but think of everything as streaming in the Plex world, so in other words, if all your data is stored (not files but traces of where the files are) on server A and you are using device B to see it its streaming.

  1. Bandwidth- locally you should never see an issue here. Streaming across the web you might. If you are only concerned with local, bandwidth should not be a concern but #2 could be…
  2. Transcoding- this can effect local and external streaming. Transcoding is CPU intensive. The less transcoding the better. For instance, my NAS can handle about 3 transcodes at once. Some PCs can handle 5/10/15 at once. Some PCs can’t handle any. I can Direct stream to about 10 people at once since direct stream is really more bandwidth restricted and not CPU restricted.

Kodi also has a local media player built in making it easier for them to handle more formats and containers. Plex (not worth the argument :slight_smile: ) relies on the local players (iOS, Fire etc) internal media player meaning it completely depends on what formats it will support. For instance RasPlex supports nearly everything, so it’s all Direct Play, but the Apple TV supports basically just MP4 and AAC.

407

@Plexhilarated

Don’t be modest. We all know you already knew this info. :slight_smile:

@kieranc
NO, not in the traditional standpoint.

Kodi doesn’t really have a client/server type environment. Kodi “clients” have access to the file system directly. So the Kodi client plays the file directly.

Where do you draw the line if it’s streaming or not?
Excellent question I must say. It’s not really about the “speed” of the pipe between the client and the media but more about the protocols it has to use to “read/understand” the media.

If you can replace the client (plex or kodi, whatever) with something like VLC/windows media player and you can access the media directly via a drive letter or network path then you are playing it back “locally”. The software has the ability to directly read the file and can directly navigate the media and pull back the info and atoms of the software. It doesn’t matter if the media is half way around the world and you used a VPN. If you access it as a “local drive” then you still have direct access to the media and all parts of it.

This is different when you are streaming. You can’t just jump to “block 50” or jump to 5GB into the file. You have to “navigate” the media using the protocols available to you. Webservers work a certain way and have certain protocols. You have to issue command to tell the server to “seek” to a certain portion of the file and send that to you. You might then need to read that and “jump” to another portion of the file which requires telling the server to seek to another portion of the file. Then you get the atoms you need to “understand” the media. Now you instruct the server to go back to the beginning and start playing the file. But you now have the structure and knowledge to be able to seek (FF/RW), etc.

So when using any client that has “direct access” to the file you can do all kinds of stuff and MKV is perfect for this. When you are dealing with true web servers and true streaming then it gets much more complicated and you have to play “nice” and request things in a given standard matter.

Part 2 of question:
If you can live with the start up delay and have a client that can playback AVI or MKV (examples) then you may not have any additional “penalties” for using those formats. Once you get past the “atom” issues it’s a lot easier to play the media back (locally at least).

If you want additional info just let me know.

Carlo

PS EDIT: @don.alcombright replied while I was typing but is 100% correct in what he said IMHO.

408

@cayars , but you were more elegant in your reply

Plex does confuse a bit when it’s terms are:

  1. Direct Play
  2. Direct Stream
  3. Transcode

All 3 are streaming, but Direct Play is just passing the file along the chain (that chain being the file location -> server (or your middle man) -> player). Transcoding is when it completely redoes the file on the fly to adhere to the players requirements (this is done by the middle man). Direct stream is a middle ground between the two IMHO.

409

@cayars said:
Don’t be modest. We all know you already knew this info. :slight_smile:
Carlo

I must be putting on a good front then… I was honored in fact, to be in the same sentence with you in the comment that @MikeG6.5 posted in your Wish List thread.

410

If you have two machines, one with the media on it and the other you want to play that media on, and you connect to the first using a network connection, serial or even some USB types of connections you are technically streaming that media from one device to the other. The first machine in this case acts as a server, serving the media to the second machine, which is then the client.

DLNA, or any derivative is also streaming. Kodi/XBMC, Emby and Plex are a more robust method of doing what I used as an example above. You are sending that data (the media) from one machine to another. Regardless of it being local or remotely… And generally those applications have other features that add to the streaming experience. Tracking Watched, presenting the media in a eye pleasing display, sharing to other users, etc. Any device running these apps and streaming media out is a server, and device watching the media from these devices is a client.

Local streaming is a bit easier to do in most cases, as you have higher bandwidth to use between the devices than you normally do with a remote connection. Remote streaming is limited by the speeds you have at both ends of the equation. Upload for the source of the media and download at the playing machine.

You are NOT streaming media if you have it on the same device that you are playing it on. You are just playing it then. Sure there may be some transcoding going on, converting the media from one codec or file container to another, but it’s not streaming from one device to another. It’s taking the media from disk and displaying it. I suppose it could be both a server and a client of itself, as Plex’s Web App could be seen to be a client of the PMS if you play media on it…

You can have client apps that can also be seen as servers, such as the Android app, where you can advertise on the network as a server and stream synced media from it to another Plex client app on the local network, but it won’t be able to transcode or Direct Stream for the client machine in any way, shape or form.

This is really an over-simplification of the whole process but it gets things in a nutshell. I’m sure there are others that will build on this, such as using a USB drive on a PC, etc. (Because, technically this could be seen as streaming, too, I suppose…)

So to answer this question: “Would you then consider using Kodi to play content on the same local NAS to be streaming as well” Yes, you are streaming media from the “server” (NAS) to the “client” (Another machine playing the stream).

411

@don.alcombright said:
@cayars , but you were more elegant in your reply

Plex does confuse a bit when it’s terms are:

  1. Direct Play
  2. Direct Stream
  3. Transcode

All 3 are streaming, but Direct Play is just passing the file along the chain (that chain being the file location → server (or your middle man) → player). Transcoding is when it completely redoes the file on the fly to adhere to the players requirements (this is done by the middle man). Direct stream is a middle ground between the two IMHO.

Agreed, and even in the “plex ranks” we can’t all agree on what the terms mean. In another thread just this past week we had a “heated” discussion as to what the terms mean. Here is the thread: https://forums.plex.tv/discussion/201568/roku-4-review-a-real-scam/p1

Interesting thread if you have 20 minutes of you life to kill.
I’ll just reference this wiki entry: https://support.plex.tv/hc/en-us/articles/201566396-How-are-Direct-Play-Direct-Stream-and-Transcoding-different-

I’ll hold to “defending” the terms as best I can to make sure people use the terms to describe the same things. :slight_smile:

If long term plex aficionados can’t agree on the terms then…

But yes, the waters are murky to say the least. The terms people throw out have to read into the conversation at hand and not taken at face value. You have to take what’s said in context.

@Plexhilarated said:

@cayars said:
Don’t be modest. We all know you already knew this info. :slight_smile:
Carlo

I must be putting on a good front then… I was honored in fact, to be in the same sentence with you in the comment that @MikeG6.5 posted in your Wish List thread.

Don’t get used to it. We throw you a bone every once in a while. :slight_smile:

Moo-Haa-Haa

412

@MikeG6.5 said:
If you have two machines, one with the media on it and the other you want to play that media on, and you connect to the first using a network connection, serial or even some USB types of connections you are technically streaming that media from one device to the other. The first machine in this case acts as a server, serving the media to the second machine, which is then the client.

I’d disagree with this statement. You are accessing the file via a local/network path and it’s not streamed. You are free to use operating system calls to access the file in a random fashion. You can freely jump to any part of the file by “just accessing it”.

DLNA, or any derivative is also streaming. Kodi/XBMC, Emby and Plex are a more robust method of doing what I used as an example above. You are sending that data (the media) from one machine to another. Regardless of it being local or remotely… And generally those applications have other features that add to the streaming experience. Tracking Watched, presenting the media in a eye pleasing display, sharing to other users, etc. Any device running these apps and streaming media out is a server, and device watching the media from these devices is a client.

Local streaming is a bit easier to do in most cases, as you have higher bandwidth to use between the devices than you normally do with a remote connection. Remote streaming is limited by the speeds you have at both ends of the equation. Upload for the source of the media and download at the playing machine.

You are NOT streaming media if you have it on the same device that you are playing it on. You are just playing it then. Sure there may be some transcoding going on, converting the media from one codec or file container to another, but it’s not streaming from one device to another. It’s taking the media from disk and displaying it. I suppose it could be both a server and a client of itself, as Plex’s Web App could be seen to be a client of the PMS if you play media on it…

You can have client apps that can also be seen as servers, such as the Android app, where you can advertise on the network as a server and stream synced media from it to another Plex client app on the local network, but it won’t be able to transcode or Direct Stream for the client machine in any way, shape or form.

This is really an over-simplification of the whole process but it gets things in a nutshell. I’m sure there are others that will build on this, such as using a USB drive on a PC, etc. (Because, technically this could be seen as streaming, too, I suppose…)

So to answer this question: “Would you then consider using Kodi to play content on the same local NAS to be streaming as well” Yes, you are streaming media from the “server” (NAS) to the “client” (Another machine playing the stream).

I’d agree with mostly everything else.

413

@cayars , what is your best recommendation for generating a list of file locations (for files needed to be be converted)?

My original thought was the get the list of file paths of these files and then run it via a batch command 1 at a time so I could do lets say 50 a day or so and actually monitor the outcome easily, aka if audio messes up or i lose something i didn’t mean too etc etc. But I can’t seem to find a way to get the full list of files exported to a list, or better yet a full list of files that should be converted to a list.

something like this so I can monitor it in the bat file (and comment out the ones I want to do that day:

c:\python27\python manual.py -a -i “\hades\movies\3 Days to Kill (2014)” >> D:\Convert\log.txt
c:\python27\python manual.py -a -i “\hades\movies\8 Mile (2002)” > D:\Convert\log.txt
c:\python27\python manual.py -a -i “\hades\movies\8MM (1999)” > D:\Convert\log.txt

414

Thanks to @MikeG6.5 I now have the scripts up and running.
Now to the main question (and don’t put me in the eternal flame for this question…), when I now have converted my Alien Directors Cut, 1080, DTS, Bluray mkv-file to mp4, have I lost anything in terms of quality?

From what I understand from the threads, MP4 is the way to go to reduce transcoding but I don’t want to convert the MKVs if I need to keep them as well.

415

A purist might be able to tell the differences, but I couldn’t when I tried playing them side by side on the same device… and I was testing it with a really high bitrate scene, as Carlo suggested to me when I started testing. (Transformers and , during the shift from a car to a bot, Saving Private Ryan when the tank explodes at the end, etc.) Of course I have a 40" and only about 8’ away… On a 65" UHD it might be different…

416

@MikeG6.5 said:
A purist might be able to tell the differences, but I couldn’t when I tried playing them side by side on the same device… and I was testing it with a really high bitrate scene, as Carlo suggested to me when I started testing. (Transformers and , during the shift from a car to a bot, Saving Private Ryan when the tank explodes at the end, etc.) Of course I have a 40" and only about 8’ away… On a 65" UHD it might be different…

Might be best to test both version on the 105" first then… :slight_smile:

417

@Old-T said:

@MikeG6.5 said:
A purist might be able to tell the differences, but I couldn’t when I tried playing them side by side on the same device… and I was testing it with a really high bitrate scene, as Carlo suggested to me when I started testing. (Transformers and , during the shift from a car to a bot, Saving Private Ryan when the tank explodes at the end, etc.) Of course I have a 40" and only about 8’ away… On a 65" UHD it might be different…

Might be best to test both version on the 105" first then… :slight_smile:

Yes, please test on your 105". I “only” have a 75" to test with.
Typically as screen size goes up your need for higher quality goes up with it. I can show you how to adjust the script up a notch or two IF NEEDED. Basically changing the Constant Rate Quality. This will cause your files to be a bit bigger and will take slightly longer to transcode. This change will only affect transcodes and not remuxes of course. So let me know.

Carlo

418

@don.alcombright said:
@cayars , what is your best recommendation for generating a list of file locations (for files needed to be be converted)?

My original thought was the get the list of file paths of these files and then run it via a batch command 1 at a time so I could do lets say 50 a day or so and actually monitor the outcome easily, aka if audio messes up or i lose something i didn’t mean too etc etc. But I can’t seem to find a way to get the full list of files exported to a list, or better yet a full list of files that should be converted to a list.

something like this so I can monitor it in the bat file (and comment out the ones I want to do that day:

c:\python27\python manual.py -a -i “\hades\movies\3 Days to Kill (2014)” >> D:\Convert\log.txt
c:\python27\python manual.py -a -i “\hades\movies\8 Mile (2002)” > D:\Convert\log.txt
c:\python27\python manual.py -a -i “\hades\movies\8MM (1999)” > D:\Convert\log.txt

OK. first and for most I want you to shut down Plex and copy the plex database out to a new location to play with as a test. Re-Start plex again. This is just for safety and always a good idea to do when testing anything new.

Download and install SQLite3 binaries onto your plex server from here: SQLite Download Page
You will want SQLite3 and all the files below in the same directory.

What’s sharing from here on out is for Windows since it uses batch files but would be super easy to change for Linux. The important part is to get the idea how to do it.

create a new text file called transcode.txt and put the following in it:
.open ‘C:\PlexData\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.db’
.output Transcode.bat
SELECT ‘c:\python27\Python C:\convert\manual.py -a -i "’ || file || ‘"’ FROM media_parts join media_items
on media_parts.media_item_id=media_items.id
where container=‘mp4’ and optimized_for_streaming <> 1
order by file;

In the above file change the first line to point to your copy of the database. After you are sure all is working correctly you can change this to point back to your real version.

Now all you have to do is run:
sqlite3 < transcode.txt

You can put that in a batch file if you want.

When you run that it fires up SQLite and uses the contents of the text file to tell it what database to use. It tells it to pipe the output out to transcode.bat (so you can run it when done). It also contains the SQL code used to run against the database.

You can adjust as needed (or I’ll help) to match your system and needs. The output as it stands will generate something like the following in the batch file:
c:\python27\Python C:\convert\manual.py -a -i “F:\TV Shows\Ongoing\Top Shot (2010)\Season 0\Top Shot - S00E01 - Additional Footage.MP4”
c:\python27\Python C:\convert\manual.py -a -i “F:\TV Shows\Ongoing\Top Shot (2010)\Season 0\Top Shot - S00E02 - Contestant Bios.MP4”
c:\python27\Python C:\convert\manual.py -a -i “F:\TV Shows\Ongoing\Top Shot (2010)\Season 0\Top Shot - S00E03 - Elimination Interviews.MP4”

Hope this helps!

Carlo

PS this one particular one above looks ONLY for MP4 files that aren’t “web optimized” (proper moov atom at start of file).

This would be the first iteration you would want to run. After this you can adjust it to look at things like ref frames, 10 bit color, no AAC as first track, or other types other than MP4.

Of course once you take care or your existing MP4s you can just run the normal conversion scripts against your library to take care of AVIs, MPG, MKV, etc.

I’ll help with the different iterations you will need but want to start you off with an EASY SQL to get your “feet wet”.

419

One of the “problems” with the conversion script is that if the video is already in h.264 format it will copy the video instead of transcoding it. Generally speaking this is a good thing with a couple of “gotchas”. If the video is 10 bit it will often cause the Plex Transcoder to kick in because many clients won’t use video with more than 8 bits.

How to handle this:
First, run the file through the conversion scripts as normal and then add it to Plex. We can then run some SQL against the database to look for certain things like 10 bit video which we’ll want to convert. Since this file has already been run through the script the audio will be fine. So we’ll setup to transcode the video while copying the audio tracks.

Create a text file call transcode.txt and put the following inside the text file:
.open ‘C:\PlexData\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.db’
.output Transcode.bat
select ‘C:\Convert\ffmpeg -i "’ || file || ‘" -c:v libx264 -c:v libx264 -preset slow -crf 18 -acodec copy -map 0 -profile:v high -level 4.0 "C:\convert\done’ ||
replace(file, rtrim(file, replace(file, ‘’, ‘’ ) ), ‘’) || ‘"’
FROM media_parts join media_items
on media_parts.media_item_id=media_items.id
where media_parts.extra_data like ‘%AVideoProfile=high%%2010%’
order by file
limit 100;

You can remove the limit 100 from above but leave the semi-colon at the end. The 100 simply builds a file of 100 files to process per batch. You will need to change the first line to point to your database. Change the 3rd line to point to your version of ffmpeg.

This uses the sqlite3 program mentioned in the previous post so make sure that is installed.
This will create a batch file called transcode.bat that you can run. It doesn’t use the conversion scripts but instead calls ffmpeg directly which we need to do because the scripts will not transcode the video since it’s already in h.264 format.

Output of the files will be in c:\convert\done directory but you can change that location above as well.

Carlo

420

@cayars
Maybe this is more appropriate for a PM, if so say so. :slight_smile:
How do you implement your method of sharing your Plex media with several friends? Do you do this via Plex Home, or some other way? Thanks, and sorry if this was explained and I missed it…