Cayars - Setup walk through and some tips and tricks

Another update:

I moved the VM to my beefier server, increased the CPU count to 12 CPU (threads, really - end result is the equivalent of 6 X5680 @ 3.33GHz CPU’s) and pushed the RAM to 48G.

I then processed Avatar. Original file was 39,060,973,670 bytes and the processed file was 20,290,862,516 bytes. That’s almost a 50% decrease in size, and I wasn’t able to observe and difference in PQ from the original. Total process time for everything with all of the extra power was about 4.5 hours. I saw the CPU run pretty solidly at 100% while the memory didn’t seem to go beyond 16G.

Still, running the -same- original file through HandBrakeCLI resulted in an equally similar quality file but only 4,946,985,515 bytes in size. And the processing time was similar.

Here’s where I think I’m at with this:

  • HandBrakeCLI gets me the reduction in file size that I want along with enabling forced subtitles correctly
  • Carlo’s scripts get me a volume boost with the audio

Although both take some time to run, it would seem that running everything through -both- processes will get me to where I want to be with file size, h264 level (4.0), forced subtitles, audio boost, and tweaks like moving the MOOV atom.

One last update…

I just ran another MKV file from a BluRay rip through HandBrakeCLI. Took a little under 2 hours to pare it down to about 1/6 the original size. Passed it through Carlos’ scripts to “finish it off”… Only took about 5 minutes.

Same VM as above was used for this.

About the Audio Codecs.
If in the ini File DTS is set, the File contains Dts HD MA - will it transcode it down to dts?

Something to check if you are worried about file size over viewable quality is to change the CRF setting. I use 18 by default and you could go as high as 22 (default of ffmpeg) and be “normal”. There is a good chance you are using a setting of 22 in Handbrake anyway.

So if you want to compare apples to apples instead of apples to oranges try changing the CRF to 22 and then run another file through the script. :slight_smile:

If DTS is set then that will allow DTS to be copied over to the new file it produces. The only “audio conversion” per say that is done is the creation of the two channel AAC audio channel (first track) and this can be done from multiple different sources.

@cayars said:
Something to check if you are worried about file size over viewable quality is to change the CRF setting. I use 18 by default and you could go as high as 22 (default of ffmpeg) and be “normal”. There is a good chance you are using a setting of 22 in Handbrake anyway.

So if you want to compare apples to apples instead of apples to oranges try changing the CRF to 22 and then run another file through the script. :slight_smile:

If DTS is set then that will allow DTS to be copied over to the new file it produces. The only “audio conversion” per say that is done is the creation of the two channel AAC audio channel (first track) and this can be done from multiple different sources.

Thanks, Carlo. I’ll be a guinea pig and run some more tests. :slight_smile:

I -believe- the default CRF setting for ffmpeg is actually 23, but that might just be the particular flavor / version.

I’ve re-configured the INI file to use 22 and ran a file through again. I ended up reducing a file of size 29,210,551,473 bytes to 5,413,933,570 bytes using only your scripts in 2 hours 31 minutes. Using my HandBrakeCLI transcode coupled with your scripts to relocate the MOOV atom, I have previously reduced this same file to 6,431,698,226 bytes in what I believe was a similar amount of time.

As before, I’m using the same VM to do all of the testing, so the only thing that I’m changing is the set of programs / scripts that do the actual work. Source and destination files are on the same servers, on the same disks for all tests with drives nfs-mounted. Watching the network load via the virtual server management program shows me that the network is not factoring in (not even close) to performing the conversions. In fact, the only time I see the conversions slow down is if I’m doing a large file transfer across the network involving the server that stores the source and destination files. The high network load from the large file transfer slows down the communications for this process during the period of the transfer. So, I’ve restricted any large transfers to be done outside of my testing windows.

Since the long-term goal is to get to a point where I can set up a “large conversion job” to sweep through my entire library unattended, I’m going to look at creating an out-of-band network specifically for the VM to access the two file servers so that no data transfers or access on the front-end network will be able to slow things down for the transcoding / conversion.

Also, I looked at the preset I’m using in HandBrake, and it -appears- to be using an RF of 18 (that’s their CRF variable). Seems odd to have such a stark difference in final size if the RF/CRF and max video settings of 4 were both the same.

@cayars Everything works now.
When I set DTS at the codecs in the .ini File and the Source has DTS HD MA Codec it keeps this Track - perfect
One thing about Subtitles.
Got a BD Rip with Subtitle Codec bdpg
Log says:
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #3: hdmv_pgs_subtitle [ger].
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #4: hdmv_pgs_subtitle [ger].
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #5: hdmv_pgs_subtitle [eng].

But it does not extract the Subtitles?

@loibi2 said:
@cayars Everything works now.
When I set DTS at the codecs in the .ini File and the Source has DTS HD MA Codec it keeps this Track - perfect
One thing about Subtitles.
Got a BD Rip with Subtitle Codec bdpg
Log says:
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #3: hdmv_pgs_subtitle [ger].
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #4: hdmv_pgs_subtitle [ger].
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #5: hdmv_pgs_subtitle [eng].

But it does not extract the Subtitles?

What are your various subtitle settings?

I’m reasonably sure that MP4 only supports a single subtitle track burned into the file itself, and I believe that you would only want to do this for Forced Subtitles (when the native language isn’t being spoken in the movie). Otherwise, the subtitles would be stored in separate files and I know that Carlo has accounted for the ability to download subtitles and store them in the subtitle tracks (although I don’t have a use case for this).

@ember1205 thank you but I ended up using the SubZero Plugin, works perfect for me.

I’ve tried to setup Carlo’s scripts but my linux proficiency is ZERO and my PMS runs on Mac OS (media stored on a NAS). I’m having major issues playing back all 1080p HD Plex DVR recordings in Plex on my Apple TV 4 so I’m hoping that maybe these scripts can convert the .ts files to something playable.

I’m located in Australia and many (not all) of our TV networks use h264 video and ac3 audio. A few I think use MPEG2 video and some similar audio.

Would Carlo’s script actually make my DVR recordings playable on the apple TV?

Can any kind soul help me with setting it up via perhaps a Team Viewer remote session? I’m happy to donate some funds via Paypal for a box of beers for your time.

@jayz78 said:
I’ve tried to setup Carlo’s scripts but my linux proficiency is ZERO and my PMS runs on Mac OS (media stored on a NAS). I’m having major issues playing back all 1080p HD Plex DVR recordings in Plex on my Apple TV 4 so I’m hoping that maybe these scripts can convert the .ts files to something playable.

I’m located in Australia and many (not all) of our TV networks use h264 video and ac3 audio. A few I think use MPEG2 video and some similar audio.

Would Carlo’s script actually make my DVR recordings playable on the apple TV?

Can any kind soul help me with setting it up via perhaps a Team Viewer remote session? I’m happy to donate some funds via Paypal for a box of beers for your time.

BluRay (1080) rips are typically .m2ts (when they’re not .mkv) and not .ts, so I’m wondering where the files came from / how you got them.Have you tried just renaming a .TS to .MPG and seeing if it will play any better?

Have you tried looking a the web interface for the server while you’re trying to play back a file to see what method it’s using? For example, if it’s transcoding, could you switch it to direct play instead? And, if it’s transcoding, is the Mac underpowered to convert on the fly?

What does your NAS have in it for a network interface? What about the Mac?

When you’re asking about Linux assistance, are you talking about the Mac? That’s not actually, Linux, although it is very similar (it’s FreeBSD-based). Personally, I wouldn’t install all of the extra pieces onto the Mac - I would build a Linux host for the purpose of doing the conversions.

@ember1205 said:

@jayz78 said:
I’ve tried to setup Carlo’s scripts but my linux proficiency is ZERO and my PMS runs on Mac OS (media stored on a NAS). I’m having major issues playing back all 1080p HD Plex DVR recordings in Plex on my Apple TV 4 so I’m hoping that maybe these scripts can convert the .ts files to something playable.

I’m located in Australia and many (not all) of our TV networks use h264 video and ac3 audio. A few I think use MPEG2 video and some similar audio.

Would Carlo’s script actually make my DVR recordings playable on the apple TV?

Can any kind soul help me with setting it up via perhaps a Team Viewer remote session? I’m happy to donate some funds via Paypal for a box of beers for your time.

BluRay (1080) rips are typically .m2ts (when they’re not .mkv) and not .ts, so I’m wondering where the files came from / how you got them.Have you tried just renaming a .TS to .MPG and seeing if it will play any better?

Have you tried looking a the web interface for the server while you’re trying to play back a file to see what method it’s using? For example, if it’s transcoding, could you switch it to direct play instead? And, if it’s transcoding, is the Mac underpowered to convert on the fly?

What does your NAS have in it for a network interface? What about the Mac?

When you’re asking about Linux assistance, are you talking about the Mac? That’s not actually, Linux, although it is very similar (it’s FreeBSD-based). Personally, I wouldn’t install all of the extra pieces onto the Mac - I would build a Linux host for the purpose of doing the conversions.

FYI @jayz78 is talking about the DVR recordings, not Blu-ray rips.

-shark2k

@shark2k said:

FYI @jayz78 is talking about the DVR recordings, not Blu-ray rips.

Thanks for the reminder. I saw that, but it didn’t stick with my brain as I continued reading. I only honed in on the 1080 piece.

My DVR recordings are all in .MKV format which is another reason that it didn’t stick with me mentally that we’re talking about the DVR piece.

@loibi2 said:
@cayars Everything works now.
When I set DTS at the codecs in the .ini File and the Source has DTS HD MA Codec it keeps this Track - perfect
One thing about Subtitles.
Got a BD Rip with Subtitle Codec bdpg
Log says:
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #3: hdmv_pgs_subtitle [ger].
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #4: hdmv_pgs_subtitle [ger].
2017-07-21 10:14:39 - MANUAL - INFO - Subtitle detected for stream #5: hdmv_pgs_subtitle [eng].

But it does not extract the Subtitles?

I know you found a workaround already, but for others, hdmv_pgs_subtitle is an picture format subtitle, which is not supported in an .mp4 container.

@ember1205 said:
Have you tried looking a the web interface for the server while you’re trying to play back a file to see what method it’s using? For example, if it’s transcoding, could you switch it to direct play instead? And, if it’s transcoding, is the Mac underpowered to convert on the fly?
It’s a bit of a mixed bag. Sometimes it’ll say it’s Direct Play (when it shouldn’t be given it is a .ts file with AC3 audio and other times it shows transcoding H264 to H264.

What does your NAS have in it for a network interface? What about the Mac?
My NAS is a Synology 1512+. The PMS runs on a Mac Mini 2011 i5 running Sierra.

When you’re asking about Linux assistance, are you talking about the Mac? That’s not actually, Linux, although it is very similar (it’s FreeBSD-based). Personally, I wouldn’t install all of the extra pieces onto the Mac - I would build a Linux host for the purpose of doing the conversions.
Well I thought that it would be easier to get FFMPEG and Carlo’s scripts to work on the Mac given I understand it a lot better than Linux and since my PMS is on the same system as well. However the other option is getting it working on the NAS where the media is stored however with previous experience I know it’s Linux based operating system is a BIG pain for newbies like me when it comes to sorting out permissions, SSH and all the other geeky stuff :slight_smile:

@jayz78 said:

@ember1205 said:
Have you tried looking a the web interface for the server while you’re trying to play back a file to see what method it’s using? For example, if it’s transcoding, could you switch it to direct play instead? And, if it’s transcoding, is the Mac underpowered to convert on the fly?
It’s a bit of a mixed bag. Sometimes it’ll say it’s Direct Play (when it shouldn’t be given it is a .ts file with AC3 audio and other times it shows transcoding H264 to H264.

What does your NAS have in it for a network interface? What about the Mac?
My NAS is a Synology 1512+. The PMS runs on a Mac Mini 2011 i5 running Sierra.

When you’re asking about Linux assistance, are you talking about the Mac? That’s not actually, Linux, although it is very similar (it’s FreeBSD-based). Personally, I wouldn’t install all of the extra pieces onto the Mac - I would build a Linux host for the purpose of doing the conversions.
Well I thought that it would be easier to get FFMPEG and Carlo’s scripts to work on the Mac given I understand it a lot better than Linux and since my PMS is on the same system as well. However the other option is getting it working on the NAS where the media is stored however with previous experience I know it’s Linux based operating system is a BIG pain for newbies like me when it comes to sorting out permissions, SSH and all the other geeky stuff :slight_smile:

Your NAS has two Gig NIC’s in it, the Mac Mini has one. Even though the Mac has to use the NIC to draw the file in and then send it back out, your consumption level from your client device should be well below 100Mb and this shouldn’t cause you any issues.

It’s possible that the Mac is simply underpowered in some facet - how much RAM does it have? Have you run Activity Monitor on it to see if -IT- is the bottleneck?

I have a large Anime collection and I want to add some of my series to my plex server. Is there an option in the convert scripts that it will take the subtitles in the .mkv file hardcode them during the conversion to mp4?

Carlo,

Looking for some insight here…

I’ve run your scripts against a number of my media files and have used the settings you’ve suggested for controlling quality and such. I’m seeing potentially a 50% space savings by running my media files through HandBrake for transcoding (which, I believe, is a front-end to ffmpeg).

How can I properly evaluate image quality from the two methods to determine whether or not I actually see a difference and whether that difference is worth one over the other?

My 2016 75" 4K Sony TV does not seem to show me any perceptible difference in quality using the Plex app to play back movies. But there HAS to be a difference somewhere that I can see… So, how do I find it?

A step by step video guide would be really nice addition.

I’ve been watching this thread for a long time now. Never really understood what it was all about but it interested me. I now have a file with the extension .ts, it plays perfect with VLC and from Plex but I wanted it to be MP4. It is a Polish TV program, in VLC it has options for English & Polish Subs and under the ‘Audio’ Tab it has options for English and Polish ‘Track 1 being English and Track 2 being Polish’. Even when I click for the English track all the spoken words are Polish. The Subtitle options work OK as in if I click English I get English Subs and likewise with Polish.
I decided to try Carlo’s scripts to see if I could get it to play with English audio or at least extract the English Subs the my choice is to burn the subs in to the media file (I do this for my Mums benefit as she can never remember how to get subtitles when watching Plex).
What I end up with is a .srt file that is empty and a .mp4 file that only plays audio in Polish (even though it now has 2 Audio options both English) and no subtitle options any more. I have attached the output for VLC Tools-Codec Information from the original file where it appears to say it has English & Polish audio & subtitles.
Am I doing something wrong is there something wrong with the file I.E. does it not contain English audio at all, but it does contain English subs as they work in VLC but don’t get extracted using Carlo’s scripts.
Sorry this is so long I just wanted to get as much info in as I could.


The above is from original file and below is after it has been through the scripts.

I hope this makes sense and any help would be very much appreciated.
Thanks Grant

@Grantp said:
I’ve been watching this thread for a long time now. Never really understood what it was all about but it interested me. I now have a file with the extension .ts, it plays perfect with VLC and from Plex but I wanted it to be MP4. It is a Polish TV program, in VLC it has options for English & Polish Subs and under the ‘Audio’ Tab it has options for English and Polish ‘Track 1 being English and Track 2 being Polish’. Even when I click for the English track all the spoken words are Polish. The Subtitle options work OK as in if I click English I get English Subs and likewise with Polish.
I decided to try Carlo’s scripts to see if I could get it to play with English audio or at least extract the English Subs the my choice is to burn the subs in to the media file (I do this for my Mums benefit as she can never remember how to get subtitles when watching Plex).
What I end up with is a .srt file that is empty and a .mp4 file that only plays audio in Polish (even though it now has 2 Audio options both English) and no subtitle options any more. I have attached the output for VLC Tools-Codec Information from the original file where it appears to say it has English & Polish audio & subtitles.
Am I doing something wrong is there something wrong with the file I.E. does it not contain English audio at all, but it does contain English subs as they work in VLC but don’t get extracted using Carlo’s scripts.
Sorry this is so long I just wanted to get as much info in as I could.
The above is from original file and below is after it has been through the scripts.

I hope this makes sense and any help would be very much appreciated.
Thanks Grant

Is there anyway you can upload the file so I can take a look at it? PM me a link if necessary.

I have a feeling this is probably just an issue with sickbeard’s mp4 automator, as the creator has added in support for languages other than english to be handled, but I doubt he’s tested it heavily as he uses english in his media, and something in this file is breaking it.