Cayars - Setup walk through and some tips and tricks

General Discussions
1161

@weezzelboy said:

Is there a way to Have this script search for anything that is not MP4 and than convert accordingly? The issues i have is that sometimes i have to restart the server or have to close the conversion script and when i restart it it want to convert everything all over again.

If it were me, I would write a “control” script that traverses your filesystem, identifies files not yet converted, then passes each in turn to the processing script that Carlo has provided. This would allow you to handle things like moving the file out of the library (or not), processing, creating files that show that the file is in progress, deleting status files when work is done, etc.

1162

You could modify the SQL script to look for container not equal to mp4. If you want help with this just say the word.

Carlo

1163

@cayars said:
Here is a set of scripts that you can use to automate mp4 creation of files.

I’d recommend running your files through FileBot first to rename them correctly before running them through this script.  It can pull SRT files so everything ends up with proper names this way.

The only pre-requisite is to have python 2.7.9 installed. Python Release Python 2.7.9 | Python.org

(I’m running 64bit version for windows installed to C:\Python27)

This is a modified version of sickbeard_mp4_automator with modifications by me. I pulled out all the integration features to make setup much easier.  I also modified it to produce higher quality MP4 files that will direct play better with Plex.

Download convert.zip from ftp://ayars.tv

username and password are both guest

Unzip to C: drive. You will end up with C:\Convert

There are two directories (can be changed) inside this folder that are used for the media

Process is where you put any files you want to convert

Done is where the files will end up after processing

You can run this multiple ways but the easiest is to use the Run.bat file.

If you edit this file you will notice it’s one line that reads: c:\python27\python manual.py -a -i c:\convert\process

If you install python to a different folder then change the python directory at the beginning of this line.

If you want to process files from a different location then c:\convert\process then just change this in the batch file.

If you want to change the final location of where the final files are stored then edit autoProcess.ini file and change the output_directory setting.

That’s pretty much it.

Put files in the process directory and run the batch file

What this will do:

It will process each and every file in the “process” directory.

It will remux files that are h.264 and will transcode files that aren’t h.264 video

It will pull out any english subtitles and create SRT files from them.  If you want to pull out other languages then modify the subtitle-language setting in the ini file.  Multiple languages are supported by commas

The completed file will not have subtitles (we pulled them out)

It will clear all tags in the mp4 file so Plex doesn’t pick up and use these tags instead of the proper meta-data

It will create an AAC track as the first track upto 256K in size.

It will remove all audio tracks not in english.  If you want other languages included modify audio-language setting (ini file) and include any languages you want kept in the MP4 file.  Multiple languages are supported by commas

When it transcodes it wil use an h.264 profile of high; a level of 4.0; crf setting of 20.  These are basically the same as Handbrake High Profile settings with a bit more refinement for Plex use that allows it to direct play more often.

In all cases the final MP4 file has the MOOV ATOM at the beginning of the file (ie web optimized)

Advanced setting:

You can limit the total bitrate for the video.  If for example you use a client that has an upper limit of 12000 KB you can set the option video-bitrate option to something like 11000 which gives you a bit of headroom for a couple of audio tracks.

There is more that can be done with this set of scripts but lets start off with these. :slight_smile:

Let me know what you guys think,

Carlo

PS I’ll show you how later to pull info directly from the Plex database to know what files need converting depending on your criteria.

Hey, I came across this script but the FTP seems to be down. Is there any way I can still get it? And can this script also do things like, if it’s bigger than 720p, make it 720p, if it’s equal or less, keep the resolution and same with, if it’s higher than 44,1KHz, make it 44,1KHz, if it’s equal or less keep it that way. I’m currently using the conversion feature of VLC, but I ran across the same problem using handbrake. Which is logical. Some files turn out bigger than before. I guess your script might finally put me on the right way.

1164

The script itself (here in the forums an not available via FTP anymore) won’t handle the music or resolution filtering on it’s own. However you can set these as options that will get used for any reprocessing of files (half way there).

Then you can use SQL to query to the Plex database looking for resolution, audio tracks, etc to find files that match that criteria that you want to reprocess. Basically with a slight bit of work to create the one SQL statement you will need, you can achieve this.

Carlo

1165

And where can I find the script? Everything links back to the same post where the script is said to be available via FTP.

1166

PS: I just “found out” it should be on page 52. But on page 52 there’s only talk about hard disks and usb enclosures.

edit:
Nevermind, found it on page 51. Thank you :slight_smile:

1167

@cayars I need a very simple FFMPEG post processing script created to remux Plex DVR reordering from TS to MKV. I have no idea how to accomplish this and was hoping you might be able to help me out?

1168

@hstamas said:
@cayars I need a very simple FFMPEG post processing script created to remux Plex DVR reordering from TS to MKV. I have no idea how to accomplish this and was hoping you might be able to help me out?

All that a “script” is is a combination of the various commands you would run from the command line, possibly allowing for variables. If you already know the specific commands that you need, combining them together into a script usually isn’t too difficult.

Apologies if you’re already at this level of understanding, but thought I would at least add in case others following along aren’t thinking about it this way.

1169

@ember1205 said:

@hstamas said:
@cayars I need a very simple FFMPEG post processing script created to remux Plex DVR reordering from TS to MKV. I have no idea how to accomplish this and was hoping you might be able to help me out?

All that a “script” is is a combination of the various commands you would run from the command line, possibly allowing for variables. If you already know the specific commands that you need, combining them together into a script usually isn’t too difficult.

Apologies if you’re already at this level of understanding, but thought I would at least add in case others following along aren’t thinking about it this way.

Yep and thats where Im stuck. I know what I need but have no idea how to write it. Im looking for a script that I can plug into Plex’s “Postprocessing Scripts” section of the DVR. All I need is for it to remux the recorded TS file to a MKV file and overwrite the original.

Sounds easy enough but completely foreign to me Im afraid.

1170

The script> @hstamas said:

@cayars I need a very simple FFMPEG post processing script created to remux Plex DVR reordering from TS to MKV. I have no idea how to accomplish this and was hoping you might be able to help me out?

Setup the script as is and it will do exactly what you are asking for except it will create MP4 vs the MKV “out of the box” which works on more devices.

1171

Part of what you’ll need to understand is what variables get passed by Plex over to the various scripts so that you can account for them in your script. Plus, the exact operating system that you’re working with and what sort of “scripting language” you have to work with will play into this.

I do a lot with basic shell scripts in Linux but those don’t port to Windows directly - I have to rewrite them in CMD compatible terms to run on Windows. I’ve actually written my own very basic script for Linux that does a little “work” with some files and moves things around, then calls Handbrake to do some processing, and THEN calls Carlo’s scripts for the final steps (I actually handle the transcoding to MP4 and file size reduction with Handbrake and not the FFMPEG stuff that Carlos does).

My script has to be able to account for the various file name changes as the extensions change, so I’ve had to use a lot with variables.

#!/bin/bash
if [ "$1" == "dir" ] && [ "$#" -eq 2 ]; then
  SOURCEDIR=$2
  for dir in $SOURCEDIR/*
    do
    for file in "$dir"/*.mkv
      do
      FILENAME=`basename "$file"`
      EXTENSION="${FILENAME##*.}"
      FILENAME="${FILENAME%.*}"
      HandBrakeCLI -i "$dir/$FILENAME.$EXTENSION" --preset="HQ 1080p30 Surround" -s scan  -o "$dir/$FILENAME".mp4 ; python /download/Convert/manual.py -i "$dir/$FILENAME".mp4
      #HandBrakeCLI -i "$dir/$FILENAME.$EXTENSION" --preset="HQ 720p30 Surround" -s scan  -o "$dir/$FILENAME"-720.mp4 ; python /download/Convert/manual.py -i "$dir/$FILENAME"-720.mp4
      done
      mv "$dir/$FILENAME.$EXTENSION" /mnt/media/done
      rm "$dir/$FILENAME".mp4
      #rm "$dir/$FILENAME"-720.mp4
      mv "$dir" /mnt/media/staging/approved/Movies
    done
fi
if [ "$1" == "file" ] && [ "$#" -eq 2 ]; then
  FILENAME=$2
  EXTENSION="${FILENAME##*.}"
  FILENAME="${FILENAME%.*}"
fi
1172

@cayars said:
The script> @hstamas said:

@cayars I need a very simple FFMPEG post processing script created to remux Plex DVR reordering from TS to MKV. I have no idea how to accomplish this and was hoping you might be able to help me out?

Setup the script as is and it will do exactly what you are asking for except it will create MP4 vs the MKV “out of the box” which works on more devices.

Ive downloaded your convert-2.2 package @cayars. I see that I need to edit autoProcess.ini which I can do but I see mention of MKV to MP4. Will this work with TS to MP4 without having to change anything?

Finally I just point the Plex Postprocessing dialog in the DVR section to your file “postConversion.py” and thats it?

UPDATE: Trying it out with the above assumptions and as expected the post processing is failing. I do see, in the various .py files mentions of the FFMPEG path pointing to FFMPEG.exe. Since I am on a Mac there is no EXE extension. I did change the autoProcess.ini fine to point to “/Applications/FFMPEG/ffmpeg”. Is that enough? This is why these things are so frustrating. Unless you know the coding its impossible to figure out on your own!

1173

Before attempting to run this in post processing (which I wouldn’t do anyway), I’d suggest making sure it’s working correctly from the command prompt.

Make sure you following the instructions in the text file to install python and the needed libs. Then edit the batch file to point to a location with a TS file.

Once you have this working I can give you a modified file you can drop in that will also do deinterlacing automatically for files that need it (all SD and 1080 recorded content).

Carlo

1174

@cayars
Thank you for the excellent script and all the helpful results!

I have been back and forth on this thread and searched around quite a bit, I think I found answers to a few questions but have some remaining I haven’t found straight answers for yet (and some were from 2016, want to ensure it is still correct).

Here goes…

  1. I am embedding subtitles, and it is using mov_text, any reason I shouldn’t do this over not embedding and using SRT? Maybe a third choice?
  2. I am running on an i7-7700K, I keep bouncing back and forth on using QuickSync to transcode as it is faster, but in this thread and elsewhere I hear about worse quality, file size issues, I found something from a year or so ago that you suggested using software for this run, and hardware may be good for live transcoding. Is this still your view?
  3. I am running the script against my Movies library and TV Shows library. Using some information in this thread, I am going with two instances of the script, one for Movies and one for Shows, but don’t want to kill my server’s ability to run Plex, so I set the config for the script to “Threads - 1”. I saw you mention a while back that this is good for ensuring transcode quality as well. This still good?
  4. Last question, I think… Should relocate_moov be set to True or False? I thought True, based on things I have found elsewhere, but I found in here that you responded saying something along the lines of, “with how I changed the script, it should be set to ‘False’”. Is this still the case, or was that for something unique? I couldn’t tell for sure.

Thank you again!

1175

  1. I prefer EXTERNAL SRT subtitle files. Just knowing how Plex works this makes subtitles easier to use when transcoding is involved. Having them as SRT and external can be the difference of direct playing vs transcoding when used. Can be worse on 4K.

  2. I like QuickSync or other GPU (I’m using NVENC myself) for real-time transcoding inside Plex but I don’t use it in these scripts personally. While you can get close to the same quality it comes as needing more bitrate and hence bigger files. So for anything I’m arching it CPU only!

Now with that said, I did/do sometimes use GPU encoding on some files. I have the script installed in two different directories so one is for CPU and the other for GPU. I recorded ALL the March Madness NCAA College basketball games. So I used GPU to encode these and deinterlace them. My dad has a Roku which doesn’t care to play TS/mpeg files so I deinterlace and convert these to H.264 at 60fps for 1080i content. I 'm not worried about streaming these outside the house nor was I worried about them archiving them. I record several shows a day off FOX, CNN and MSNBC as well as record local news at 10/11pm on a few local stations and a couple of late night shows like Jimmy Kimmel. I keep a weeks worth of these shows so these too get processed by GPU for the same reason. Faster and they take up limited more space.

  1. While it won’t hurt it’s really not needed anymore. It used to have a bigger affect. However you can test this yourself. Grab a 30 minute show and process it both ways. Then compare file sizes as quality will likely be the same. Limiting the threads will keep the CPU use much lower however. Play with the thread count if you wish.

I just let it use all threads and peg the CPU. Plex never gives me a problem because I’m using QuickSync for real-time encoding if needed. :slight_smile:

  1. With latest versions keep it relocate_moov = True

  2. I should give you guys an update. I’ve only made 1 change that allows deinterlacing when needed which I don’t think the versions you guys have does this. It’s pretty much just been working well so I haven’t touched it in quite a while. I’ll get a new update out shortly with new exes as well.

1176

@cayars said:

  1. I prefer EXTERNAL SRT subtitle files. Just knowing how Plex works this makes subtitles easier to use when transcoding is involved. Having them as SRT and external can be the difference of direct playing vs transcoding when used. Can be worse on 4K.

  2. I like QuickSync or other GPU (I’m using NVENC myself) for real-time transcoding inside Plex but I don’t use it in these scripts personally. While you can get close to the same quality it comes as needing more bitrate and hence bigger files. So for anything I’m arching it CPU only!

Now with that said, I did/do sometimes use GPU encoding on some files. I have the script installed in two different directories so one is for CPU and the other for GPU. I recorded ALL the March Madness NCAA College basketball games. So I used GPU to encode these and deinterlace them. My dad has a Roku which doesn’t care to play TS/mpeg files so I deinterlace and convert these to H.264 at 60fps for 1080i content. I 'm not worried about streaming these outside the house nor was I worried about them archiving them. I record several shows a day off FOX, CNN and MSNBC as well as record local news at 10/11pm on a few local stations and a couple of late night shows like Jimmy Kimmel. I keep a weeks worth of these shows so these too get processed by GPU for the same reason. Faster and they take up limited more space.

  1. While it won’t hurt it’s really not needed anymore. It used to have a bigger affect. However you can test this yourself. Grab a 30 minute show and process it both ways. Then compare file sizes as quality will likely be the same. Limiting the threads will keep the CPU use much lower however. Play with the thread count if you wish.

I just let it use all threads and peg the CPU. Plex never gives me a problem because I’m using QuickSync for real-time encoding if needed. :slight_smile:

  1. With latest versions keep it relocate_moov = True

  2. I should give you guys an update. I’ve only made 1 change that allows deinterlacing when needed which I don’t think the versions you guys have does this. It’s pretty much just been working well so I haven’t touched it in quite a while. I’ll get a new update out shortly with new exes as well.

Thank you so much for the quick response and excellent detail! I made changes to subtitles and threads based on what you said, and am going to keep charging forward with software transcoding, letting Plex do hardware as what you said makes sense.

Keep up the great work and have a good weekend!

1177

@cayars We just missed the olympics, but they’ll come up again so I’ll ask: with your personal media stuff dedicated to the NFL, have you found a similar way to manage the Olympics events?

1178

I have football for both pro (NFL) and college, Basketball for NCAA (college), Sixers only for NBA, NHL, 4 different types of auto racing, Boxing, MLB and probably some other sports I can’t think of off the top of my head that I record.

I have this as a separate Library just for sports. Under each I use folders for each category. I have the library type set to type OTHER in Plex. Given the choice I’ll use an Emby client for Sports since it handles this type of thing a bit better in the way it displays names.

When “special” things come up like March Madness (NCAA basketball), Olympics, etc I’ll create a new folder for these and dump the files here after removing commercials and processing them to MP4 (using script). I don’t really do anything special with these if I’m planning on watching and then deleting (as watched on soonish).

With NFL I like to keep all games so I break them down into YEAR/WEEK.

Basically, you can setup the directories the way you want for this “OTHER” type as you will want to use FOLDER view in the client so you can navigate the directory tree you created.

Olympics are hard to categorize since it’s common to have a broadcast recording that covers multiple sports.

Carlo

PS I still haven’t deleted any of my Olympic recordings yet. :slight_smile:

1179

Has anyone had a issue with CMD lagging.freezing while running the scripts? Normally happens when it goes to “Trying to write tags”. It normally sits there while the cursor is flashing waiting on a input.

1180

I’ve never seen that before.

I’d try doing a complete reinstall again over top of what you have to make sure you didn’t miss any python modules.