Cayars - Setup walk through and some tips and tricks

General Discussions
1121

@cayars said:
Yea don’t know. Originally I did a bunch of things and gave them back. Most of my changes are now in mp4_sickbeard.

I do have some (minor) changes for sound normalization, HW transcoding, and a couple of logic changes but that’s about it at this point. The current version I started that you guys don’t have is working on auto de-interlacing and probably comskip support to cut commercials from DVR recordings.

I see. It’s just that I see that mp4_sickbeard is now updated again and haven’t had the chance to go check the changelogs yet but there are a couple of new options in autoprocess.ini. Are these new features incorporated on the version that you’re working on?

1122

At this point in time it’s pretty easy to switch back and forth between versions. Mine is just tweeked for Plex use and direct transcoding with audio normalization. They are so close I could just abandon mine at this point and help people setup for HW transcoding when needed or to get audio normalization working.

I’m seriously thinking of starting clean with a focus on DVR recorded files which need proper de-interlacing, CC to subtitle support as well as MPEG2 to H.264 support along with most of the conversion features already in the script. Sans 3rd party support. I’d much rather work in C++ than python which a hate. I’d kick out releases so much quicker this way!

Carlo

1123

@cayars said:
At this point in time it’s pretty easy to switch back and forth between versions. Mine is just tweeked for Plex use and direct transcoding with audio normalization. They are so close I could just abandon mine at this point and help people setup for HW transcoding when needed or to get audio normalization working.

I’m seriously thinking of starting clean with a focus on DVR recorded files which need proper de-interlacing, CC to subtitle support as well as MPEG2 to H.264 support along with most of the conversion features already in the script. Sans 3rd party support. I’d much rather work in C++ than python which a hate. I’d kick out releases so much quicker this way!

Carlo

Off topic, but I’m curious as to why you hate Python (really just a curiosity question, not challenging you)? Can send in a PM if you don’t want to have it derail the thread.

-Shark2k

1124

@cayars said:

@almontef said:
@cayars I installed mediainfo and selected the media in question, but I don’t know where in the output I should be looking for to determine if mediainfo recognizes the media as web optimized or not. I did not see any data in the output to be able to make that determination. Perhaps you could point me to what output I should be looking at? Thanks in advance.

Let’s kill a couple of steps in one shot. Try running something like this line after adjusting for your system:
C:\Python27\Python C:\Python27\Scripts\qtfaststart -l “f:\movies op gun (1986) op gun (1986).mp4”

Obviously change the path to your python directory as well as the file name you want to check. You should get an ouput similar to:

ftyp (24 bytes)
moov (2759896 bytes)
mdat (3217089508 bytes)
free (55 bytes)

What we are interested in is the MOOV and MDAT items. If the index to the MOOV is lower/less than the index to the MDAT then this is a WEB OPTIMIZED file. If the MOOV index number is larger than the MDAT number then the moov atom comes after the data and it’s not optimized.

If you get an error or the qtfaststart file isn’t in the your scripts directory under the python install directory then you didn’t follow the directions correctly when installing the script or skipped a PIP install. In which case just follow the instructions.txt file in the zip again (won’t hurt to rerun commands).

Carlo

I did a bit more digging into this. So like I mentioned in my previous post the output of the above command gives me this:

So according to this output it does appear to be web optimized, yet after analysis of the media within Plex it still shows as not being web optimized.

Just for some background, I noticed this issue using some selects provided in this thread to identify non-web optimized files. Maybe 50-75 files came back. After also running some selects to identify mksv that could be converted, I went ahead and ran those files through the @cayars scripts. When I ran the select statement to identify non-web optimized files after the scripts completed, I was surprised to see that my 50-75 non web optimized files are still there along with hundreds of newly converted MKVs. This is what lead me on this journey.

So, getting back to the problem, I went ahead and changed “relocate_moov = True” in the ini file and ran this media through the script again. Here is the comparison of the qtfaststart for comparison:

Before:
ftyp (32 bytes)
free (8 bytes)
mdat (1785348534 bytes)
moov (11128395 bytes)

After:
ftyp (32 bytes)
moov (11128583 bytes)
mdat (1785348400 bytes)

So in both cases they are web optimized as far as qtstart is concerned. Not sure if this could be related but note that mdat is listed before moov in the before output. So, after analyzing the media, it appears that Plex now sees the media as web optimized!

So it seems (at least for me) that moving the atom outside of the ffmpeg processing is confusing Plex for some reason and it refuses to recognize the media as web optimized. @cayars I would love to get your thoughts on how to proceed given this new information. I would hate to have to run my media back through the scripts if in fact they are already optimized from the first run, but I am concerned that Plex may serve out the media differently to the clients if it thinks the media is not web optimized. Thanks again for all your help.

1125

Following This

1126

I answered this before but don’t see my post anywhere.

If you are creating MKVs then you’re using the script in a way it wasn’t designed to be used. MKV files them selves can’t be web optimized per say as don’t work in the same way as it would in an MP4.

So using MKV they will always be non optimized since that’s the nature of the beast with that container. It’s a storage container but wasn’t designed for streaming. Plex does work arounds with MKV and other file types that aren’t natively streamable.

1127

Is there a current version of the script ( convert.zip / manual.py ) available to download? I tried ftp://ayars.tv/ from https://forums.plex.tv/discussion/comment/931888/#Comment_931888 but it looks to be down.

1128

On page 52, second or third post.

1129

5th post actually.
From @cayars

1130

Thank you!

1131

@cayars - Long ago in this thread (yes I went through all nearly 60 pages), you mentioned some script work that would allow you to set the “Date Added” in the database to be equal to the “File Date”, allowing one to move items around in and out of the database without everything suddenly showing back up as Recently Added. Can you point me in the right direction? I would love to be able to do this.

Thanks for all you do - and a great job with your conversion script work.

1132

@cayars Can you do pleeeeeeease make a transcoding script that works in Linux? I want to move to Ubuntu for reduced administration this year, but transcoding (which I was doing with MCEBuddy) is holding me back.

I think your current script could work with the right binaries installed and a little tweaking.

1133

@chilson said:
@cayars Can you do pleeeeeeease make a transcoding script that works in Linux? I want to move to Ubuntu for reduced administration this year, but transcoding (which I was doing with MCEBuddy) is holding me back.

I think your current script could work with the right binaries installed and a little tweaking.

I use his scripts in Linux (openSUSE) and they work fine. If you have the correct dependencies installed, and the right version of Python, there should be no issue.

1134

@ember1205 said:

@chilson said:
@cayars Can you do pleeeeeeease make a transcoding script that works in Linux? I want to move to Ubuntu for reduced administration this year, but transcoding (which I was doing with MCEBuddy) is holding me back.

I think your current script could work with the right binaries installed and a little tweaking.

I use his scripts in Linux (openSUSE) and they work fine. If you have the correct dependencies installed, and the right version of Python, there should be no issue.

Good to know. Looking at using Ubuntu, and when 18.04 comes out just sticking with it so I can just use an LTS for years. From downloading his conversion script and looking in the ZIP, I saw a lot of EXEs and assumed the scripts also use Windows file paths, etc.

1135

The zip comes ready to use for Windows but is easy to use for Linux as well. You should already have ffmpeg installed so just mod the config file with the proper paths.

Carlo

1136

@cayars

This is phenomenal. It did take me some tinkering to get going on Linux, but I have successfully done one file.

So first and foremost, thank you for your hard work and contribution to this community. If you cannot or have no desire to answer my question, I understand, but I am grateful for what you’ve done here.

I suppose I could also be asking @ember1205 for some assistance with this, because s/he is working with openSUSE.

First: I have no idea how to properly transpose the .bat file into a script or python file that I could use for an “automated” process. If there is an easy way to do this, that would be fantastic, but I am not expecting anyone to do this for me.

Second: With the basic configuration, it pulls from the processing folder and moves to the done folder, but my existing library reads like this: Movies > Movie(Folder) > Movie(File). I have to retain this file structure because of things you won’t go into on this forum ( :smile: ); ergo, moving all those files from separate folders, processing, then moving them back is going to be very time consuming. I’m also looking to replace the file, as physical space prevents me from making copies of my entire library. Is there something I am missing in here that will be able to look into these directories and move forward? I do understand that would require a batching script to function properly, but I am very interested in getting this figured out.

If I have to do it by hand, by all means, I will, but shortcuts are always welcome when you are dealing with several terabytes of files.

If anyone is willing to help me or even just point me in the right direction, I would be eternally grateful.

Again, thank you for your hard work, Carlo.
Also @ember1205 thank you, I was able to fumble my way through figuring out what I have because of your input here as well.

Happy new year and I look forward to any response!

1137

A “.bat” file is a series of commands, useful in various versions of DOS / Windows. Basically, it’s the various commands you would type by hand at the command prompt, one after another. In this case, it also contains a loop so that it can continue to iterate over a location and work on files one at a time.

In Linux, you can accomplish the same thing with a shell script (there are some other ways as well, but sh is the simplest). What I did was to put the utilities into /download/Convert and then I created /bin/transcode.sh as follows:

#!/bin/bash
if [ “$1” == “dir” ] && [ “$#” -eq 2 ]; then
SOURCEDIR=$2
for dir in $SOURCEDIR/*
do
for file in “$dir”/.mkv
do
FILENAME=basename "$file"
EXTENSION="${FILENAME##.}"
FILENAME="${FILENAME%.}"
HandBrakeCLI -i “$dir/$FILENAME.$EXTENSION” --preset=“HQ 1080p30 Surround” -s scan -o “$dir/$FILENAME”.mp4 ; python /download/Convert/manual.py -i “$dir/$FILENAME”.mp4
done
mv “$dir/$FILENAME.$EXTENSION” /mnt/media/done
rm “$dir/$FILENAME”.mp4
mv “$dir” /mnt/media/staging/approved/Movies
done
fi
if [ “$1” == “file” ] && [ “$#” -eq 2 ]; then
FILENAME=$2
EXTENSION="${FILENAME##.}"
FILENAME="${FILENAME%.*}"
fi

What I do is this: After ripping content, I end up with /path/to/Movie (year)/Movie (year).mkv

I move the result to /media/work so that it looks like this: /media/work/Movie (year)/Movie (year).mkv

I then change to the work directory with “cd /media/work” and run the script as “transcode.sh dir .”

If I have multiple movies under /media/work, it processes each of them and moves them to the “approved” directory shown in the script. I have other scripts that pick up from there to do other things, but that’s not relevant here.

1138

@ember1205 said:
snip

This looks amazingly useful. I am going to try putting it to work and I will report back to you with how it went.

Thank you so much for the help, you are awesome.

1139

@ember1205 said:
A “.bat” file is a series of commands, useful in various versions of DOS / Windows. Basically, it’s the various commands you would type by hand at the command prompt, one after another. In this case, it also contains a loop so that it can continue to iterate over a location and work on files one at a time.

In Linux, you can accomplish the same thing with a shell script (there are some other ways as well, but sh is the simplest). What I did was to put the utilities into /download/Convert and then I created /bin/transcode.sh as follows:

#!/bin/bash
if [ “$1” == “dir” ] && [ “$#” -eq 2 ]; then
SOURCEDIR=$2
for dir in $SOURCEDIR/*
do
for file in “$dir”/.mkv
do
FILENAME=basename "$file"
EXTENSION="${FILENAME##.}"
FILENAME=“${FILENAME%.}"
HandBrakeCLI -i “$dir/$FILENAME.$EXTENSION” --preset=“HQ 1080p30 Surround” -s scan -o “$dir/$FILENAME”.mp4 ; python /download/Convert/manual.py -i “$dir/$FILENAME”.mp4
done
mv “$dir/$FILENAME.$EXTENSION” /mnt/media/done
rm “$dir/$FILENAME”.mp4
mv “$dir” /mnt/media/staging/approved/Movies
done
fi
if [ “$1” == “file” ] && [ “$#” -eq 2 ]; then
FILENAME=$2
EXTENSION="${FILENAME##.}”
FILENAME=“${FILENAME%.*}”
fi

What I do is this: After ripping content, I end up with /path/to/Movie (year)/Movie (year).mkv

I move the result to /media/work so that it looks like this: /media/work/Movie (year)/Movie (year).mkv

I then change to the work directory with “cd /media/work” and run the script as “transcode.sh dir .”

If I have multiple movies under /media/work, it processes each of them and moves them to the “approved” directory shown in the script. I have other scripts that pick up from there to do other things, but that’s not relevant here.

SOUND!

1140

Hi guys,

Sorry if this is a little off topic, but I have decided I am going to attempt to ditch my Plex cloud server + manual uploads to put together a Linux VPS setup with the help of the setup info here. I use a couple of roku devices at home and also use my phone, so looking forward to some auto stereo downmixes for the Roku & some sync capability for the mobile.

Can the scripts be setup to just mux the existing video with converted audio? I don’t want to transcode on the VPS if not needed. Also could I get away with 1G RAM / 250G local storage (+GSuite), or should I stump up for 2G RAM/500G local storage?

Many thanks,