Cayars - Setup walk through and some tips and tricks

General Discussions
841

@KarlDag said:

@cayars said:
I’ve got it working in the scripts.

Got some nice little improvements if things seem to keep working well. Due to popular demand I added CRF back as an ini option and I’ve defaulted the video to use Intel QSV which is now giving me really good results and is much faster and it doesn’t conflict with the CRF option now.

I also have audio normalization working but it’s not perfect. It’s not as good as a true two pass (scan and recode) but for our purposes I’m hoping it’s good enough. I’ve got to find some of my bad files to use to test with.

All these settings are ini based and can be changed.

Carlo

That’s awesome. I still didn’t take the time to do this with my files because I figured it would be too time consuming, but if qsv can easily be used and the quality is nice then that’s very interesting.

I’m doing my testing on and older Sony laptop that has an Ivy Bridge GT2 i5 CPU with onboard Intel HD Graphics 4000. So not even a current version of QSV. I’m getting really good results. Hopefully others will get results as good as I am getting.

842

I have released for testing Version 2.2 of the conversion scripts.

ftp://ayars.tv
username: guest
password: guest

If you’re already running version 2.x you can unzip this over top your current setting otherwise if this is a new install or you’re presently running the old version you need to follow the instruction in the instruction.txt file.

What’s new from 2.1 is a couple more formats that it can convert from, such as real media & flash.

I’ve also added Dynamic Audio Normalization. The Dynamic Audio Normalizer re-adjusts the gain factor to the input audio. This allows for applying extra gain to the “quiet” sections of the audio while avoiding distortions or clipping the “loud” sections. In other words: The Dynamic Audio Normalizer will “even out” the volume of quiet and loud sections, in the sense that the volume of each section is brought to the same target level. Note, however, that the Dynamic Audio Normalizer achieves this goal without applying “dynamic range compressing”. It will retain 100% of the dynamic range within each section of the audio file.

Also added to this release officially is GPU (hardware) encoding for both Intel Quick Sync & Nvidia NVENC.

There are now 3 sample INI files named:
autoProcess - CPU.ini - standard CPU conversion
autoProcess - nvenc.ini - For use with nVidia GPU
autoProcess - QSV.ini - For use with Intel QuickSync

Copy one of the above and replace the standard ini file name “autoProcess.ini” with the version you wish to use. Right now the hardward encoding ini files are “experimental” and could still have issues.

I’ll explain later how you can tune this a bit for your specific system after we get some testing done with this release. I’ve only done limited testing for both GPUs mainly to adjust the quality vs file space requirements. While these might get further tuned I’m getting very good results on my computers with the current settings. Let me know how they work for you.

If you’re going to continue to use CPU encoding you will see a new option in the ini called video-crf with a setting of 18. This should give near “lossless” quality at the expense of file space. I prefer quality over size so I use a lower number (lower is higher quality). Depending on your needs you will probably find something between 18 and 23 to be good for you.

If you are NOT writing your output files to c:\convert\done location you will need to modify the ini file and change the “output_directory” setting to match your system.

Happy testing,
Carlo

EDIT: right after posting I tried converting 4K HEVC files with hardware encoding and it choked. Working though it. Both GPUs it appear need different parameters depending on size, type of video. Going to be fun!

843

So after some testing it appears things are working correct for now. If using Nvidia NVENC and have an older card that doesn’t support HVEC it will not process or do a conversion from h.265 to h.264.

If you scroll down on this page https://developer.nvidia.com/nvidia-video-codec-sdk#gpulist you can check what you card family can process. NVENC has worked for me for everything except the HVEC source files I’ve test. I’m still exploring other option but for now understand this may or may not work for if you have HVEC files you are trying to convert. At present it dies and writes the info to the log but doesn’t fall back to CPU based encoding. I’ll explore this but for now just want some testing on what we have.

On the other hand QuickSync is processing all files including weird HVEC 10 with goofy color space to h.264.

844

nvenc supports encoding with HEVC 8 bit on 9xx/10xx cards, HEVC 10 bit is supported on 10xx cards.

If you’re going to add in gpu encoding, you should just use my fork of sickbeard’s mp4 converter – https://github.com/Collisionc/sickbeard_mp4_automator – as I have near-complete support for every nvenc option there is, and handling for the weird edge cases that I’ve run into over the last few months… mostly relating to h265 → h264 conversion and/or colorspace conversion.

I added in support for maxrate, minrate, bufsize, scale_npp, video decoding with nvenc/DXVA2, temporal aq, nvenc presets, nvenc rate control, selecting which nvenc gpu to use, rc_lookahead, and setting output bitrate based on the resolution of the output.

DXVA2 is rock solid for gpu decoding, nvidia’s decoder is faster but will drop frames in certain scenes so I don’t use it currently.

At a minimum, I would enable DXVA2 decoding as it reduces cpu workload by quite a bit. :slight_smile:

Full git difference between my fork and the original here - https://github.com/mdhiggins/sickbeard_mp4_automator/compare/master...Collisionc:master

Explanation of options here -
https://github.com/Collisionc/sickbeard_mp4_automator/blob/master/README.md

845

@Collisionc I’ll check out your fork and play with it. As you probably know these scripts are based of the sickbeard’s mp4 converter. It’s just not forked on Github. Thought about doing that but our needs are slightly different and I didn’t want to support other users beside the Plex base. The first version of my scripts were heavily modified but lots of changes made it back into sickbeard converter making changes easier for me. This new version is much closer to the source but has subtle changes.

In general I don’t do any bit rate limiting nor do I want it and I use the equivalent of software based CRF so that once you have a quality that gives the results you want it “just works” and only uses the bitrate needed to give constant results.

You just nailed the problem I’m having right now which is nvidia h.265 to h.264 which just doesn’t seem to want to work on Gen 1 Maxwell cards due to lack of h.265 support. Going to probably have to filter in the software based decode and let NVENC just handle the h.264 encoding. Not sure it will be worth the effort. I too added some options to NVENC but not rate stuff as we are focused on best quality over file size so we don’t want to restrict the bitrate for our use. I’ve taken all bitrate limiting out of the scripts except where needed as well as fixed the audio handling over time to work properly for ios which is now mostly standard. I also add in more support for codecs but that’s simple.

I’m using the defaults for GPU selection since I don’t think it’s going to affect many people and it can be added via the ini file as a post processing option anyway. I’ve defaulted the look ahead to 0 since the earlier cards don’t support anything else and would make compatibility much easier. Planning to add dynamic tuning to both NVENC and QuickSync depending on what hardware the user has available. Many of my changes aren’t published yet as I’ve got more testing to do and got side tracked on something else I’m working on.

I’m using decoding on my copy but it’s not in the release I just made since I want to be able to debug anyone’s problems easier and have less hardware issues to start with. I’m also working on more audio options for level adjustments besides just the RMS normalization but it will a two pass approach (or sampled or scanning the first 5 or 10 minutes of the file). There’s a couple of bugs in ffmpeg that I’m working with to get fixed concerning audio and hardware. I’d rather get them fixed at the source then have to use workarounds.

I’ll definitely take another look at what you’ve done to see how close it is to my current version but last time I checked there were changes you made that broke or modified code and settings that affect CPU or QuickSync functionality. My goal is to support CPU as well as GPU include QuickSync, Nvidia and AMD so the code needs to support them all and not be tied to one GPU.

But like I said I’ll play with your version again after the Holiday to see the plus/minus of using it.

Gotta love these forums and the helpful people here!

Carlo

846

Let me know what is broken for cpu/quicksync, It’s probably just something I overlooked since I don’t use those very often.

edit Was probably when I removed the threads option, thinking that it was useless. I just reverted it, so it should be fine now.

847

I have been itching to try out @cayars converter script but every time I try to download the file it gets to the last bit and times out. never really done much with ftp but thought it was basic but I must be doing something wrong. any help would be much appreciated. Thanks
Steve

848

I’m trying to run the conversion scripts on my PC running Windows 10 and it’s just not working and I don’t understand why. When I run the bat file I am getting the following error.

ANY help would be appreciated! It’s driving my nuts.

849

You didn’t follow the instructions in the text file. You need to import all the modules via PIP.

850

@cayars said:
You didn’t follow the instructions in the text file.

Any thoughts on what would cause the script to continually loop through files? It finished each file successfully, but it looped through them at least 3 times before I caught it. The only things I changed was setting subtitles and artwork to false, and the move directory to blank (I don’t want the files moved).

Here is a copy of the relevant portion of the autoProcess.ini -
[MP4]
ffmpeg = ffmpeg.exe
ffprobe = ffprobe.exe
threads = auto
output_directory =
copy_to =
move_to =
output_extension = mp4
output_format = mp4
delete_original = True
relocate_moov = True
video-codec = h264
video-bitrate =
video-max-width =
h264-max-level = 4.1
use-qsv-decoder-with-encoder = True
ios-audio = True
ios-first-track-only = True
ios-audio-filter = dynaudnorm
max-audio-channels =
audio-codec = ac3,aac,mp3,dts,dca
audio-language = eng
audio-default-language = eng
audio-channel-bitrate = 256
audio-filter =
subtitle-codec = srt
subtitle-language = eng
subtitle-default-language = eng
subtitle-encoding =
fullpathguess = True
convert-mp4 = True
tagfile = False
tag-language = en
download-artwork = False
download-subs = False
embed-subs = False
sub-providers = addic7ed,podnapisi,thesubdb,opensubtitles
permissions = 0777
post-process = False
pix-fmt =
aac_adtstoasc = False
postopts =-preset,slower
preopts =
video-crf = 18

EDIT - realized it was the run.bat script and not the autoProcess.ini file doing it. Can you explain a use-case where I would want the script to restart after the 60 second timer?

851

Are you trying to process the files in place? I believe that’s what you’ve done when you changed the ini setting.

852

@cayars said:
Are you trying to process the files in place? I believe that’s what you’ve done when you changed the ini setting.

Yes, that is what I wanted to happen. I realized it was the run.bat script and not the autoProcess.ini file doing the looping.
Can you explain a use-case where I would want the script to restart after the 60 second timer?

EDIT - sorry, I realize now that the default action of the autoProcess.ini makes this treat the Process folder like a “watch” folder. My mistake for not paying closer attention!

853

Absolutely, most of use have an IN folder and and OUT folder that we use to process files.

So regardless of how you source your files you just drop them into the IN folder and they get processed. So with the batch file and the 60 second approach it will restart itself and check for new files to process. Thus it will constantly process your IN folder.

As you will find out these scripts can be used many different ways once you understand them!

854

Thanks for all the advice, @cayars! Very helpful. :smile:

855

Figured out one more possible use case if it would not be hard to adjust to.
Working Directory: I would like the ability to transcode to a local directory, and then move it to the destination where I got the file from.

Unless I am totally doing it wrong, setting an output directory will do the transcode, but then unless I have a copy/move directory it will leave the file there. And since I want to run it against a tree structure, adjusting the move to each time would be a pita.

-Izz

856

Search the thread for this. It’s in here somewhere.

Personally I wouldn’t do it that way. You could change the IN folder to point to your present media and let it do sections of your library at a time. Then just copy the finished media back This would make it much easier to track what has and what hasn’t been done yet.

857

@cayars
Any idea why tagging would freeze when using nvenc?
It converts the file fine, but when it goes to apply the meta data it just hangs there and never finishes.

I will just think about reworking how I am doing the transcoding… I didn’t want to have to do a re-import of the content into Sonarr, as historically when I do that, it records a 1080p as SD since the file name no longer has the quality and type. So again, not sure I am doing something wrong, just been my experience.

-Izz

858

It’s probably due to one of the import libraries not working correctly.

Personally I don’t use the tagging because I find it messes up a couple every hundred files I process then I have to Plex dance the files to get them fixed. Instead what I do is turn off tagging which then would leave the original tags in place (often worse) but then you can modify the ini to set this:
postopts = -metadata,title=

That tells ffmpeg to erase or set the title to “” or blank. Since I use FileBot to rename my files prior to running them through the scripts this works great. Filename is all Plex needs and the meta data will often just mess things up.

Carlo

859

Ah, the blank would work! My stuff has all been filebot’d as well, or processed ahead of time by CouchPotato/Sonarr.

-Izz

860

This is something I just recently started doing a couple of days ago which is why I’ve never mentioned it before. I might default the scripts to work this way in the next version with instructions to clear that line and turn tags back on if needed.

I think this makes for better Plex recognition and for less conflict in meta-data shown by Plex.

Carlo