I made sure my Linux host was fully patched and then verified Python 2.7 was installed. A upgraded PIP to the latest version, installed FFMPEG, and used my package manager to install python setup_tools. Requests installed fine, but requests[security] crashes every single time. I have no idea what it’s trying to do, but it appears that it’s attempting to compile pieces from source and I had no gcc available. Fixed that, and it’s now complaining about I-have-no-idea-what. On one hand, it appears to be that development libraries for python aren’t installed (and they are). On another hand, it appears that openssl libraries might be missing (they’re installed). The error message is pretty general in response to a command-line that I don’t understand.
I will not move this to a Windows host, so I either need a lot of help or I won’t be able to use any of this fine work.
I use openSUSE and am currently running LEAP 42.2. I’ve been using this distro since it was known as SUSE Professional, and Linux for over 20 years. I cut my teeth on Slackware, so I’m not new to compiling code.
@ember1205 said:
Well, I’m done before I could get started.
I made sure my Linux host was fully patched and then verified Python 2.7 was installed. A upgraded PIP to the latest version, installed FFMPEG, and used my package manager to install python setup_tools. Requests installed fine, but requests[security] crashes every single time. I have no idea what it’s trying to do, but it appears that it’s attempting to compile pieces from source and I had no gcc available. Fixed that, and it’s now complaining about I-have-no-idea-what. On one hand, it appears to be that development libraries for python aren’t installed (and they are). On another hand, it appears that openssl libraries might be missing (they’re installed). The error message is pretty general in response to a command-line that I don’t understand.
I will not move this to a Windows host, so I either need a lot of help or I won’t be able to use any of this fine work.
I use openSUSE and am currently running LEAP 42.2. I’ve been using this distro since it was known as SUSE Professional, and Linux for over 20 years. I cut my teeth on Slackware, so I’m not new to compiling code.
I don’t give up easily…
python-devel (python-dev, depends on distro) or python3-devel / python3-dev as well as openssl-devel / openssl-dev is required for the requests[security] package to install along with gcc or gcc++ (which is what I installed). Should probably add these as pre-requisites to your notes, @cayars .
None of the rest of the PIP pieces gave me any troubles. Now it’s on to testing and experimenting…
@loibi2 said:
Hi,
I tried to use your script Version 2.2 to transcode most of my BD Remuxes to a lower Quality for Archiving (720p oder lower Bitrate)
I tried to just adjust the video-bitrate (8000) or video-max-width (720) in the qsv template and used it. The Resulting File was tiny with a Bitrate below 1Mbit
Any Idea what I did wrong?
@cayars Also tried the CPU-Version of your script - only adjusted the video-bitrate Setting zu 4000.
The Resulting MP4 is only about 1G smaller as the Original and has Bitrate of about 20Mbit
Any Idea why it does not use the Setting?
@loibi2 said:
Hi,
I tried to use your script Version 2.2 to transcode most of my BD Remuxes to a lower Quality for Archiving (720p oder lower Bitrate)
I tried to just adjust the video-bitrate (8000) or video-max-width (720) in the qsv template and used it. The Resulting File was tiny with a Bitrate below 1Mbit
Any Idea what I did wrong?
If using QSV don’t use the bitrate controls as they are meant for CPU based encoding. HW transcoding is no where near as flexible as software based encoding.
@loibi2 said:
Hi,
I tried to use your script Version 2.2 to transcode most of my BD Remuxes to a lower Quality for Archiving (720p oder lower Bitrate)
I tried to just adjust the video-bitrate (8000) or video-max-width (720) in the qsv template and used it. The Resulting File was tiny with a Bitrate below 1Mbit
Any Idea what I did wrong?
@cayars Also tried the CPU-Version of your script - only adjusted the video-bitrate Setting zu 4000.
The Resulting MP4 is only about 1G smaller as the Original and has Bitrate of about 20Mbit
Any Idea why it does not use the Setting?
I have an idea. The older setting for max bitrate has probably been succeeded when I moved the scripts to Quality based encoding. So the setting “video-crf = 18” in the ini file is over riding some other settings so that it can meet the quality setting of 18.
I’ve never been interested in creating such and such bitrate files because I find that kind of dumb. Instead I focus on using however many bits are needed to achieve an outcome of a certain quality. How many bits to use for a particular move with vary from movie to movie and show to show so much it would be hard to universally use it. On the other hand using a CRF (constant rate factor) makes it easy to target a certain quality.
So if you want smaller files across the board and are willing to give up some quality increase the video-crf number. The default if not set is 22 to give you an idea. 18 is what I choose to use because it keeps enough detail that it can be hard at times to tell a pure BR rip from the compressed video.
As an example if you took a 1 minute segment from a Transformers movie where the car/truck transforms into the robot or vice versa there is a LOT of info in those quick scenes. At 18 it’s close to pure looking but at 22 you can see lots of artifacts. So in a nutshell these scripts are about keeping the quality as high as possible while giving us files that are direct playable. But you can certainly adjust the CRF upward to get smaller files. Quick action scenes like those mentioned above will be good for testing. Maybe pull down some high quality trailers to use to test with.
@ember1205 said:
Well, I’m done before I could get started.
I made sure my Linux host was fully patched and then verified Python 2.7 was installed. A upgraded PIP to the latest version, installed FFMPEG, and used my package manager to install python setup_tools. Requests installed fine, but requests[security] crashes every single time. I have no idea what it’s trying to do, but it appears that it’s attempting to compile pieces from source and I had no gcc available. Fixed that, and it’s now complaining about I-have-no-idea-what. On one hand, it appears to be that development libraries for python aren’t installed (and they are). On another hand, it appears that openssl libraries might be missing (they’re installed). The error message is pretty general in response to a command-line that I don’t understand.
I will not move this to a Windows host, so I either need a lot of help or I won’t be able to use any of this fine work.
I use openSUSE and am currently running LEAP 42.2. I’ve been using this distro since it was known as SUSE Professional, and Linux for over 20 years. I cut my teeth on Slackware, so I’m not new to compiling code.
I don’t give up easily…
python-devel (python-dev, depends on distro) or python3-devel / python3-dev as well as openssl-devel / openssl-dev is required for the requests[security] package to install along with gcc or gcc++ (which is what I installed). Should probably add these as pre-requisites to your notes, @cayars .
None of the rest of the PIP pieces gave me any troubles. Now it’s on to testing and experimenting…
Tried to launch the manual.py file and found a few different issues…
The README file says that you can run the manual.py file by typing its name - on Linux, the execute bit needs to be set for this to work and it wasn’t. Also, you need to provide the path to it if it isn’t in your path (which it isn’t and likely won’t be). An alternative is to invoke it with “python /path/to/manual.py” or “python manual.py” if it’s in the current directory.
The README file says that you need to rename autoProcess.ini.sample to autoProcess.ini - there is no autoProcess.ini.sample in the zip file. It appears that it’s now distributed as autoProcess.ini.
There were still a number of dependencies not met for the actual processing, and I suspect that this is because I am not using any of the media managers. I needed to install the A/V CODECs for openSUSE in order to get it to actually run. Maybe the README can be updated to show all of the various dependencies that the scripts have, even those that are met by installing / using other programs?
@cayars - I don’t recall how fluent you are in Linux, so I’ll just blindly offer to help with the doc updates I mentioned above if it’s helpful. My slant would be towards openSUSE because it’s what I use, but I suspect that it would be applicable to a variety of distro’s. I could probably also help with Ubuntu since I can build up a host fairly easily.
I’ll be curious to see how differently the conversions turn out using your scripts versus GUI-based HandBrake on my Mac versus CLI-based HandBrake on my Ubuntu-based content ripping machine (headless).
The zip as is can be pretty easily made to work on Mac and Linux as you observed but it’s geared to make it easy to install on Windows which is the largest source of Plex servers. I assumed if a person was running Linux they would know how to create their own scripts and know how to setup python or to use the windows instructions as as base for Linux translation.
I don’t consider myself a Linux guru by any means but I can follow the instructions provided and get it easily working on Linux. That’s been what I’ve seen from other users over the years with the script as well.
But by all means shoot me a PM with additional institutions you found you needed and I’ll take a look at them and probably add them to the readme file.
PS you are correct on a couple of ini name things which I should update. I changed the names and added a few additional ini files as samples that I posted here in the forums but probably didn’t update in the readme. I can certainly easy enough address those as well.
@Sm0k3d0uT said: @cayars so I guess there is no way to edit the script to only contain the converted AAC audio track in the final file?
Did you see this in the README file pertaining to the .ini file?
" - audio-codec = set your desired audio codecs. May specify multiple comma separated values (ex: ac3, aac). The first value specified will be the default conversion choice when an undesired codec is encountered; any codecs specified here will be remuxed/copied rather than converted."
" - max-audio-channels = Sets a maximum number of audio channels. This may provide an alternative to the iOS audio option, where instead users can simply select the desired output codec and the max number of audio channels without the creation of an additional audio track."
I’ll be curious to see how differently the conversions turn out using your scripts versus GUI-based HandBrake on my Mac versus CLI-based HandBrake on my Ubuntu-based content ripping machine (headless).
Some initial findings:
Initial rip of John Wick using makemkvcon on Linux produced a file of size 26,459,503,936 bytes.
Running that file through Carlo’s scripts produced a file of size 26,132,342,886 bytes. A few hundred megabytes less.
Processing the original file with HandBrake 1.0.7 (GUI) on Mac, compressing, and adding a stereo audio channel produced a file of size 4,305,934,044 bytes. This represents a reduction of about 5/6 or 83%.
Processing the original file with HandBrake 1.0.4 (command line) on Linux, compressing, and (maybe? - I have to test) adding a stereo audio channel produced a file of size 2,482,667,202. This represents a reduction of over 90% in size.
The HandBrake processing was done using the same “preset” (“HQ 1080p30 Surround”) on both platforms.
The scans using HandBrake took three or more hours each to run while the scan using Carlo’s scripts took roughly two hours from start to finish. I have yet to play any of the files to see what sort of picture quality they all have.
@cayars , is there something in the script controls that would afford me the ability to have your processing scripts compress the content? I thought that they were supposed to at least be able to do that, if not do it by default. Since I saw pretty much no change from the lossless MKV container to the converted M4V container (I opted for this as opposed to MP4), I figured I’d ask.
Something to keep in mind, this isn’t a “compression” run per say. It’s not set out to compress files on purpose. It’s purpose is to make the files direct playable. One of the first things it checks is to see if the video portion of the file is already direct playable (h.264 with proper level) and if so it will not (by default) reprocess the video but instead will remux it. This is both much faster and will not cause any degradation in quality.
You can certainly override these settings but by default it sounds like it’s working correctly.
Out of curiosity after you ran the rip through makemkvcon what format was the video in?
makemkvcon rips everything into an MKV container w/ h264-encoded video.
The files are quite large in some cases, and I’d like to bring them down at least some. When you’re talking about 15TB on disk of files, even an average of 10-15% reduction is a considerable amount of disk space overall. Some of this content is ripped from the BluRay disc that’s included with a UHD/HDR disc. So, the largest of displays in the house won’t watch the ripped content and that content can be compressed without anyone complaining (since it will be viewed on smaller sets and/or lower resolution sets).
Any suggestions on settings to modify that will cause a bit of compression using your scripts? Or, should I consider using your scripts to “fix” the content and then pass them through HandBrakeCLI afterward to get them compressed? Your scripts take about 25% less time to run (with default settings for video - no compression) than HandBrakeCLI. Running both would mean a single, large file (like Avatar) could take close to a day to process on my systems if I have ro run it through both.
One simple thing I could think of off the top of my head would be to try changing the “h264-max-level = 4.1” setting to 4.0. By default your large rips should be 4.1 or higher and by limiting it to 4.0 it should cause it to redo the video as well.
So, after about 9 hours, it’s at about 72% complete.
24.7G is the original file, transcoded file so far is 4.4G. I’d say it’s reducing the file size, and I’m guessing it will end up around a 70% reduction.
The machine that’s doing the work does -not- have a GPU (it’s a VM), and has its two CPU’s (two threads, really, which is a single E5620 @ 2.4GHz core from the host machine) and 16G of RAM basically pegged. I might have to up the resources on this server and possibly even shift it to my more powerful primary host, too. I don’t mind building a “supercomputer” to do the re-work so long as I get to shut it back down when it’s done as I can’t permanently allocate it a dozen CPU and 48G of RAM.
The script took roughly 14 hours to complete and reduced the file from 24,719,165,079 bytes to 5,777,697,023 bytes. This represents about a 76% reduction in file size.
I don’t see any change in picture quality. Unfortunately, the movie I ran this test on is set in space where everything is pretty dark and resolution isn’t stellar to begin with. So, I don’t really know what sort of change in resolution might be present, if any. I will be shutting down my virtual machine, upping the processing capabilities available to it, and running more tests (more CPU and RAM should allow it to run faster).
