Batch Transcoder

Dev Community Apps & Creations
, , ,
1

Greetings folks.. long time listener, first time caller..

I put togther an “Uber” script for library conversion that may help some people. The problem with most “transcoding” scripts is they depend a lot on other infrastructure or lack a lot of checks and verifications that I wanted.. so I built my own. (after trying with Tdarr and not getting to the intelligence I wanted).. I wrote originally like 4 different versions of the script.. one for TV shows, one for Anime, one for movies and one that was a fork for Cygwin under windows.. and it worked.. but still lacked things I wanted.. so I revamped it and generated a new one that ties everything together into a single script.. checks for data integrity, attempts at healing, upscaling, logic for sizes (I’m favoring AV1 with my script with a fallback to x265). and given the size of my library, I am running (testing) it on a AMD Linux Desktop (5080 16GB and a A4500 20GB), A widows laptop with an nvidia 4070 GPU, a Macbook Pro M2 Max, and an Intel Mac with an AMD GPU (that I reformatted to be a Fedora laptop).. The script is not designed to talk to any of them, but it covers their hardware needs.. tries to use hardware encodig then failing any hardware (NVIDIA, AMD GPU, Intel quicksync) then falls back to software encoding.
Has checks for sanity, verification, indentification, organization, and even metadata cleaning).. basically the idea was turn it loose on a directory and it will organize it the way I want, repair/heal lightly damaged mkv files, (upscale where needed), and reformat the images (without deletion) to be either AV1 or x265 whichever is best based on sampling size… (

Anyway, let me know what you think, if you have questions.. or suggestions to make it better. It does depend on a number of tools.. (HandbrakeCLI.. GREAT GREAT tool, ffmpeg, ffprobe, mkvalidator, and a few others.. and will even try to help you get connected (my videos all run from my NAS so it tries to check the SMB/NFS shares and mount if not present but needed.)

2

I have a few thoughts. I recently made one for myself too, and as such my answer is colored by the needs I had for my script. Perhaps you will want to add some of it. I based mine around software encoding. I can use HW, but default is software for quality+filesize. SVT-AV1 is better quality than NVENC AV1. I built it around ab-av1 for VMAF. I want the script to change the encoding settings based on the quality of the original file. Stripping lossless sound for archive purpose is a no-go for me. I put this script through Claude and it seems to be hardcoded for your RTX5080 and A4500. My Intel Arc hw AV1 would not be used. Why upscale files? Why not let the player do that? And you accept files larger than the original. Why, if the goal is to save space? I would reccomend running it through a LLM and look at the suggestions it gives you. For me, it told me this to improve your script: VMAF-targeted encoding (ab-av1 or similar), stream-copying audio by default, dropping the upscale path, and inverting the size policy to require savings rather than tolerate growth.

Sure is fun tinkering with these scripts though. My path was the opposite of yours. I started by making the script, and ended up inside tdarr at the end. I just make tdarr run the script to encode for me.

Your EBML check was pretty nice. I might implement that too.

3

I’m allowing a slightly larger size only because I also want to get rid of older schemes like AVI/ISO/TS/MPEG2/DVIX and others).. But there are guards.. ie: I will accept a larger file only if it doesn’t exceed more than 20% of the original size. (I allow up to 50% as a MAX for upscaled files).. The reason I’m not depending the players to upscale is because not every player can/will upscale (or even upscale it correctly). So I would rather bring resolutions under 720p to 1080p.. so like SD, 480p get upscaled, otherwise we leave it alone)..

I didn’t focus on VMAF-targeted encoding becauseVMAF-targeted encoding isn’t supported by Handbrake.. (I find its output tends to be completely safe for plex and other players).

And while the script does have a lot of references for RTX5080 and A4500 (for one machine).. I wrote it so it checks for anything that can use NVIDIA, Intel Quicksync, AMD VENC.. (for example, I have a mac that has and AMD Radeon GPU that uses AMD VENC, and a another laptop that is a RTX4080 GPU, and other system that uses Intel Quicksync. (I wrote it mostly for my needs so it needed to check for all the potential GPU’s I use). And I re-encode the audio to Opus because I don’t find it looses fidelity in any meaningful way…

On average I’m dropping about 30-70% of space from my original files.. (yes, I loose a little on a few files.. but mostly those are legacy containers/file formats, or files that are low res that I’m bringing up to a standard so that just about any player can use the file and the playing is more predictable).

Yeah, I wanted to add a check, because after a long time.. while there are files that are mostly playable in plex, it will struggle because of structual issues.. so try to fix them with a simple remux, and if that doesn’t work, let me deal with it on a case by case basis.

Finally I let it fall back to x265 because not every file in AV1 tend to be low, vs. x265 sometimes works better.

I might look into VMAF to see if it looks better.. (and I might ditch handbrake over ffmpeg because ffmpeg does have more flexability.. but because it has a LOT of options, it takes a while to tweak the settings to find what’s optimal for my needs.. In the end, what I really needed was a script that would let me encode reasonably intelligently, organize (ensuring the files are arranged and named according to my structure), and attempt to automate my methods and use GPU’s when/where possible.

4

Right now, I’m focusing on encoding files in H.264 with a high bitrate. I’ve set the cutoff at 30 Mbps for now, and I’ll probably tweak it to slightly lower settings once I’m finished. I’m still in the experimental phase. Speed isn’t a big concern—I just let it run and check back in a few weeks.

What players are having issues with low resolutions? Software players on computers or hardware devices?

5

Oh, I put your script through Claude, it gave some pointers. Perhaps you’re already on top of it, but here it is:

Old bugs still unfixed

gawk-only match(,arr) in disc title scanning (macOS silently broken despite the portability banner — they fixed the identical bug in the progress parser and missed these twice now), du -sb on macOS, mv -n organize collision, O(n²) sed -n Np queue reads, and the pipeline producer still runs in a subshell so CONVERT_JOB_TOTAL/skip stats never propagate — “Convert queue finished: no items needed encoding” after encoding items is still reachable.

New problems, and one is serious for TV libraries

  1. The v5.0.7 file cache and folder-done flags are broken by Season subdirectories. Both key on the show folder’s mtime, and the comment even cites the POSIX behavior that defeats it: mtime changes only on direct-child modification. American/Show/Season 2/new-episode.mkv bumps Season 2’s mtime, not Show’s. Cache stays “valid”, done-flag stays “valid” → new episodes added by Sonarr-equivalents are invisible forever unless the user knows to --ignore-done-folders (which doesn’t even invalidate the file cache, only the done flags). For the exact ~40k-episode layout this feature was built for, it trades a 6-hour rescan for silent permanent staleness. Correct fix is keying on max(mtime) across the folder’s subtree’s directories, or caching per-Season-dir instead.
  2. Multipart merge will eat two-part TV episodes. The Part1/Part2 detection runs in every directory including season folders. Show - S01E15 - Part 1.mkv / Part 2.mkv are separate episodes in every TV naming scheme; same codec/res so the compat check passes, and mkvmerge happily concatenates them into one .merged.mkv that then gets encoded as one title. Needs an is_tv_episode/season-dir exclusion.
  3. Merging happens during counting. convert_estimate_scan_total — commented “fast find-only count (no ffprobe)” — now routes through find_convert_videos_under, which triggers multipart detection (ffprobe compat probes) and actual mkvmerge merges before the script has even chosen batch vs pipeline mode. Cached after first run, but the first run’s “estimate” can be hours of I/O.
  4. --clean-junk-apply orphan logic has a data-loss edge: a .merged.mkv whose source parts the user deleted (after verifying the merge) is classified as orphaned junk and deleted — potentially the only remaining copy. The “originals never deleted” guarantee doesn’t cover this file class.

Verdict: genuinely much better — the quality-targeting core went from the worst part to roughly your architecture, and the changelog shows real A/B discipline. But I’d tell them not to run v5.0.8’s folder-done/file-cache on a nested TV library until the mtime keying is fixed, and to gate multipart detection out of episode folders before it merges someone’s two-parter. The audio policy is their call now that it’s demonstrably deliberate; the four-times-flagged awk portability bug suggests the macOS claims should just be dropped from the header instead.

6

Had to take the version offline for a bit… SOO much to fix and change.. because I wanted to make sure everything was solid and stable… The Mac instances are cleaner and better now.. using WSL2 under windows so that the same functionality can be achieved in Windows (without using cygwin).. and of course linux works fine.. as well as linux on Mac Hardware. Basically my current “fleet” is 8 computers ( a combination of Windows 2022 server with WSL2, Windows 11 with WSL2 and a NVIDIA GPU, native Linux with Dual NVIDIA GPU’s, Native OSX and Linux running on OSX hardware.. Some with AMD GPU’s, some with Intel GPUs, some with NVIDIA GPUs, and some with pure CPU. Thus making it both a good cross section of a range of hardware and what I’m doing the encodes on myself.

I added a lot of checks and validations to clean up a lot of potential source issues (like finding “dead” subtitles, length issues, variable encodings, etc…

As always the goal of this script is 2 fold: 1: Unification of a single video container standard (eliminating all the legacy containers) and 2 (MOST IMPORTANT): To reduce space where possible so it will try AV1 and then fall back to X265, if there is geniunely no space that can be preserved without ruining the source, it will leave it as its native encoding, but will remux to MKV.

Basically given the size of my MEdia library (I’m near 400TB of video content), this is going to take a while, so I’m getting any/all my “spare” computers involved with this, hence the “fleet” approach. (I’m looping in a few other gaming laptops that are sitting idle doing nothing, so converting them to Linux). FYI, I’m using a combination of Fedora and Ubuntu for my linux systems and will eventually work on a native windows version of the script so WSL isn’t needed (and thus we gain some performance)