Yes I did in fact add the 1TB Samsung Pro SSD to my system and it was like a night and day difference in speed of my system.
I couldn’t imagine running my system without it these days.
@vanstinator said:
@neebski said:
@cayars said:
I’ve decided it’s time to add a 1 TB SSD drive to my system exclusively for the MetaData and Transcoding folders.I am considering doing this with my 245GB of Plex Metadata, did you notice a big difference?
A bunch of us have noticed a huge difference when serving the metadata off of an SSD. I made the move a few months ago and haven’t looked back
Appreciate it, been considering it for a while now. I’ll have to make the purchase.
What a monster thread! Could someone help me, perhaps cayars himself?
I want to batch transcode every film in my library that would be above 4mbps 720p to a separate library that I would then share with my remote users. I just don’t have the upload speed to manage them anymore. What would be the sql query that would then output the appropriate text file to run ffmpeg against? Ideally I would like to run this directly on my server which is an Openmediavault (Debian) box.
Thanks!!
What about 1080p 4mb? Is your goal to limit by size or mb?
You could use the OM feature to make a 4Mbps version, put into a different folder (must be in a folder in the original library) and then make a new library pointing to that folder. As an OM version gets made, the OM library is updated, as well as the original library.
This is a lot more disk use, but it worked OK in my testing
@MikeG6.5 said:
You could use the OM feature to make a 4Mbps version, put into a different folder (must be in a folder in the original library) and then make a new library pointing to that folder. As an OM version gets made, the OM library is updated, as well as the original library.This is a lot more disk use, but it worked OK in my testing
Problem with this is that if the file doesn’t need converting it will never show up in the 2nd bitrate limited library.
To solve this problem (if it were me) I’d setup a batch file/sql similar to what I’ve shown earlier in the thread (recommended/seasonal) to search for files that don’t need the OM. So this way you could have a “bitrate limited” library for sharing that would have the best of both worlds. Real OM versions for files that are over X size and hard links for those that aren’t over the limit.
Wouldn’t take a lot to get this working.
Carlo
@cayars said:
1 - Install SQLITE32 - Put the following in a text file called transcode.txt and put this in the same directory as the sqlite files.
—everything after this line----
.open ‘C:\PlexData\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.db’
.output Transcode.bat
SELECT ‘c:\python27\Python C:\convert\manual.py -a -i "’ || file || ‘"’ FROM media_parts join media_items
on media_parts.media_item_id=media_items.id
where container=‘mp4’ and optimized_for_streaming <> 1
order by file;
—everything before this line—2A - Modify the first line so it points to your DB file
2B - Modify the third line so it points to your convert directory3A - from command window change to sqlite directory and type
3B - sqlite3 < transcode.txtYou should now have a new file called transcode.bat. Open it up and look at it to make sure it looks good. But you should be able to run this.
As a precation I would remove all lines from this transcode.bat file but the first line (so it can only process one file). Make a backup of that directory and test.
This way if something goes wrong you don’t lose anything.I didn’t get a chance to look this over as I’m at work and just grabbed it and posted this on a tablet. Let me know if you have any problems or if it’s not quite working and I’ll help adapt it to your system. But as you can see in the WHERE clause this is just looking for MP4 that aren’t web optimized. It’s not looking for AAC audio or anything like that but we can make it so. Let me know what you need and we’ll tweek it. However as a first run if it isn’t web optimized it needs to get processed anyway. If you know SQL you can also do a check for anything not MP4 and run them too.
Just let me know what you need,
Carlo
Hello Carlo, etc.
I have a question. I want to create a batch file for all films that are 720p or higher that I can then use to transcode all of them to 720p 3mbps. My goal is to create a separate remote library. What exactly would that query look like? I just can’t get it to output correctly. The closest I have found is this query which I found on reddit but it returns titles, not filenames:
SELECT metadata_items.title, media_items.width
FROM media_items
INNER JOIN metadata_items
ON media_items.metadata_item_id=metadata_items.id
WHERE media_items.width>1000 AND metadata_items.metadata_type=1
Please help if you can anyone. THANKS!!
If you are talking about "normal “720p vs 1080p” then it’s the height of the video and not the width.
You could try something similar to this:
select ‘C:\Convert\ffmpeg -i "’ || file || ‘" -c:v libx264 -vf scale=“trunc(oh*a/2)*2:720” -pix_fmt:0 yuv420p -b:v 3000k -maxrate 3000k -bufsize 6000k -profile:v main -level 4.0 -preset veryslow -crf 18 -c:v libx264 -acodec copy -c:s copy -map 0 -movflags faststart "C:\convert\done’ ||
replace(file, rtrim(file, replace(file, ‘’, ‘’ ) ), ‘’) || ‘"’
FROM media_parts join media_items
on media_parts.media_item_id=media_items.id
where media_items.height>=720
That should get you in the ball park. I just wrote that but didn’t try it so it might need a tweek here or there.
If you want the highest quality encodes and you aren’t worried about time I’d suggest modifying it to use two pass encoding.
This also should copy all the audio and subtitle tracks. If you don’t want all of these copied then modify as needed. You can also set this up to only create an AAC audio track discarding other tracks.
Do me a favor and post back your results after you fine tune it. I’m sure someone else will find it useful.
What I would suggest it to give what I wrote a shot and see if it errors out or works. You might need to remove or replace parts of the ffmpeg command line to achieve what you are after. But once you find your “sweet spot” for the command line you should be able to easily modify the script above to match it.
Carlo
Hi I have a quick question regarding your Convert scripts!
I have scanned the posts since the initial thread but can’t see an answer sorry if I have missed it!
Is it possible to set an output directory but also include a dynamic Show name / Season folder?
e.g when output set to X:\media store\TV Shows
Processing file C:\SRProcessed\showname\Season 1\Showname.S01E07.Extra info.mp4
X:\media store\TV Shows\Showname.S01E07.Extra Info.mp4 created.
I would really like to get this in 1 step to :
X:\media store\TV Shows\showname\Season 1\Showname.S01E07.Extra Info.mp4
Currently no. I’ve thought about doing this but different people prefer to name their files their own way or with a specific tool.
There are also tools such as filebot that you can script to do renaming and file moving as well.
Ok thanks, but a shame!
My C drive area is the dumping ground from Sickrage and the files are already named correctly and in the full plex friendly folder structure at that point!
The X drive area is the top end of the NAS storage that holds all 30,000 + episodes so it would be great if there was an option to process and move in 1 go, I used to do this with a watch folder mod of handbrake but this broke a while ago unfortunately!
I ill have a look at filebot to see if I can script something there!
Moving and naming all sorted with filebot scripts - thanks for the suggestion.
I do have 1 last question regarding the actual scripts :
I been doing some testing and have set the variable : video-bitrate = 2000
I run a test across a file that is 6,472 Kbps, the option seems to be accepted in the logs / and the running screen but the output file has a bitrate of 4,121 nowhere near the 2000 requested max?
@cayars said:
Here is a set of scripts that you can use to automate mp4 creation of files.I’d recommend running your files through FileBot first to rename them correctly before running them through this script. It can pull SRT files so everything ends up with proper names this way.
The only pre-requisite is to have python 2.7.9 installed. Python Release Python 2.7.9 | Python.org
(I’m running 64bit version for windows installed to C:\Python27)
This is a modified version of sickbeard_mp4_automator with modifications by me. I pulled out all the integration features to make setup much easier. I also modified it to produce higher quality MP4 files that will direct play better with Plex.
Download convert.zip from ftp://ayars.tv
username and password are both guest
Unzip to C: drive. You will end up with C:\Convert
There are two directories (can be changed) inside this folder that are used for the media
Process is where you put any files you want to convert
Done is where the files will end up after processing
You can run this multiple ways but the easiest is to use the Run.bat file.
If you edit this file you will notice it’s one line that reads: c:\python27\python manual.py -a -i c:\convert\process
If you install python to a different folder then change the python directory at the beginning of this line.
If you want to process files from a different location then c:\convert\process then just change this in the batch file.
If you want to change the final location of where the final files are stored then edit autoProcess.ini file and change the output_directory setting.
That’s pretty much it.
Put files in the process directory and run the batch file
What this will do:
It will process each and every file in the “process” directory.
It will remux files that are h.264 and will transcode files that aren’t h.264 video
It will pull out any english subtitles and create SRT files from them. If you want to pull out other languages then modify the subtitle-language setting in the ini file. Multiple languages are supported by commas
The completed file will not have subtitles (we pulled them out)
It will clear all tags in the mp4 file so Plex doesn’t pick up and use these tags instead of the proper meta-data
It will create an AAC track as the first track upto 256K in size.
It will remove all audio tracks not in english. If you want other languages included modify audio-language setting (ini file) and include any languages you want kept in the MP4 file. Multiple languages are supported by commas
When it transcodes it wil use an h.264 profile of high; a level of 4.0; crf setting of 20. These are basically the same as Handbrake High Profile settings with a bit more refinement for Plex use that allows it to direct play more often.
In all cases the final MP4 file has the MOOV ATOM at the beginning of the file (ie web optimized)
Advanced setting:
You can limit the total bitrate for the video. If for example you use a client that has an upper limit of 12000 KB you can set the option video-bitrate option to something like 11000 which gives you a bit of headroom for a couple of audio tracks.
There is more that can be done with this set of scripts but lets start off with these.
Let me know what you guys think,
Carlo
PS I’ll show you how later to pull info directly from the Plex database to know what files need converting depending on your criteria.
Hello Carlo,
First, let me say a big THANK YOU for this thread. It is a true treasure.
I’ve been reading the whole thing over the last couple of days and thinking how hard would it be to modify your solution to work under Mac OS X 10.11, which comes with Python 2.7.10 pre installed.
I downloaded Convert.zip from your FTP server and modified only the following lines in autoProcess.ini:
[MP4]
ffmpeg = ffmpeg
ffprobe = ffprobe
output_directory = ~/Desktop/Convert/done
I renamed run.bat to run.sh, modified it to read as follows and made it executable with chmod +x run.sh:
python manual.py -a -i ~/Desktop/Convert/process
Finally, I replaced ffmpeg.exe, ffprobe.exe and ffplay.exe with corresponding files downloaded from ffmpeg.org, in order to have the Mac OS X compatible versions (I got version 3.0 release files, BTW).
When I ran the above mentioned script, I got the following output, which I cannot decipher, since I have no knowledge of Python myself. My best guess is that Python is checking the encoding of something in my environment, but I have no idea how to fix this.
Cesars-Mac-mini:Convert cesarvog$ ./run.sh
Traceback (most recent call last):
File “manual.py”, line 14, in
from mkvtomp4 import MkvtoMp4
File “/Users/cesarvog/Desktop/Convert/mkvtomp4.py”, line 8, in
from converter import Converter
File “/Users/cesarvog/Desktop/Convert/converter/init.py”, line 7, in
from converter.ffmpeg import FFMpeg, FFMpegError, FFMpegConvertError
File “/Users/cesarvog/Desktop/Convert/converter/ffmpeg.py”, line 13, in
console_encoding = locale.getdefaultlocale()[1] or ‘UTF-8’
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py”, line 543, in getdefaultlocale
return _parse_localename(localename)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/locale.py”, line 475, in _parse_localename
raise ValueError, ‘unknown locale: %s’ % localename
ValueError: unknown locale: UTF-8
Your help would be much appreciated.
Thanks once again for providing so much high quality information on your thread.
Cesar
@garypower yep, currently broken.
I’ll try and take a look at it this weekend when I have a couple of free hours.
@cesarvog I don’t have a MAC so this may be a bit tougher than usual for me to help diagnose/fix. I’m sure we’ll get it working. Hang on a couple of days and I’ll see what I can figure out.
Thanks for the kinds words on the thread!
Carlo
@cayars said:
@cesarvog I don’t have a MAC so this may be a bit tougher than usual for me to help diagnose/fix. I’m sure we’ll get it working. Hang on a couple of days and I’ll see what I can figure out.Thanks for the kinds words on the thread!
Carlo
Carlo,
I googled the error and found out I had to create a hidden file to declare some environment variables. (see: Mac OS X: ValueError: unknown locale: UTF-8 in Python (Example))
Once that was corrected, I started getting the following error:
Cesars-Mac-mini:Convert cesarvog$ ./run.sh
Manual processor started.
64bit Python.
Processing file /Users/cesarvog/Desktop/Convert/process/Bound for Jail.avi
An unexpected error occurred, processing of this file has failed
[Errno 2] No such file or directory
I think it has something to do with directory/file parsing, probably something relative to the output directory.
I re-edited autoProcess.ini fourth line to read as follows, which now includes the full output folder path. I think it must be right, because, as you can see above, the script has no trouble reading the input directory and file name.
output_directory = /Users/cesarvog/Desktop/Convert/done
I’m ok with baby steps in trying to solve this 
@cayars said:
@garypower yep, currently broken.I’ll try and take a look at it this weekend when I have a couple of free hours.
Thanks, I though I was going mad, but tried a few files and that setting seems to not be honored!
yea, I don’t think it was done correctly. I will take a look at this as I’ve been wanting to add this functionality so it would work similar to Media Optimizer anyway. It will probably need a slight adjustment here or there for this strict purpose as we may not want to remove the original file.
I was thinking that since we have the original file (AVI, MKV, MP4) the time to do true optimizations is with the original file if at all possible which will give us the best transcode.
Carlo,
I fixed the problem reported in my last post above.
After much a lot of trial and error, I found out I still had to edit the following files, in order to add the FULL PATH to both ‘ffmpeg’ and ‘ffprobe’ OS X binaries:
autoProcess.ini
readSettings.py
mkvtomp4.py
Once the full path to the binaries were correctly set, your scripts started to work fine in OS X.
So, in order to recap everything that was needed to make your set of scripts work in OS X, here is a summary:
- download ffmpeg, ffplay, ffprobe binaries for OS X from ffmpeg.org.
- rename run.bat to run.sh, edit it to have the full path to the Convert/process folder, make it executable, by issuing ‘chmod +x run.sh’ command;
- export environment variables as per https://coderwall.com/p/-k_93g/mac-os-x-valueerror-unknown-locale-utf-8-in-python);
- Edit autoProcess.ini, manual.py, readSettings.py and mkvtomp4.py to get rid of all mentions to Windows executables (ffmpeg.exe, ffprobe.exe, ffplay.exe), by replacing them with the full path to the corresponding OS X binaries downloaded in step1.
Thanks once again for your work and this thread.
Cesar
Very cool you got it all working!
Have you processed any files yet?
What do you think of the results?
Carlo
@cayars said:
Have you processed any files yet?
What do you think of the results?
I have just started processing a few AVI’s, M4Vs and MKVs, but haven’t watched anything yet. Inspection of processed files with MediaInfo appears to show everything is as expected, that is: english language set for audio, additional stereo english AAC track added, soft subtitles removed from container, external srt file created for english language subs and H264 High profile L4.0 codec for video. I suppose processed files will work nicely on all my plex client devices (AppleTV4 with Plex client app, Infuse4 Pro app, MrMc app, Simple X app, as well on the Raspberry Pi 2 with Rasplex 1.0.3)…
On a side note, I thought that, in order to AVOID transcoding from the Plex Server when playing on compatible devices (e.g. AppleTV4), one should have the subs EMBEDDED onto the container file, not removed… Can you please elaborate?
Additionally, how can I modify the script so it creates external subs for ALL soft subtitles it finds inside containers?
Best regards,
Cesar
