I was tinkering with the idea of a show based tuning file, since I’ve had mixed results on the same station. It’s a lot of overhead to keep tuning them, especially since different providers change things. Difference from OTA to FIOS to Comcast, etc.
@bjs1400 said:
@robertclemensI think I see what you were actually asking. You want to have some list of comskip ini files tailored for specific TV stations from which a recording is recorded from. Then have a script automatically select the appropriate comskip.ini and pass it on to comchap/comcut.
I’ve done some tinkering before and may have a possible solution one could implement.
One way I could see doing this is querying the SQLite database that Plex uses. I’ve found that all recordings are stored in a table and stores information such as which channel the recording was recorded from as well as a bunch of other stuff. What’s nice is that this information is pretty easy to query given a “Grab ID.”
You may have noticed that recordings are stored in a .grab directory with a subdirectory with a seemingly random folder name. This folder name is actually the same name as the Grab ID stored in the database. This way you can retrieve the what station the recording was recorded from.
The problem I see is that, in my testing, there is no record in the database for the DVR recording until AFTER post processing is ran. This would mean that there’s no way of running the query to determine the TV Station a recording was done in a post-processing script. I’m unaware of a way of a good way of getting the Grab ID after post processing either.
It may be possible for a post processing script to fork a new process passing the Grab ID with a delay of some time (5 seconds?). This would allow for the post-processing script to complete and have Plex update the database with the required information. From which the script could run and get the needed information to pick the correct comskip.ini file and pass it along to comchap/comskip/whatever.
I’d be interested to see if anyone actually implements something like this.
^- This. Basically if we had the data to query from which is commonly the station then you could interpret the correct comskip.ini accordingly. The only other way I can think of to it it is to build in logic that if the filename being processed is “The Big Bang Theory” then use the CBS comskip file. But of course you would have to manually and statically define the processing by the postprocessing script to handle this. And this is not the ideal method but would work.
It’s good to know that the grab id is a key field. The only other thing to remember is that you would still have to define the mapping somewhere and if the map didn’t exist to use the default comskip.ini.
But as you noted there are a few hurdles on translating the two sources together although it’s plausible to do so.
@robertclemens said:
@bjs1400 said:
@robertclemensI think I see what you were actually asking. You want to have some list of comskip ini files tailored for specific TV stations from which a recording is recorded from. Then have a script automatically select the appropriate comskip.ini and pass it on to comchap/comcut.
I’ve done some tinkering before and may have a possible solution one could implement.
One way I could see doing this is querying the SQLite database that Plex uses. I’ve found that all recordings are stored in a table and stores information such as which channel the recording was recorded from as well as a bunch of other stuff. What’s nice is that this information is pretty easy to query given a “Grab ID.”
You may have noticed that recordings are stored in a .grab directory with a subdirectory with a seemingly random folder name. This folder name is actually the same name as the Grab ID stored in the database. This way you can retrieve the what station the recording was recorded from.
The problem I see is that, in my testing, there is no record in the database for the DVR recording until AFTER post processing is ran. This would mean that there’s no way of running the query to determine the TV Station a recording was done in a post-processing script. I’m unaware of a way of a good way of getting the Grab ID after post processing either.
It may be possible for a post processing script to fork a new process passing the Grab ID with a delay of some time (5 seconds?). This would allow for the post-processing script to complete and have Plex update the database with the required information. From which the script could run and get the needed information to pick the correct comskip.ini file and pass it along to comchap/comskip/whatever.
I’d be interested to see if anyone actually implements something like this.
^- This. Basically if we had the data to query from which is commonly the station then you could interpret the correct comskip.ini accordingly. The only other way I can think of to it it is to build in logic that if the filename being processed is “The Big Bang Theory” then use the CBS comskip file. But of course you would have to manually and statically define the processing by the postprocessing script to handle this. And this is not the ideal method but would work.
It’s good to know that the grab id is a key field. The only other thing to remember is that you would still have to define the mapping somewhere and if the map didn’t exist to use the default comskip.ini.
But as you noted there are a few hurdles on translating the two sources together although it’s plausible to do so.
I haven’t tried the database yet and not sure how to access it (are the usernames defined somewhere)? If I could periodically or even on a trigger dump a sql query to output a dump to a file containing the show and station then the postprocessing could parse that. Maybe?
Perhaps someone should make a feature request to write to the table prior to post processing is done including a new status of something like “PostProcessing”.
I needed this exact thing for generating nicely formatted emails with metadata retrieved from the Plex database, fanart.tv, and thetvdb. The email would get sent after recordings have finished so I actually remember to watch it. Right now I just send myself the filename of the recording. It works, but would be nice if it were prettier.
@bjs1400 said:
Perhaps someone should make a feature request to write to the table prior to post processing is done including a new status of something like “PostProcessing”.I needed this exact thing for generating nicely formatted emails with metadata retrieved from the Plex database, fanart.tv, and thetvdb. The email would get sent after recordings have finished so I actually remember to watch it. Right now I just send myself the filename of the recording. It works, but would be nice if it were prettier.
I’ve added an issue on github and I have no clue on how to fix that bug. Could you give me some light?
@robertclemens said:
The one thing that surprises me is the lack of comskip.ini files. Everyone always posts to the comskip forum and I find it lacking. At least it is to say that comskip.ini files are scattered at best.
That bothered me for a while too and so I spent a little time looking into it. I scoured the internet for USA-OTA comskip.ini files and then I wrote a small Perl script to extract and compare each of the settings for each of the files. What I found was there were actually very few of the parameters that people were actually modifying. In most cases it boiled down to changing the ‘detect_method’. Most everything else was cosmetic or had very little effect from my test runs. So what I ended up doing was setting the detection method to 107 and using default settings for just about everything else except:
- thread_count=4
- hardware_decode=1
- output_edl=1
- Various ones to turn off all other outputs.
A good way to understand a little more is to download the Windows version of comskip and then use it’s INI editor, which I found to be quite good.
For reference, here is my file which works nearly 100% accurate on everything except for the Big Bang Theory (for some reason):
detect_method=107 ; 1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all validate_silence=1 ; Default, set to 0 to force using this clues if selected above. validate_uniform=1 ; Default, set to 0 to force using this clues (like pure white frames) if blackframe is selected above. validate_scenechange=1 ; Default, set to 0 to force using this clues if selected above. verbose=0 ; show a lot of extra info, level 5 is also OK, set to 0 to disable ticker_tape=0 ; Amount of pixels from bottom to ignore in all processing. Use 120 for tickers, if present cut_on_ar_change=1 ; set to 1 if you want to cut also on aspect ratio changes when logo is present, set to 2 to force cuts on aspect ratio changes. set to 0 to disable max_brightness=60 ; frame not black if any pixels checked are greater than this (scale 0 to 255) test_brightness=40 ; frame not pure black if any pixels checked are greater than this, will check average brightness (scale 0 to 255) max_avg_brightness=25 ; maximum average brightness for a dim frame to be considered black (scale 0 to 255) 0 means autosetting non_uniformity=500 ; Set to 0 to disable cutpoints based on uniform frames max_volume=500 ; any frame with sound volume larger than this will not be regarded as black frame min_silence=12 ; Any deep silence longer than this amount of frames is a possible cutpoint brightness_jump=200 ; New max_commercialbreak=600 ; maximum length in seconds to consider a segment a commercial break min_commercialbreak=25 ; minimum length in seconds to consider a segment a commercial break max_commercial_size=125 ; maximum time in seconds for a single commercial or multiple commercials if no breaks in between min_commercial_size=4 ; mimimum time in seconds for a single commercial min_show_segment_length=125 ; any segment longer than this will be scored towards show. padding=0 ; time in seconds to reduce beginning and end of commercial break remove_before=0 ; amount of seconds of show to be removed before ALL commercials remove_after=0 ; amount of seconds of show to be removed after ALL commercials give_up_logo_search=2000 logo_at_bottom=0 ; Set to 1 to search only for logo at the lower half of the video, do not combine with subtitle setting punish=0 ; Compare to average for sum of 1=brightness, 2=uniform 4=volume, 8=silence, 16=schange, set to 0 to disable punish_threshold=1.3 ; Multiply when amount is above average * punish_modifier=2 ; When above average * threshold multiply score by this value logo_percentile=0.92 ; if more then this amount of logo is found then logo detection will be disabled logo_threshold=0.75 punish_no_logo=1 ; Default, set to 0 to avoid show segments without logo to be scored towards commercial aggressive_logo_rejection=0 connect_blocks_with_logo=1 ; set to 1 if you want successive blocks with logo on the transition to be regarded as connected, set to 0 to disable logo_filter=0 ; set the size of the filter to apply to bad logo detection, 4 seems to be a good value. shrink_logo=5 ; Reduce the duration of the logo with this amount of seconds before_logo=0 ; Set to number of seconds before logo appears comskip should start to search for silence to insert an additional cutpoint after_logo=0 ; set to number of seconds after logo disappears comskip should start to search for silence to insert an additional cutpoint delete_logo_file=1 ; set to 1 if you want comskip to tidy up after finishing delay_logo_search=0 ; delete_show_after_last_commercial=0 ; set to 1 if you want to delete the last block if its a show and after a commercial delete_show_before_or_after_current=0 ; set to 1 if you want to delete the previous and the next show in the recording, this can lead to the deletion of trailers of next show delete_block_after_commercial=0 ;set to max size of block in seconds to be discarded, set to 0 to disable delete_show_before_first_commercial=0 ; New, default 0 ms_audio_delay=5 volume_slip=20 skip_b_frames=0 ; Set to 1 to force Comskip to skip frames for higher processing speed. max_repair_size=200 ; Will repair maximum 200 missing MPEG frames in the timeline, set to 0 to disable repairing for players that don't use PTS. disable_heuristics=4 ; Bit pattern for disabling heuristics, adding 1 disables heristics 1, adding 2 disables heristics 2, adding 4 disables heristics 3, 255 disables all heuristics output_framearray=0 ; create a big excel file for detailed analysis, set to 0 to disable output_data=0 ; create a dump of the user data channel, used for CC and XDS (such as V-Chip info). Can be use together with output_framearray to remote debug CC decoding output_debugwindow=0 output_videoredo=0 output_womble=0 output_mls=0 ; set to 1 if you want MPeg Video Wizard bookmark file output output_cuttermaran=0 output_mpeg2schnitt=0 output_mpgtx=0 output_dvrcut=0 output_zoomplayer_chapter=0 output_zoomplayer_cutlist=0 output_edl=1 output_dvrmstb=0 ; Set to 1 if you're running DVRMS-Toolbox output_edlx=0 output_vcf=0 output_bsplayer=0 output_btv=0 ; set to 1 if you want Beyond TV chapter cutlist output output_projectx=0 ; set to 1 if you want ProjectX cutlist output (Xcl) output_avisynth=0 output_vdr=0 ; set to 1 if you want XBMC to skipping commercials output_demux=0 ; set to 1 if you want comskip to demux the mpeg file while scanning sage_framenumber_bug=0 sage_minute_bug=0 live_tv=0 ; set to 1 if you use parallelprocessing and need the output while recording live_tv_retries=4 ; change to 16 when using live_tv in BTV, used for mpeg PS and TS dvrms_live_tv_retries=300 ; Only used for dvr_ms standoff=0 ; change to 8000000 when using live_tv in BTV cuttermaran_options="cut=\"true\" unattended=\"true\" muxResult=\"false\" snapToCutPoints=\"true\" closeApp=\"true\"" mpeg2schnitt_options="mpeg2schnitt.exe /S /E /R25 /Z %2 %1" avisynth_options="LoadPlugin(\"MPEG2Dec3.dll\") MPEG2Source(\"%s\") " dvrcut_options="dvrcut \"%s.dvr-ms\" \"%s_clean.dvr-ms\" " windowtitle="Comskip - %s" hardware_decode=1 variable_bitrate=0 subtitles=0 maxbright=1 noise_level=5 global_threshold=1.05 lowres=0 play_nice_start=-1 play_nice_end=-1 play_nice_sleep=0 thread_count=4 ar_delta=0.08 added_recording=14 min_commercial_break_at_start_or_end=39 always_keep_first_seconds=0 always_keep_last_seconds=0 length_strict_modifier=3 length_nonstrict_modifier=1.5 combined_length_strict_modifier=2 combined_length_nonstrict_modifier=1.25 ar_wrong_modifier=2 excessive_length_modifier=0.01 dark_block_modifier=0.3 min_schange_modifier=0.5 max_schange_modifier=2 logo_present_modifier=0.01 border=10 logo_max_percentage_of_screen=0.1 ignore_side=0 logo_percentage_threshold=0.25 edge_level_threshold=5 edge_radius=2 edge_weight=10 edge_step=1 logo_fraction=0.4 min_black_frames_for_break=1 cc_commercial_type_modifier=4 cc_wrong_type_modifier=2 cc_correct_type_modifier=0.75 output_plist_cutlist=0 output_default=0 edl_skip_field=0 output_edlp=0 output_ipodchap=0
Maybe your ini file just doesn’t work for terrible TV shows. :))
I had to move back to PlexComskip.py until comcut’s chapter time bug gets fixed. I tried fixing it myself but my bash knowledge is a little rusty.
Just found comchap and comcut - Thank you!
I am running ubuntu 16.01 on a headless server with comskip and ffmpeg installed. Keep getting this error running comcut:
[h264 @ 0x2296d00] gray decoding requested but not enabled at configuration time [h264 @ 0x239e580] gray decoding requested but not enabled at configuration time 0:15:50 - 66696 frames in 45.69 sec(1459.75 fps), 1.00 sec(1108.00 fps), 49% 67175 frames decoded in 46.13 seconds (1456.21 fps) [h264 @ 0x35cb3c0] error while decoding MB 14 22, bytestream -13
Then after what seems like ffmpeg is called, this error shows up 3 times (which may correlate to the number of commericals detected):
Chapter #0:2: start 697.550000, end 1239.520000 [NULL @ 0x1972ac0] start time for stream 0 is not set in estimate_timings_from_pts [mpegts @ 0x196e080] Could not find codec parameters for stream 0 (Video: h264 ([27][0][0][0] / 0x001B), none): unspecified size Consider increasing the value for the 'analyzeduration' and 'probesize' options
The resulting file is aout 85kB while the original is 200MB+
Thoughts on what to do?
Comskip 0.93i.000, made using avcodec
ffmpeg version 2.8.10-0ubuntu0.16.04.1
Changed to:
Comskip 0.81.089, made using ffmpeg
Still not working.
I don’t know specifically for your error, but I can say I had a strange crash with comskip and it ended up being the comskip.ini file that was my issue. I had almost all the detection methods turned on and when I changed it to what I have posted above, it worked.
If that’s not your issue, then sorry I’m not sure what it is.
The comskip.ini was created what I ran it the first time. Followed the instructions and put the one line with the edl in it.
comskip seems to be working as it is identifying the commericals.
Here is where the problem starts:
[mpegts @ 0x12d1f40] H.264 bitstream malformed, no startcode found, use the video bitstream filter 'h264_mp4toannexb' to fix it ('-bsf:v h264_mp4toannexb' option with ffmpeg) av_interleaved_write_frame(): Invalid data found when processing input frame= 1 fps=0.0 q=-1.0 Lsize= 58kB time=00:00:00.96 bitrate= 498.2kbits/s video:11kB audio:52kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Conversion failed! ffmpeg version 2.8.10-0ubuntu0.16.04.1 Copyright (c) 2000-2016 the FFmpeg developers
What did you use to encode to H.264? HDHomeRun encodes to mpeg2, so maybe whatever you are using to encode to H.264 is introducing issues? That’s the only idea I can come up with.
I figured it out. I did not compile ffmpeg. I used the Ubuntu server version. Once I rebuilt the ffmpeg it worked.
Strange. I’ve done test setups on fresh Ubuntu installs and didn’t have any problems. I wonder if you were just missing an added package to handle whatever codec the recording was done with.
@bjs1400 said:
I just made another change to comcut so it no longer uses mktemp for making temp files for each part.
I just installed and ran your latest (2 days ago) on FreeNAS 9.10. Thank you so much for such a good script. With your script, kaashoek’s comskip and Bollar’s instructions I am thrilled!!!
I ran into a problem, however, in comchap because mktemp does not support --suffix. I have a proposed solution:
Replace:
`
tempfile=`mktemp --suffix=."$outextension" "$outdir"/XXXXXXXX`
echo Writing file to temporary file: "$tempfile"
if $ffmpegPath -loglevel error -hide_banner -nostdin -i "$infile" -i "$meta$
mv -f "$tempfile" "$outfile"
echo Saved to: "$outfile"
else
echo Error running ffmpeg: "$infile" 1>&3 2>&4 >&2
exitcode=-1
fi
`
With:
`
tempdir=`mktemp -d "$outdir"/XXXXXXXX`
tempfile="$tempdir"/temp.$outextension
echo Writing file to temporary file: "$tempfile"
if $ffmpegPath -loglevel error -hide_banner -nostdin -i "$infile" -i "$meta$
mv -f "$tempfile" "$outfile"
echo Saved to: "$outfile"
else
echo Error running ffmpeg: "$infile" 1>&3 2>&4 >&2
exitcode=-1
fi
rm -r "$tempdir"
`
I could fork your github and issue a pull request if you like, but it is so small I think by the time you review it, you will have it included (with any necessary mods).
It would probably be better to not use mktemp at all (that’s what I did in comcut). Comchap needs to write to a new file before it replaces the old file with the new. Instead of using mktemp it could easily just use some sort of convention like:
{old_file_name_without_extension}.temp.{target_extension}
I got myself busy with other stuff at the moment. This should be a simple change though. Feel free to submit a pull request if you feel inclined.
bjs1400, thank you for the script. It has been working great. I currently just use the comchap to scan for commercials.
What program do people use to review/tweak the edl files for found commercials? I want to review the markers to make sure they are correct before removing the commercials.
Thanks!
@Reed97123 said:
@robertclemens said:
The one thing that surprises me is the lack of comskip.ini files. Everyone always posts to the comskip forum and I find it lacking. At least it is to say that comskip.ini files are scattered at best.That bothered me for a while too and so I spent a little time looking into it. I scoured the internet for USA-OTA comskip.ini files and then I wrote a small Perl script to extract and compare each of the settings for each of the files. What I found was there were actually very few of the parameters that people were actually modifying. In most cases it boiled down to changing the ‘detect_method’. Most everything else was cosmetic or had very little effect from my test runs. So what I ended up doing was setting the detection method to 107 and using default settings for just about everything else except:
- thread_count=4
- hardware_decode=1
- output_edl=1
- Various ones to turn off all other outputs.
A good way to understand a little more is to download the Windows version of comskip and then use it’s INI editor, which I found to be quite good.
For reference, here is my file which works nearly 100% accurate on everything except for the Big Bang Theory (for some reason):
For BBT I changed the global_threshold to 1.6. If you run the comskipgui from windows you can see that it is scoring it higher at the end of the show because of the logo that is missing. I also told it to score the logo from the bottom of the screen which helped a little too. My conversion seems good now.
I just use a seperate ini for this show.
I need to write a script to filter by show name because eventually I want to run all of the coversion on my linux tub.
I am using MCEBuddy right now which is filtering and applying different comskip.ini’s to the problem children shows.
Mash and BBT are two of them. Having and issue with A-team where the station ID is on the screen the entire time and the program logo is just above it so it is scoreing the station ID and not the program ID. Still working on this one
-Casey
Just an FYI, for anyone who cares:
There was a somewhat large bug with comcut where it would leave out the final part of the recording. I have now fixed that so if your recording ends with non-commercial content, it will now be included in the resulting file. Everything has been updated to github, so make sure to get the latest from there.
Ha! I thought that was due to some wonky comcut.ini file I had. Thanks for fixing!
