Plex DVR with MCEBuddy handling news

General Discussions Plex Features
1

Configuration:
Dual Xeon 2.93ghz x5670 hex core (24 virtual cores total, 12613 passmark)
72 gb ECC RAM
Windows 7
OS and plex drive: 500 gb SSD
Storage drive: SATA 2tb hybrid
Secondary storage drive: 4tb WD MyCloud NAS
Plex Version: 1.5.5.3634
Tuner: HD Homerun Prime w/ latest fw
TV provider: Cablevision, via Cablecard M3
Supplement Plex server: Remote Seedbox account w/Plex and Sickrage

I am interested in simply recording all major news station episodes for my area. So I installed an HD HR Prime on a refurb server and off we go. As time goes on I begin to notice some things not being recorded at all (several different errors). Also, I am finding strange failures in MCEBuddy as well as episodes in my DVR incoming and .grab folders. These files should have moved down the line as the recording is finished and MCEB converts it to a different destination. At times I’ll find “copy 1” appended to the filenames and errors in the log, especially if I manually initiate it in MCEB.

My first attempt was to adjust MCEB to append a simple HH MM to the end of the filename to make sure they are all unique in the end. I ran this for a week or two and watched. I saw pretty much the same thing as before, just slightly less frequent. I looked closely and I think I understand what is happening. I have plex configured to drop all recorded content to an unshared temporary library, say DVR INCOMING. After MCEB intercepts and converts it, it is moved to the shared News library where users can access it. They are converted from .ts to .mp4 with the commercials removed,mostly.

On a normal record (single episode) the flow is as follows:

  1. DVR records first airing to .grab folder in the DVR INCOMING library.
  2. Plex moves the completed file to DVR INCOMING library root.
  3. MCEB converts the file, saves an MP4 to the News library and deletes the .ts version.
  4. Plex indexes the new file in the News library.
  5. Done.

On a NON normal record (single episode, several airings potentially back to back)

  1. DVR records first airing to .grab folder in the DVR INCOMING library.
  2. Plex moves the completed file to DVR INCOMING library root.
  3. MCEB STARTS converting the file.
  4. DVR records next airing to .grab folder in the DVR INCOMING library.
  5. Plex moves the completed file to the DVR INCOMING library root. **problem begins, MCEB isn’t always done with the previous one in time…
  6. Maybe Plex indexes the new file in the News library…
  7. Maybe something didn’t get converted or recorded…
  8. Maybe the DVR failed a recording from the inability to create the file…
  9. Maybe the recording is stuck in the .grab folder.
  10. MCEB completes converting the FIRST file, saves it to the News library and deletes the .ts version.
  11. Plex indexes the new file in the News library.
  12. done?? not really the remaining recordings are stuck or digital dust.

I think in order to handle the news correctly it needs to be addressed in two ways. First, correct the DVR filename timestamp to the scheduled time for that airing opposed to “00 00 00”. This will address the filename collisions. Next, Plex needs to treat “news”, (maybe as a new library type??) a bit differently than shows. Perhaps episode linking, like multi-part movies. Or better yet, possibly in addition and list them as a group opposed to individual episodes with the same episode number. This way you can jump right into a specific airing from the start, or just watch the whole group (group meaning all airings with the same episode number) in sequence automatically as if it was one long episode. The strength here is the ability to reference a single airing as well as the whole group.

It was noticed in the attempts to bypass the issue above that there is a clear bug with the “Limit to airing” feature. Say for example there is a news “airing” on from 4:30am to 5:00am then the same episode but a different airing (NOT THE SAME CONTENT) starting at 5:00am to 7:00am. The “Limit to airing” options supplied in the drop down menu reads 3:30am and 4:00am in contradiction to the schedule. Since none of the airings start at either of those times, the feature is unusable. Nothing will be recorded if one is chosen. The option must stay on Any.

If I am able to resolve these issues I plan on adding a second HD HRP because at peak times I may want to record up to 5 airings at the same time. The spare tuners may also be handy in the future for live tv if the option becomes available…Please! :slight_smile:

To be clear, I think Plex rocks, kicks ass, is the future of personal media management and will likely be apart of my home and mobile media center for many years. :sunglasses:

2

If your shows are not processing before next recording finishes it sounds like you are possibly using an MP4 Normal or High Quality profile in mcebuddy. I found that mcebuddy was often taking 20+ minutes to transcode a 30 minute HD show from ts to mp4 when I was using a normal setting, high quality was often taking 40-50 minutes for a 30 minute HD show. I have switched to MP4 Unprocessed. Recordings are now processed much faster:
1-3 minutes for a 30 minute SD show
2-4 minutes for a 60 minute SD show
4-6 minutes for a 30 minute HD show
8-9 minutes for a 60 minute HD show

Downside is a larger file size.

I also have a DVR library where Plex records to. I use Post Processing to process the recording with mcebuddy instead of having mcebuddy monitor a folder.

The only time I typically see a recording stuck in the grab folder is if there was a recording issue. In those cases normally files have a 0 byte file size.

3

I have noticed something similar. I started with MCEBuddy doing the conversion in a post processing script. The task in MCEBuddy was configured to use its mp4 normal profile which would often take a while to convert. The result of that is that plex would record any content not converted and put in the library again if the post processing didn’t complete before the next showing.

Something else to think about is that if you are using the MCEBuddy monitoring instead of using a post processing script you are depending on MCEBuddy finding the show details in its online resources instead of using the gracenote data plex has.

There is also an issue with what @johnm_ColaSC stated above. I have found in many cases that simply putting the video in a MP4 file causes transcoding to occur. I don’t think it should, but it is something I have seen it. I would suggest using the mp4 fast conversion profile.

4

@johnm_ColaSC

Absolutly correct, 2 pass high quality. These recordings are sometimes played on a large screen and the lower quality sucks for that. But re-encoding time isn’t a concern. Its fine if a show takes up to an hour or two before becoming available. Reliability of a scheduled airing automatically making it to a good mp4 in the library is my focus here. I’ll work on speeding it up after it works correctly.

As for post-processing vs monitored folder, weighing out the pros/cons I don’t see an advantage of changing to it. The monitored folder will easily allow me to distribute the conversion load to other computers later via shared network folders. If there were to be an error or something, folder monitoring can pick up where it left off automatically opposed to just failing requiring user intervention. Also, post processing applies to ALL recorded content the same, so having different items handled differently will have to be done via a script of some type. Between setup and changes and digging into command line parameters, the time needed for that is not realistic.

This all loops back around to the simple point that as of now, on a single Plex server, I am unable to successfully record all daily airings of a multi-part news episode to .ts files. I don’t think It can be done even if I had several plex servers because the limit to airing feature is broken.

Side note about faster conversions:
I am waiting for proper GPU support for the encoding process. I just hope they supplement the CPU encoding instead of just offloading to the GPU completely. GPU alone would be a step backwards. If it wasn’t for the bandwidth needed and the cost, I would just use Amazon services and convert everything in minutes with a monster virtual computer. But the file transfer back and forth would kill my broadband speed.

5

@mavrrick said:
I have noticed something similar. I started with MCEBuddy doing the conversion in a post processing script. The task in MCEBuddy was configured to use its mp4 normal profile which would often take a while to convert. The result of that is that plex would record any content not converted and put in the library again if the post processing didn’t complete before the next showing.

Something else to think about is that if you are using the MCEBuddy monitoring instead of using a post processing script you are depending on MCEBuddy finding the show details in its online resources instead of using the gracenote data plex has.

There is also an issue with what @johnm_ColaSC stated above. I have found in many cases that simply putting the video in a MP4 file causes transcoding to occur. I don’t think it should, but it is something I have seen it. I would suggest using the mp4 fast conversion profile.

I found it better to have the DVR record to a “temporary library” that does’t matter what is or isn’t indexed because its not used by the users. After MCEB is done with it it will then be moved to the actual library for the users to access. This will avoid ghost versions of the origonal .ts in the actual library.

As far as the show details, there really are none, or very little. This is news, not I Love Lucy. These news shows are hardly supported in gracenote/tvdb or whatever. So as long as the show has been initially set up with the proper art, it just needs to add the airings to the existing group. Until there is a news show db, main cover art and random screen shots of each airing is all we get. Besides, i was under the impression that Plex will use the filename of the mp4 and attempt to get the show data at time of being added so internal tags are not that important, its all in the filename.

6

My point was simply that MCEBuddy is dependent on a few things to identify the content. It sources data from a few places and if they don’t have info for the shown MCEBuddy won’t match it well. MCEBUDDY uses community driven sources as well so an even greater chance of incorrect matches.

At least gracenote tells plex what it is recording at the time it happens.

7

@GroupMaster mp4 unprocessed does not mean lower quality. It literally just converts from ts to mp4 with no quality changes as listed in the description from mcebuddy:
Very fast but limited functionality. Use this profile if the original video has H.264/MPEG video and you want to remove the commercials and convert the file format to MP4 (e.g. WTV to MP4) without any additional processing (deinterlace, resizing, volume, cropping etc). It will copy the original video and audio and put it into a MP4 format unaltered in quality.

MP4 High Quality description:
WARNING: Slow, high profile, high quality 2 pass MP4 (H.264/AAC) conversion. This can be helpful with low quality source videos but will not make much difference for HD or high quality source videos. Takes the most time but produces the best results.

If your news recorded is in HD you are not getting any large benefit from a High Quality conversion.

8

Interesting. My .ts files are mpeg-2. Good call. Thanks.

9

@mavrrick said:
My point was simply that MCEBuddy is dependent on a few things to identify the content. It sources data from a few places and if they don’t have info for the shown MCEBuddy won’t match it well. MCEBUDDY uses community driven sources as well so an even greater chance of incorrect matches.

At least gracenote tells plex what it is recording at the time it happens.

I think I may be indirectly doing that. Plex is what does the recording and file name creation. So Plex’s resources are used to establish the filename. The filename is what I use in the end when adding to the library in its final version. The gracenote data is retained no matter the conversion type as long as the filename is intact. Is this correct? I honestly wasn’t aware that MCEB even needed to mess with identifing it.

10

@johnm_ColaSC

I looked back at our conversation an I see what you are saying. Using the unprocessed is fine (and I will) but I can’t bank on IF one is done in time. One news provider on saturday morning has 13 airings, some back to back and nonstop along with any other shows recording as well. Depending on other tasks the computer is handling (at higher priority) the processing rate varies. So a backlog is very possible. Even with quick 30 min airings. And thanks again for the conversion tip. :star:

11

@GroupMaster It is my understanding that any file manipulation after Plex finished the recording will cause the Gracenote metadata to be lost. I have even opened another thread trying to get clarification on that.

The MP4 Unprocessed profile will just change the container and can create issue with playback. My server was transcoding almost everytime i did playback with that profile doing the conversion. I have switched to MP4 Fast becuase it does single pass conversion and it is very unlikely you will see any loss in quality. It doesn’t take hardly any time at all to do the conversion either. My system which is far less powerful then yours can convert a 30 min show with commercial identification in about 20 min. With your machine you should be able to do it much faster. I would suggest you use a donator version of COMSKIP to get the best performance from it, and if you want to speed up encoding you can enable hardware encoding in MCEBUDDY with a paid verison. I am not sure you will see a huge improvment with hardware encoding though since you have a pretty robust CPU

12

@mavrrick ,

I have started a full days run as unprocessed and I am seeing a very big difference in speed. When I looked at the filesizes I did the math and it means that I will have to go over my budget for more space. I am absolutly going to try a single pass encode and test it on a big screen. I am honestly considering staying with the two pass for several reasons. One is HDD space. Second is quality, which would make me like the unprocessed version but that brings us back to number one. Lastly is device compatibility. If I can have a single copy (no optimized versions) playback on most devices without transcode, I can have more users OR have more cpu for the other tasks of the server. But, 2 pass also burns up alot of cpu defeating that anyway. I just need to find the balance or compromise that suits my needs or bite the bullet and double my HDD space. But… if I continue with 2 pass, I can easily restructure the DVR to share the 2 pass conversion with other computers putting me back in the game and easily scale to my needs using one or more barebone used PCs (like I said, budget lol). This sounds like a better design. I wonder how long a Rasberry Pi would take to do a 30 min vid on 2 pass. Lol, they are a great player!

Speaking of other server roles, I have an IP cam that pushes motion detected vids to my Plex cam library. Works like a charm as long as the motion detection is set up correctly. A very easy set up using ftp, so remote cams can be added just as easily. Plex, if you are listening, all I want for christmas (or before) is for a way to specify what timeframe to use for the thumbnails of an entire library folder. This way all the thumbnails are of the triggering event in each video. :blush:

13

The documentation for MCEBuddy stated that 2-Pass encoding is only really beneficial if you have poor quality source material. If the news is from HD channels single pass should be good.

Just for giggles can get the Media details of a file that isn’t processed my MCEBuddy and still in a .ts container. I am curious of the Bitrate and codec being used.

14

@mavrrick, easy enough.
Media

Video Resolution 720p
Duration 2:00:00
Bitrate 13897 kbps
Width 1280
Height 720
Aspect Ratio 1.78
Container MPEGTS
Video Frame Rate 60p
Video Profile main
Part

Duration 2:00:00
File Good Day New York (2007) - 2017-05-08 00 00 00 - Episode 05-08.ts
Size 11.65 GB
Container MPEGTS
Packet Length 188
Video Profile main
Codec MPEG2VIDEO
Bitrate 13257 kbps
Bit Depth 8
Chroma Subsampling 4:2:0
Color Range tv
Frame Rate 59.94 fps
Height 720
Level 4
Profile main
Ref Frames 1
Scan Type interlaced
Stream Identifier 288
Width 1280
Codec AC3
Channels 5.1
Bitrate 448 kbps
Language English
Audio Channel Layout 5.1(side)
Sampling Rate 48000 Hz
Stream Identifier 289
Codec AC3
Channels Stereo
Bitrate 192 kbps
Language Español
Audio Channel Layout stereo
Sampling Rate 48000 Hz
Stream Identifier 290

15

With those media details a few things become obvious. First the Media details show the codec is mpeg2. Converting to h.264/acc will reduce the size to about 1/4 the size with the same video quality as the mpeg. If you have the time to go to h.265 it will be between 1/2 and 1/4 the size. But h.265 is much more cpu intensive to encode.

From a compatiability perspective you can’t beat h.264 with acc audio. Pretty much everything will direct play that. If you use the mp4 unprocessed profile you simply change the container and not the codec of the video part of the file. That means the video is still encoded with mpeg2 with it’s old inefficient methods which is the reason for the very larger files. I have also found several occurances were client devices require transcoding because they seem to not like having a mpeg encoded file in a .mp4 container. Technically in don’t think it should be an issue, but regardless of that the clients want to assume mp4 means h.264/AVC video. So that means you are actually better off leaving it in .ts container so that there isn’t a codec/container mismatch.

From my experience with AVC encoded video you can easily have a good 720p video quality at around 6mbps Bitrate. That is good because the Bitrate recorded is much higher so should provide good results in the conversion on 1pass encoding.

One more question. Where are these coming from. Is it from cable or from a atsc OTA tuner?

16

I posted this on another thread, but it seems relevant - I record a lot of news as well.

I gave up on the post-processing scripts. Instead, I have MCEBuddy watch the folders where the recordings are saved. When a new recording comes in, it grabs the file, processes it, and replaces it in the same spot. Works great.

A couple tricks I’ve found along the way:
Be sure to set the DVR to remux (to MKV) and then chose MKV profiles in MCEBuddy with no renaming. That way Plex thinks it’s the same file and preserves the metadata.

Secondly, you can set MCEBuddy to only run a given profile on certain locations - I have two sets of locations (TV with commercials, TV without commercials) and another corresponding pair for movies. Since I know by channel if a given recording will have commercials or not, I chose where the recording is stored, and based on that, MCEBuddy pulls commercials out or not. It works great. For no commercials, I use MKV High and for commercial removal, I use MKV Fast in MCEBuddy.

17

So can you confirm that as long as the naming is the same the meta data isn’t lost. I have asked that question and the only thing I could find pointed to that not being the case.

18

Yes - the metadata is preserved. In the configuration above, the filename doesn’t change - only the codec. If you want to play the transcoded file immediately, you need to have Plex refresh it, otherwise the scheduled task picks up that it’s shifted from MPEG2 to MP4 in a MKV container.

19

@mavrrick said:
With those media details a few things become obvious. First the Media details show the codec is mpeg2. Converting to h.264/acc will reduce the size to about 1/4 the size with the same video quality as the mpeg. If you have the time to go to h.265 it will be between 1/2 and 1/4 the size. But h.265 is much more cpu intensive to encode.

From a compatiability perspective you can’t beat h.264 with acc audio. Pretty much everything will direct play that. If you use the mp4 unprocessed profile you simply change the container and not the codec of the video part of the file. That means the video is still encoded with mpeg2 with it’s old inefficient methods which is the reason for the very larger files. I have also found several occurances were client devices require transcoding because they seem to not like having a mpeg encoded file in a .mp4 container. Technically in don’t think it should be an issue, but regardless of that the clients want to assume mp4 means h.264/AVC video. So that means you are actually better off leaving it in .ts container so that there isn’t a codec/container mismatch.

From my experience with AVC encoded video you can easily have a good 720p video quality at around 6mbps Bitrate. That is good because the Bitrate recorded is much higher so should provide good results in the conversion on 1pass encoding.

One more question. Where are these coming from. Is it from cable or from a atsc OTA tuner?

I am using the Silicon Dust HD Homerun Prime, with conversion off. I had poor outcome on the 1st computer i tried anything other then off. Maybe things have changed since… (Cablecard driven)

20

@WCTschumy said:
I posted this on another thread, but it seems relevant - I record a lot of news as well.

I gave up on the post-processing scripts. Instead, I have MCEBuddy watch the folders where the recordings are saved. When a new recording comes in, it grabs the file, processes it, and replaces it in the same spot. Works great.

A couple tricks I’ve found along the way:
Be sure to set the DVR to remux (to MKV) and then chose MKV profiles in MCEBuddy with no renaming. That way Plex thinks it’s the same file and preserves the metadata.

Secondly, you can set MCEBuddy to only run a given profile on certain locations - I have two sets of locations (TV with commercials, TV without commercials) and another corresponding pair for movies. Since I know by channel if a given recording will have commercials or not, I chose where the recording is stored, and based on that, MCEBuddy pulls commercials out or not. It works great. For no commercials, I use MKV High and for commercial removal, I use MKV Fast in MCEBuddy.

.-.-.-.-.-.-.-…-…–.-…-…----…-.-…-.-.

Yes, I plan on using different profiles eventually but as of now, all news has commercials and I don’t think I want different formats in the end.

I read somewhere that a decent amount of clients may have issues with the .ts format and force transcoding. Again why the 2 pass mp4 seems best, it’s just a CPU strain that I should be able to manage. Side note, I may in the future be looking at up to 6 recordings at the same time during peak broadcast hours, so ■■■■■ gunna be cranking. The actual .ts recording seems to be very low overhead, so the hard nut to crack is the post-encoding process. But with all the input here I think its going to be sweet in the end.

I just did a reinstall of windows on my desktop with a new SSD and already have it set up to assist with the encoding. Quick and easy and if the PC has to be rebooted or is offline for a little it will automatuically pick up and continue where it left off instead of fighting with a script. The only script I may end up doing is to coordinate the remote encoding PCs so it is more FIFO to prevent random hangups or out of order processing. I am keeping in mind that in the end, i want the episodes ready as soon after the broadcast as possible.

Has anyone had he issue of MCEB failing to delete the .ts after a successful encoding? I have seen this a few times and I think it may be at time of deletion Plex is indexing it… This is my best guess without looking at it closely.