How to transcode MPEG2 stream?

Dev Community Dev/API Corner
1

I’ve started development on a channel to view live TV from my HD Homerun Prime cablecard box. Silicon Dust recently released beta firmware that makes accessing the MPEG2 steams for each channel using an http GET. The code below allows me to view a TV channel on the Windows since the stream does not need to be transcoded. However on my iPhone I get the error saying that the video is in a format that the iPad does not understand.



In the example below, my HDHomeRun Prime IP address is 192.168.1.113, and I’m attempting to view channel 274



<br />
def MainMenu():<br />
<br />
	oc = ObjectContainer(view_group='InfoList')<br />
<br />
	oc = ObjectContainer(title1="HDHomeRun Prime Player")       <br />
	mo = MediaObject(parts=[PartObject(key=HTTPLiveStreamURL("http://192.168.1.113:5004/auto/v274"))])<br />
	vco = VideoClipObject(title="Play channel 274 stream", url='http://192.168.1.113:5004/auto/v274')<br />
	vco.add(mo)<br />
	oc.add(vco)<br />
<br />
	return oc<br />


I see the following entries in the PMS log when I attempt to view the TV channel on my iPhone:


<br />
Dec 03, 2012 18:46:23:753 [3308] DEBUG - Request: GET /video/test?X-Plex-Client-Capabilities=protocols%3Dhttp-live-streaming%2Chttp-mp4-streaming%2Chttp-streaming-video%2Chttp-streaming-video-720p%2Chttp-mp4-video%2Chttp-mp4-video-720p%3BvideoDecoders%3Dh264%7Bprofile%3Ahigh%26resolution%3A1080%26level%3A41%7D%3BaudioDecoders%3Dmp3%2Caac%7Bbitrate%3A160000%7D&X-Plex-Client-Platform=iOS&X-Plex-Product=Plex%2FiOS&X-Plex-Version=3.0&X-Plex-Token=aHe16y6rsA3AVGnkkVrp&X-Plex-Device-Name=Eric%27s%20iPhone [192.168.1.114:61143] (1 live)<br />
Dec 03, 2012 18:46:23:753 [3308] DEBUG -  * X-Plex-Client-Capabilities => protocols=http-live-streaming,http-mp4-streaming,http-streaming-video,http-streaming-video-720p,http-mp4-video,http-mp4-video-720p;videoDecoders=h264{profile:high&resolution:1080&level:41};audioDecoders=mp3,aac{bitrate:160000}<br />
Dec 03, 2012 18:46:23:754 [3308] DEBUG -  * X-Plex-Client-Platform => iOS<br />
Dec 03, 2012 18:46:23:754 [3308] DEBUG -  * X-Plex-Product => Plex/iOS<br />
Dec 03, 2012 18:46:23:754 [3308] DEBUG -  * X-Plex-Version => 3.0<br />
Dec 03, 2012 18:46:23:754 [3308] DEBUG -  * X-Plex-Token => aHe16y6rsA3AVGnkkVrp<br />
Dec 03, 2012 18:46:23:754 [3308] DEBUG -  * X-Plex-Device-Name => Eric's iPhone<br />
Dec 03, 2012 18:46:23:755 [3308] DEBUG - Plug-in com.plexapp.plugins.test has been used 54 times.<br />
Dec 03, 2012 18:46:23:762 [3308] DEBUG - [com.plexapp.plugins.test] Sending command over HTTP (GET): /video/test<br />
Dec 03, 2012 18:46:23:762 [3308] DEBUG - (Capabilties) Passing down capabilities of 'protocols=http-live-streaming,http-mp4-streaming,http-mp4-video,http-mp4-video-720p,http-streaming-video,http-streaming-video-720p;videoDecoders=h264{profile:high&resolution:1080&level:41};audioDecoders=mp3,aac{bitrate:160000}' to plug-in.<br />
Dec 03, 2012 18:46:23:763 [3308] DEBUG - HTTP requesting to: http://127.0.0.1:51080/video/test<br />
Dec 03, 2012 18:46:24:093 [3308] DEBUG - [com.plexapp.plugins.test] HTTP reply status 200, with 979 bytes of content.<br />
Dec 03, 2012 18:46:26:314 [4160] DEBUG - Request: GET /system/:/services/url/lookup?url=http%3A//192.168.1.113%3A5004/auto/v274&X-Plex-Client-Capabilities=protocols%3Dhttp-live-streaming%2Chttp-mp4-streaming%2Chttp-streaming-video%2Chttp-streaming-video-720p%2Chttp-mp4-video%2Chttp-mp4-video-720p%3BvideoDecoders%3Dh264%7Bprofile%3Ahigh%26resolution%3A1080%26level%3A41%7D%3BaudioDecoders%3Dmp3%2Caac%7Bbitrate%3A160000%7D&X-Plex-Client-Platform=iOS&X-Plex-Product=Plex%2FiOS&X-Plex-Version=3.0&X-Plex-Token=aHe16y6rsA3AVGnkkVrp&X-Plex-Device-Name=Eric%27s%20iPhone [192.168.1.114:61144] (1 live)<br />
Dec 03, 2012 18:46:26:314 [4160] DEBUG -  * url => http://192.168.1.113:5004/auto/v274<br />
Dec 03, 2012 18:46:26:315 [4160] DEBUG -  * X-Plex-Client-Capabilities => protocols=http-live-streaming,http-mp4-streaming,http-streaming-video,http-streaming-video-720p,http-mp4-video,http-mp4-video-720p;videoDecoders=h264{profile:high&resolution:1080&level:41};audioDecoders=mp3,aac{bitrate:160000}<br />
Dec 03, 2012 18:46:26:315 [4160] DEBUG -  * X-Plex-Client-Platform => iOS<br />
Dec 03, 2012 18:46:26:315 [4160] DEBUG -  * X-Plex-Product => Plex/iOS<br />
Dec 03, 2012 18:46:26:315 [4160] DEBUG -  * X-Plex-Version => 3.0<br />
Dec 03, 2012 18:46:26:315 [4160] DEBUG -  * X-Plex-Token => aHe16y6rsA3AVGnkkVrp<br />
Dec 03, 2012 18:46:26:315 [4160] DEBUG -  * X-Plex-Device-Name => Eric's iPhone<br />
Dec 03, 2012 18:46:26:316 [4160] DEBUG - [com.plexapp.system] Sending command over HTTP (GET): /system/:/services/url/lookup?url=http%3A//192.168.1.113%3A5004/auto/v274<br />
Dec 03, 2012 18:46:26:316 [4160] DEBUG - (Capabilties) Passing down capabilities of 'protocols=http-live-streaming,http-mp4-streaming,http-mp4-video,http-mp4-video-720p,http-streaming-video,http-streaming-video-720p;videoDecoders=h264{profile:high&resolution:1080&level:41};audioDecoders=mp3,aac{bitrate:160000}' to plug-in.<br />
Dec 03, 2012 18:46:26:317 [4160] DEBUG - HTTP requesting to: http://127.0.0.1:51066/system/:/services/url/lookup?url=http%3A//192.168.1.113%3A5004/auto/v274<br />
Dec 03, 2012 18:46:26:572 [4160] DEBUG - [com.plexapp.system] HTTP reply status 404, with 0 bytes of content.<br />


I'm guessing it's failing on the call to transcode the stream. Is there anything additional I need to add in the channel code for a live stream so that the PMS transcoder will correctly transcode the stream?
2

what about that 404 error?

3


I should have mentioned that in my earlier post. Any ideas on why this request would result in a 404 error?
4


yes, 404 is the server response code for "resource not found" as i assume you are aware :P

It could be that the request headers based on iOS device cause the HDHomeRun to respond with an error for some reason? I assume you dont get the 404 when using the windows client?
5

It would be a bit of a challenge if I was trying to write a channel without knowing what an http status code was… :slight_smile:



I don’t see the same call to /services/url/lookup when I run the Windows Plex client. From what I’ve found in the documentation, the url lookup is a call to the plugin to retrieve metadata about the video. Why would this not get called when using the Windows client?



I’m using the url service implementation from the HGTV plugin as a starting point, but it will be slow going at first. I’m a C# developer, not a python guy…



–Eric

6

I honestly wouldn’t use the HGTV URL service as a starting point … from what I remember we have to do some not-so-standard things within that one to get at what we’re looking for. If you haven’t already seen it, check out this article on the dev blog which is probably more helpful in terms of URL services:



http://devblog.plexapp.com/2012/05/11/the-power-of-the-url-service/

7

I’ve had issues trying to transcode an MPEG2 stream while I was working on the Tivo Plugin. Its been a while since I looked at it (reallife got in the way), but for me it was down to two issues and I wasn’t sure which one was causing the problems.




  1. The Tivo plugin takes a while to start streaming. My Tivo’s are a bit slow, plus the tivo stream needs to be decoded, so it was sometimes take up to 30 seconds to start streaming.


  2. The transcoder seems to make mulitple requests to the stream url. Since this is a “stream” there really is only one file to send. I’m not sure if there is a configuration I could set to prevent this, but it seems to cause issues.



    #1 doesn’t sound like your issue, but perhaps #2 is what is happening?
8


I think this is definitely the problem.

I've just put the beta firmware on my HDHomeRun Prime (omg tv over an HTTP GET!!) and its awesome.

I've tried to use the generic transcoder to get some transcoding going, and it just craps out. When looking at the tuner status of my HDHR Prime, it has taken the channel I wanted to stream, and tuned it twice. This leads me to believe that, yes, the transcoder is hitting this URL multiple times.

Why would it ever do this? Is it doing this on normal file:// handles? Is it doing this on all normal urls to mp4s etcs?

Is there a way to stop this? What needs to happen is, the transcoder needs to hit the url and stay open, just like the normal video player does. I've confirmed that both XBMC and Plex both play from these urls perfectly. But getting it through the transcoder is where the amazing stuff will happen!!

Anyone's insight on this would be super helpful. This is live tv over HTTP people! This is awesome.

http://imac.local:32400/video/:/transcode/generic.m3u8?format=m3u8&videoCodec=libx264&vpre=video-embedded-h264&videoBitrate=5000&audioCodec=libfaac&apre=audio-embedded-aac&audioBitrate=128&size=640x480&fakeContentLength=2000000000&url=http%3A%2F%2F192.168.1.13%3A5004%2Fauto%2Fv506

Some logs:


<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG - Request: GET /video/:/transcode/generic.m3u8?format=m3u8&videoCodec=libx264&vpre=video-embedded-h264&videoBitrate=5000&audioCodec=libfaac&apre=audio-embedded-aac&audioBitrate=128&size=640x480&fakeContentLength=2000000000&url=http%3A%2F%2F192.168.1.13%3A5004%2Fauto%2Fv506 [192.168.1.11:62870] (6 live)<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * format => m3u8<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * videoCodec => libx264<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * vpre => video-embedded-h264<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * videoBitrate => 5000<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * audioCodec => libfaac<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * apre => audio-embedded-aac<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * audioBitrate => 128<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * size => 640x480<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * fakeContentLength => 2000000000<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * url => http://192.168.1.13:5004/auto/v506<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG - Request range: 0 to 1<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG -  * Cookie: SESSION-GUID=XXXXXXXXXXXXX<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG - Cleaning directory for session XXXXXXXXXXXXX (/var/folders/zc/87byhp_n3pzg_knfx91jv0h40000gn/T/plex-transcode-XXXXXXXXXXXXX-756627f6c02b71fd9c58dc4ec8c9bdc98e11c89e)<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG - Starting a transcode session XXXXXXXXXXXXX at offset -1.0 (stopped=1)<br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG - Job running: /Users/jmaddox/Desktop/Plex Media Server.app/Contents/Resources/Plex Transcoder -i http://192.168.1.13:5004/auto/v506 -async 1 -f m3u8 -vcodec libx264 -vpre /Users/jmaddox/Library/Application Support/Plex Media Server/Plug-ins/System.bundle/Contents/Resources/transcode-presets/video-embedded-h264.ffpreset -b 5000k -acodec libfaac -apre /Users/jmaddox/Library/Application Support/Plex Media Server/Plug-ins/System.bundle/Contents/Resources/transcode-presets/audio-embedded-aac.ffpreset -s 640x480 -threads 0 -re -rebuf 10 - <br />
Jan 10, 2013 09:03:23 [0xb1154000] DEBUG - Started session successfully, now saving it: XXXXXXXXXXXXX<br />
Jan 10, 2013 09:03:23 [0xb11d6000] DEBUG - Started generic transcode output thread<br />
Jan 10, 2013 09:03:23 [0xb12da000] DEBUG - TranscodeOutputStream: input processing thread started at offset 0 for 2000000000 bytes.<br />
Jan 10, 2013 09:03:27 [0xb1154000] DEBUG - [Transcoder] Input #0, mpegts, from 'http://192.168.1.13:5004/auto/v506':<br />
Jan 10, 2013 09:03:27 [0xb1154000] DEBUG - [Transcoder]     Stream #0.0[0x7c0], 95, 1/90000: Video: mpeg2video (Main), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 80000 kb/s<br />
Jan 10, 2013 09:03:27 [0xb0103000] DEBUG - [Transcoder]     Stream #0.1[0x7c1](eng), 96, 1/90000: Audio: ac3, 48000 Hz, 5.0, s16, 384 kb/s<br />
Jan 10, 2013 09:03:27 [0xb0103000] DEBUG - [Transcoder]     Stream #0.2[0x7c2], 97, 1/90000: Data: [6][0][0][0] / 0x0006<br />
Jan 10, 2013 09:03:27 [0xb0103000] DEBUG - [Transcoder]     Stream #0.3[0x7c3], 0, 1/90000: Data: TVG1 / 0x31475654<br />
Jan 10, 2013 09:03:29 [0xb11d6000] DEBUG - Transcoder exited and no more data left in the pipe<br />
Jan 10, 2013 09:03:29 [0xb11d6000] DEBUG - Generic transcode session finished reading output.<br />
Jan 10, 2013 09:03:29 [0xb11d6000] DEBUG - Finished generic transcode output thread<br />
Jan 10, 2013 09:03:29 [0xb11d6000] DEBUG - Notifying stream that data is complete.<br />
Jan 10, 2013 09:03:29 [0xb12da000] DEBUG - TranscodeOutputStream: input processing thread exited after writing 0 bytes.<br />
Jan 10, 2013 09:03:29 [0xb0513000] DEBUG - Transcode output stream reached end of file.<br />
Jan 10, 2013 09:03:29 [0xb0513000] DEBUG - Removed transcode output stream, active count 1 => 0<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Shutting down idle session XXXXXXXXXXXXX (idle time is 180 seconds)<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Stopping transcode session XXXXXXXXXXXXX<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Killing job.<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Signalling job ID 11431 with 9<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Cleaning directory for session XXXXXXXXXXXXX (/var/folders/zc/87byhp_n3pzg_knfx91jv0h40000gn/T/plex-transcode-XXXXXXXXXXXXX-756627f6c02b71fd9c58dc4ec8c9bdc98e11c89e)<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Whacked session, 0 remaining.<br />
Jan 10, 2013 09:03:35 [0xb09a5000] DEBUG - Killed 1 idle sessions, and 0 remain.<br />
9

It turns out that the transcoder does not support live streams at all. So this definitely won’t work until that support exists.

10

Maddox, thanks for picking this up and working on it. My house remodeling project has been taking up all my time and I haven’t had any free coding time.



How did you find out that the transcoder does not support live streams?



–Eric

11

From the horses mouth.

12


I'm not sure if I follow. What detail makes a live stream different from a file, other than the file has a known length?

I'm really interested in making this work... I see so many posts over on SiliconDust Project:Connect having issues with iOS applications not supporting DLNA or HD vs. SD etc. Having Plex act as an in-between could overcome all the MPEG2 issues that most portable devices don't support.

If it is just the incoming protocol that is the issue, a corner-turn service could be created to control and receive the input from the HDHR Prime and make the stream look more like something that Plex will play nice with. Am I barking up the wrong tree?

X
13


That's exactly it. The transcoder just doesn't know how to do it. What's really happening here is, since there's no known length, the transcoder can't write a correct .m3u8 manifest.

With a live stream, the manifest looks completely different than what it looks like in a fixed length file. The fixed length files m3u8 has all the segments written out. With live, it constantly changes. The PMS transcoder just flat out doesn't support this right now.



I'm with you. It's breaking my heart. If only the transcoder supported this, Plex would enable one of the coolest things to happen to live tv in a LONG time. The new firmware is so awesome in that we can just pull the stream via a GET. I had it running in xbmc with some software I wrote, in about 3 minutes. It's fantastic.

But while xbmc decodes mpeg2, you still need to basically be on ethernet for it to stream because its such high bitrate, and of course it doesn't work on mobile devices. Simply piping it through Plex's amazing and simple transcoder would resolve all of this and make it the defacto way to use the HDHomeRun.
14


How do Channel/plugins transcode streams from the Internet without knowing the stream/file length? Like Netflix, Hulu...

So if for certain Plex's transcoder can't work with a "live" stream, then why not use a third party transcoder (ffmpeg) to transcode and provide a stream that looks like what would come for. Netflix, Hulu, YouTube etc. I would love to take on this challenge if there is no other way. This really seems like such a simple missing link, that there must be a political reason that nobody has aided its progress. Maybe being able to stream television from your PC to your iPad at the gym or worst to your neighbor's PC isn't such a good idea...

X
15


Exactly. I finally gave it a go and i've got a pretty great ffmpeg command that will read in from the hdhomerun prime, transcode, and automatically segment into mpeg2-ts segments. HLS.


Here's the command:


<br />
ffmpeg -i http://192.168.1.13:5004/auto/v512 -re -g 250 -keyint_min 25 -bf 0 -me_range 16 -sc_threshold 40 -cmp 256 -coder 0 -trellis 0 -subq 6 -refs 5 -r 25 -c:a libfaac -ab:a 256k -async 1 -ac:a 2 -c:v libx264 -profile baseline -s:v 1280x720 -b:v 3000k -aspect:v 16:9 -map 0 -ar 44100 -vbsf h264_mp4toannexb -flags -global_header -f segment -segment_time 10 -segment_list_flags +live -segment_list_size 5 -segment_list_type m3u8 -segment_list test.m3u8 -segment_format mpegts stream%05d.ts <br />






<br />
require 'rubygems'<br />
require 'sinatra'<br />
require 'open-uri'<br />
<br />
get '/test.m3u8' do<br />
  content_type 'video/MP2T'<br />
  open('test.m3u8').read<br />
end<br />
<br />
get %r{/(stream\d+.ts)} do<br />
  content_type 'application/x-mpegURL'<br />
  open(params[].first).read<br />
end<br />
<br />


You just need to expose the test.m3u8 playlist and the segments via http, and voila. It works. I've got it working right now, its great.

I'm working on using Sinatra to expose the segments and playlist over HTTP, but plan on wrapping it all up in the sinatra app so you can make requests for a channel, and it'll just start streaming, along with the ability to turn it off.

Once that happens, boom, you can just use this for anything. Your phone, ipad, inside plex itself. Live TV everywhere.

I'll post back later when I have more
16

wow, forums are gross.

17

Here’s a gist:



18

Also, make sure you use a very recent build of FFMPEG, or just build it yourself, its not hard. http://ffmpeg.org/trac/ffmpeg/wiki/MacOSXCompilationGuide



I was using a build from Nov 2012 and it was really buggy. I’m pretty sure the segmenting stuff was still pretty new then. I built from HEAD tonight and its running great. All my bugs went away.

19

This topic has been kept up to date on the silicondust forums. I'm not going to cross post anymore, so if you're curious about the development of all of this you should follow it here:

http://www.silicondust.com/forum2/viewtopic.php?f=47&t=14612&p=85810#p85810

20

Good news....

.... you won't need to worry about this in the future!  Transcoding will be built into next gen network tuners:

http://www.engadget.com/tag/HDHR4-CC/