URL Service

No particular problems using it.  Though I am using trial and error more than I should to construct my regex codes.

I mainly just wanted to know so I could specify what type of regex was used in the notes of my plist template.  I was just thinking that the plist file might use Perl instead of Python regex since it is an xml document.

I also think I figured out the Metadata basics. It appears you should at the least pull the title, summary, and thumb.  A lot of services also pull the duration and originally_available_at.  All the options and the proper naming for the attributes of Metadata objects are listed in the Framework Documentation.

Also, I am wondering if I need to change the extension of my videos. All the videos for this URL service are f4v format, but in the code for each video page right under the line that states the value of the video, is a line that changes the format to mp4. I called both video formats using the video's http address in VLC.  The videos plays with both the .f4v and .mp4 extensions, but it is so much faster loading in .mp4 format. So, should I use the mp4 instead of f4v extension of the video?

Generally speaking, mp4 is the gold standard for cross-platform playability. Most (if not all) Plex clients can play MP4s natively, ie. without transcoding or remuxing. So, if the videos are all available as MP4 with the added bonus of loading faster, I would definitely recommend grabbing those rather than the f4v. The other option is to provide both options to the client and allow each client to auto-magically select the media-type that works best.  You can do that by adding more than one MediaObject for each Video.

I think I have figured out how to get the attributes of MediaObjects. I found that I can open the videos in VLC directly from the site and see the video and audio codec, the video resolution, and that the audio channels are stereo. So does that mean I can pull up a few videos for testing, make sure they are all the same and just use the info I find on VLC for the attributes? I also see that alot of URL services list the bitrate. Is it better to include the bitrate? How would I determine that? The VLC statistics show the content bitrate.  Would you put the maximim, minimum, or try to determine an average?

VLC is a great tool for determining codec information and such. That's the way I usually grab those details.

The majority of the metadata attributes are (to some extent) optional. The more data you can find and make add via the URL Service, the more metadata will be presented to the end user via the client. In some cases, such as bitrate and format, the metadata will also help clients make intelligent decisions about whether they can direct-play media or need to request transcoding/remuxing via the server. Some websites/APIs provide a bitrate guideline along with the media. When that's available, I always try to include it in the metadata. If including bitrate in the metadata requires calculating an average or guessing at the MAX or MIN, I tend to ignore it except in cases where multiple qualities of the same video are available. Such as a 'hi-res'  , a 'medium-res' and a 'low-res'. Then I think it's worthwhile adding the info if possible.

Thanks Mikedm139 for all the info.

I am still confused about the process of the MediaObjects and the PlayVideo function in each MediaObjects callback.

If I define multiple MediaObject in my URL service, what determines which of those MediaObjects will be returned when a URL is passed to the service? There has got to be some data on the url sent to the service that has to be analyzed or tells what quality the of video should be returned. And I have found nothing that talks about this or discusses how you determine which quality or version to return.

You mentioned in your post that you can "allow each client to auto-magically select the media-type that works best." What do they send to the URL service or what information in the url sent to the URL service provides this "auto-magical" selection?

For example, based on your suggestion, I should have four MediaObjects defined in my URL service.  A high and low quality of the .f4v and .mp4 version. So there are four different videos urls that could be chosen by the PlayVideo function to return to the MediaObjects callback:

http://video.domainname.com/videoname_hi.f4v

http://video.domainname.com/videoname_hi.mp4

http://video.domainname.com/videoname_lo.f4v

http://video.domainname.com/videoname_lo.mp4

Since each of these MediaObjects calls the same PlayVideo function which will only returns one value for the URL of the video, I would think that the PlayVideo function has to analyze some data or value from the url sent to the URL service to determine which video url it should choose to return to the requesting MediaObject. So don't you need to know what determines which of those MediaObjects will be returned, so you can decide which video and what quality to send back to the MediaObject?

The URL Service for the website GiantBomb is a fairly basic example of offering more than one video quality in your list of media objects. In this case:

def MediaObjectsForURL(url):

	return [
		MediaObject(
			parts = [PartObject(key=Callback(PlayVideo, url=url, fmt='hi'))],
			container = 'mp4',
			bitrate = 1500,
			optimized_for_streaming = True,
			video_resolution = 'sd',
			#aspect_ratio = '1.78',
			video_codec = VideoCodec.H264,
			audio_codec = AudioCodec.AAC,
			audio_channels = 2
		),
		MediaObject(
			parts = [PartObject(key=Callback(PlayVideo, url=url, fmt='lo'))],
			container = 'mp4',
			bitrate = 700,
			optimized_for_streaming = True,
			video_resolution = 'sd',
			#aspect_ratio = '1.78',
			video_codec = VideoCodec.H264,
			audio_codec = AudioCodec.AAC,
			audio_channels = 2
		)
	]

you can see that there are 2 quality options. One with a higher bitrate labelled as the 'hd', and one with the lower bitrate labelled 'sd'.  In the 'key' for each PartObject of each MediaObject, we use a Callback for PlayVideo and you can see that there is an extra argument passed to the PlayVideo function. In this case the argument is specified as 'fmt' but what it's called matters only in terms of (1) matching how it is declared in the PlayVideo() function definition, and (2) that it make some kind of sense to someone trying to interpret the code. The metadata included in the MediaObjects is what the clients use to make their intelligent decisions about which MediaObject to select. Eg. If the highest bitrate is higher than the available network connection, request the lower bitrate option, or if the option #1 uses a codec which the client cannot play natively try option #2 or request a transcode/remux from PMS, etc.

In the PlayVideo() function:

@indirect
def PlayVideo(url, fmt=''):
clip = GetClip(url)

if not clip:
	return None

if fmt == 'hi':
	video_url = clip['high_url']
elif fmt == 'lo':
	video_url = clip['low_url']

return IndirectResponse(VideoClipObject, key=video_url)

we see that the function definition includes the 'fmt' argument with a default value of an empty string. Then the function logic takes the value of that argument into consideration when determining what video url to return to the client. In this case the PlayVideo() function uses the @indirect decorator to indicate that the response from the function will be another ObjectContainer with a pointer to the actual media rather returning a link directly to the Media. There's a good chance you won't need to worry about that at this point so, we'll deal with that if/when it becomes relevant.

Thank you Mike, you do not know how helpful that is.  I probably accessed 20 different url services to see how they were written and never found one that explained it easily like that one did.  As soon as I saw your post, the whole thing made perfect sense.

I do have a couple little questions.  The 'fmt' argument having a default value of an empty string.  I have seen this elsewhere and it confuses me.  First why does the default need to be set to empty? Wouldn't that just delete any value coming into the function from a MediaObject?

Also, what is Redirect() that you see most services use at the end of this function to return the video url? Is this a Plex API, a Python function, or something else? Is it part of the Callback function? I just like to see the documentation that tells you its description, sytnax, etc and I could find nothing on this. Or any explanation of why I am using it or its parameters.

Thank you Mike, you do not know how helpful that is.  I probably accessed 20 different url services to see how they were written and never found one that explained it easily like that one did.  As soon as I saw your post, the whole thing made perfect sense.

I do have a couple little questions.  The 'fmt' argument having a default value of an empty string.  I have seen this elsewhere and it confuses me.  First why does the default need to be set to empty? Wouldn't that just delete any value coming into the function from a MediaObject?

Also, what is Redirect() that you see most services use at the end of this function to return the video url? Is this a Plex API, a Python function, or something else? Is it part of the Callback function? I just like to see the documentation that tells you its description, sytnax, etc and I could find nothing on this. Or any explanation of why I am using it or its parameters.

I'm glad it helped.

The default value of the 'fmt' argument will never overwrite the value passed via function call. It doesn't need to be an empty string. It could just as easily take a default value of one of the likely values to be passes ('hi' or 'lo'). It doesn't necessarily need any default value at all. I generally set a default for any extra arguments, if for no other reason than to make it clear what type of value is expected (in this case a string).

The Redirect() is (I believe) a Plex Framework function. Essentially what it does is to tell the Client to look at this new URL for the actual media file. Behind the scenes, PMS behaves more-or-less like a web server and the clients are browsers requesting URLs and reading the XML trees returned.  The Redirect() essentially passes a 302-redirect http header to the client with an attached pointer to the location of the media file. Generally, a Redirect() should only take one argument: an URL to a media file.

-Edit-

It was pointed out to me that, having the default value for the 'fmt' argument as an empty string is technically wrong since the code that handles that value is only expecting one of two values (either 'hi' or 'lo'). So, a better option is to set one of the expected values as the default. I believe the official github repo has already been updated to reflect that.

Again thank that makes perfect sense.  Thanks to all your help, I think I have almost gotten this URL Service thing figured out. I just cannot figure out the testing.

Do I have to create a test function for my URL Service? Most that I have seen do not have one, so I wonder if it is required.

Then I also read that once you have the URL Service set up, you can enter an address from that site to

http://localhost:32400/system/services/url/lookup?url=

To see if it is working, but I am not sure where I would need to save the Url service to test it this way

You shouldn't have to save it anywhere different to test it by hitting that localhost link.  It can be either in your channel structure in the appropriate place or within the main Services.bundle and it will work from either place.  When PMS launches it scans and registers all the available URL services, then it finds a match for a URL you pass to it using the URLPatterns you setup in the ServiceInfo.plist file.  If the lookup is successful that local URL will return properly formed XML with all of the appropriate information from the URL service.  If there's an error it will usually show you a traceback in that XML file (and it will certainly show up in the appropriate logs).

Debugging URL services in terms of what goes into what log file can be a bit tricky sometimes.  You'll want to watch the main Plex Media Server.log file as well as the com.plexapp.system.log (in the same folder as your channel/s log would be).  Also make sure to urlencode your url you pass there. This is an easy online tool to do that for you:  http://meyerweb.com/eric/tools/dencoder/

Hope this helps.

Lastly, I have to say, the best thing for channel stuff is to honestly dive right in and try things, you'll probably make faster progress that way than waiting for someone to answer a question.  The best examples you can find are in existing channels and URL services.  When I'm looking for a working example of something I tend to just grep the URL services to see where a certain function is used and how they use it.  Most of the stuff on the channel store has been vetted somewhat, so typically it will be at least decent working examples of things if not the perfect way of doing something.  Sometimes you have to get a bit creative to work around things.

I figured out my problem.  The site I was designing the channel for just redid all their video pages and changed the access to their video files to a cloud server called Brightcove.  Looking at the code, I do not know if there is any way to access the location on Brightcove or if I could even create a Site configuration. 

The videos are still available locally on their website at the same address as before.  I can play them directly using Flash by entering the address of their swf embed player I pulled from the old embed code and by entering the address of the video in my web browser and playing them with VLC media player. But since the actual html video pages on their site no longer use these videos, I am not sure if that does me any good or if there is any way to work with just the raw video files in Plex.

It seems like every website I have considered writing a channel for are the ones that hide their video files really really well.  I know I could just find some random site with easy access and just write the channel for the practice, but what is the point in writing a channel for a site that is easy to access if the content is not worth watching.

And again thanks for all the help.  Now I at least understand the concepts, flow, and structure of a URL service. And that will help me greatly once I figure out where to go from here.

And I want to wait until I get a channel and URL Service actually working first, to know the format is correct.  But once I do, I will post the templates I have put together, with all the notes I have from everyone's advice and help.

I may have to chunk my current project but it made me start thinking about what I want to look for in the setup of a web page to put together a channel. 

And now that I understand how the MediaObjectsforURL function works.  I am completely confused about the bigger picture of how your channel interacts with the URL service .  If the URL service allows you to create metadata information for web pages and the MetadatObjectforURL function is not used by your channel, that would mean the only function you have to have in your URL service to create a channel is the MediaObjectforURL that defines the videos within the URL.  And that would also mean the MediaObjectforURL function is what allows you to create metadata for your channel.  That would mean that the URL passed to the service, must contain a video object for the URL service to create any metadata info for that URL.

And if what mmcurdy says below is accurate, that would mean that your channel does not even interact with or call the URL service until you are requesting an actual playable video like through an EpisodeObject or VideoClipObject.  That would mean you only need the URL service to pull metadata associated with playable videos.  So you can pull metadata for directory information in a channel without ever calling URL service.

Since it sounds like you're just trying (for now) to build a channel with some navigation that will be using the URL service directly, I would suggest you don't worry too much about  MetadataObjectForUrl for the time being.  This may seem counter-intuitive, but it's actually not recommended to call that function from your channel in most cases, since it incurs an additional HTTP request, and in most cases you will already have loaded everything you need to create an appropriate media object, which has all the info needed to tell Plex how to play the media itself.

So, you can look at any number of channels for examples of how to build hierarchical navigation by extracting this info from the site.  It's fine and standard practice to directly set a "global" variable in the plugin.  To choose a semi-random example, the CBS plugin does this right off the bat:

CBS_LIST = 'http://www.cbs.com/video/'

From there you would make a request for that page and build up your menus.  In the CBS case, there are a couple of levels of hierarchy, then eventually it returns ObjectContainers containing actual, playable EpisodeObjects or VideoClipObjects depending on whether we're looking at full episodes or not.  It's the "url" of these objects that gets passed to MediaObjectsForUrl -- note that MetadataObjectsForUrl is not even used in this case since we're calling it from a channel and we already have all the metadata we need to fill out the EpisodeObjects and VideoClipObjects from the requests we've been making in order to build the navigation.  Later on, once you've got this much working, you can go back and implement the MetadataObjectsForUrl so the service can get incorporated into the larger Services.bundle for use with the bookmarklet.

I know it's a little confusing at first, hopefully that helps clear things up at least a little bit.

So in theory, I should be able to create a channel __init__.py that just calls a directory object from a URL and get the title and icon of a series of videos and it would never be passed to the URL service, so I would not need to create a URL service until I actually add playable videos to my channel __init__.py code. But I cannot do that without it searching and telling me there is a problem with the connection to the server, so I am thinking that it does call the URL service even when you are just pulling metadata for the higher level directories.

So, though I understand that the URL service helps you build metadata information for an actual playable video when you pass the URL to it, how does it build the metadata for URLs you pass to it that do not include a playable video?  In order to make a manageable list of videos, you need to create directories and that requires pulling metadata from the html or xml pages that just contain the list of the shows or episodes available on the website? If the MediaObjectforURL function is actually the part of the URL service that allows for the creation of metadata, how does it create metadata for these pages, since they do not include the actual video files? 

So in theory, I should be able to create a channel __init__.py that just calls a directory object from a URL and get the title and icon of a series of videos and it would never be passed to the URL service, so I would not need to create a URL service until I actually add playable videos to my channel __init__.py code. But I cannot do that without it searching and telling me there is a problem with the connection to the server, so I am thinking that it does call the URL service even when you are just pulling metadata for the higher level directories.

So, though I understand that the URL service helps you build metadata information for an actual playable video when you pass the URL to it, how does it build the metadata for URLs you pass to it that do not include a playable video?  In order to make a manageable list of videos, you need to create directories and that requires pulling metadata from the html or xml pages that just contain the list of the shows or episodes available on the website? If the MediaObjectforURL function is actually the part of the URL service that allows for the creation of metadata, how does it create metadata for these pages, since they do not include the actual video files? 

URL Services are multi-purpose. They are used for things other than just grabbing videos/metadata for channel plugins. The most visible of the other functions is the myPlex Queue service.  That is why the MetadataObjectForURL function is included in the URL Service. So that when a user Queues a video directly from the host website in their web browser, the myPlex server can grab the appropriate metadata to display in the Plex client.  The MetadataObjectForURL function is not intended to help populate channel directories with metadata because each time the function is called, the URL Service makes an HTTP request for the video page to grab the metadata. If there's only one or two videos in the directory, that may not seem like a big deal but as the size of the directory grows, the number of HTTP requests and the amount of data being transfer grows very large.  The end result is that it takes a long time to build the directory and (more than likely) the client times out and fails to load the menu.

On the other hand, the MediaObjectsForURL function is used by channels as well as the myPlex Queue.  The intention for the MediaObjectForURL function is to return a list of media objects very quickly, without making any HTTP requests, allowing for a large directory to be populated with video objects which include details like resolution, codec, bitrate, etc. Only in the PlayVideo function should there be any HTTP requests. That is where the URL Service actually grabs the page in order to parse out the actual video file.

The channel code is responsible for grabbing all the metadata necessary to build the directory structure (thumbnail images, video descriptions, etc.) both for videos and menu hierarchy items.  Then, when lists of videos are being populated, the URL Service is queried for media-specific metadata (available bitrates, audio/video codecs, formats, etc), *without* actually querying the webpage. When the user selects a video and chooses to play it, the PlayVideo function is called which can request the actual source page for the video and return a link to the actual video file.

So the URL service is not called to produce all the metadata for your channel __inti__.py? The URL service is only called by your channel code when a list of videos are being populated?

It is confusing since the Framework documentation specifically says the URL service is for producing metadata objects.  The opening statement for the URL service says "URL services are responsible for taking a website’s URL and returning standardized Plex metadata objects. "

If the channel code can produce basic metadata without a URL service, shouldn't I be able create a channel without a URL service as long as it doesn't try to add any objects for playable videos? I created a channel bundle with an __init__.py file that just called metadata to produce a first level DirectoryObject of titles and thumbs for a list of shows. It had all the files and proper naming and folder structure for a channel, just no URL Service. So, why do I get a server error when I try to open this test channel in Plex? Shouldn't it produce the first level directory from the metadata it pulled from the web page I specified?

I would need to see the code and/or the error that got logged to make a reasonable guess as to why there was a problem.

So, though I understand that the URL service helps you build metadata information for an actual playable video when you pass the URL to it, how does it build the metadata for URLs you pass to it that do not include a playable video?

You can't. URL Services can only work with pages that contain media (video is the most common, but audio and photo are possible as well). Navigation (as in ShowsSeasonsEpisodes for example) is all done in the plugin/channel code.

I created a channel bundle with an __init__.py file that just called metadata to produce a first level DirectoryObject of titles and thumbs for a list of shows. It had all the files and proper naming and folder structure for a channel, just no URL Service.

I'm not entirely sure what you are doing, but URL Services are meant to "convert" 1 URL (to a web page that contains a video) into 1 metadata object. They're never used to populate a DirectoryObject with other DirectoryObjects.

It is not important. The channel I was creating is now sadly a dead project at least for the moment due to their changing their website this week.  Because now the video player within the urls pull the videos from an online cloud location instead of the local file location I was writing this URL service for. 

I understand, the channel I have created would have no real purpose and in no way be a working channel that brought up videos.  But, since it is supposedly possible in theory, I just wanted to test that theory and see if I could create a channel and pull metadata without there being a URL service as long as I was not calling actual video file locations.

It is just that now, because I cannot get this theory to work in practice, I am doubting my understanding of the basic structure and xpath coding of __init__.py files. I pmed mike ust to make sure I am not completely off track.

Ok two final questions about URL service just to make sure I understand all the basics of it and I will stop bugging you guys with this topic and go find a new project.

Normalize Url.  I understand the concept of their being several ways to access the same page in a website, and you want to tell the URL service to change all of those options to one format.  So do you always want it to take out any extras and make it the simplest and shortest path? Or do you have the flexibility to choose the normalized format you want to use based on how you will use the entered URL later in the code of the URL service?

Secondly, testing of the URL service. Other than adding a test URL to the ServiceInfo.plist, do you have to add any test function to your URL service for the http://localhost:32400/system/services/url/lookup?url= to work?  Do you even have to enter a test url in the ServiceInfo.plist?  When testing your URL service using the testing http address listed above, do you always enter the URL you listed for testing in the ServiceInfo.plist  or can you enter any valid URL within the site that should be processed by your URL service?

There aren't really many guidelines in terms of the NormalizeURL function. The vast majority of URL Services either don't have it or don't do anything with it. The only real rule is that the function cannot make any http requests. it has to return very quickly so any modification of the url has to be done without needing to fetch the page. Ie. using preset rules based on the format. If there is a lot of extraneous arguments included in the url and it's easier to deal with it after stripping the garbage out, then by all means do so in the NormalizeURL function.

In regards to testing, there are two places to specify test urls. They are mutually exclusive. Choose one or the other but not both. In either case, the test urls are used by the automated tester to (try to) detect errors and changes to source pages before it becomes a major issue for users.  You can include a list of (1 to 3, preferably not too many) static test urls in the ServiceInfo.plist file that the tester will use for every test. This scenario works well if the website does not remove (expire) videos after a certain time frame. If the website does remove "old" videos, then you would need to constantly update your list of test urls. In that case using a TestURLs function in the ServiceCode.pys file to dynamically generate a short list of a up-to-date test urls is the better option.  Neither of these scenarios has any impact on manually testing your service by requesting the "lookup?url". You can input any (properly encoded) url as the argument in that request to test your service (or any service). 

Thanks Mike.

One question, how exactly does this automated tester work?

"the test urls are used by the automated tester to (try to) detect errors and changes to source pages before it becomes a major issue for users."

The URL lookup just seems to mainly pull the metadata for the URL.

The automated tester runs against an instance of the Services.bundle hosted on the myPlex servers. It generates a list of test URLs for all the services using the urls provided in the ServiceInfo.plist files or by running the TestURLs() functions. Then it calls the lookup for each if those test URLs to confirm that proper metadata is returned for each one and that (where possible) the URL service returns a direct usable link to a media file.


To test that the service returns proper media, you should be checking the “key” of the media part included in the XML returned by the lookup URL. Often, it’s best to follow the URL provided in the key just as a Plex client will try to do.