XKCD update/Server Error

Dev Community Dev/API Corner
1
Problem with updating XKCD
Hello,
I am trying to update the XKCD plugin. I did it ok using the old structure, but I ran in timeout problems due to the large amount of pictures.
So I decided to rewrite it using the object structure. Here is the code:

from urlparse import urljoin<br />
from math import ceil<br />
<br />
NAME    = 'xkcd'<br />
PLUGIN_PREFIX   = '/photos/xkcd'<br />
ART      = 'art-default.jpg'<br />
ICON     = 'icon-default.png'<br />
NB_IMAGES = 200<br />
CACHE_1YEAR = 365 * CACHE_1DAY<br />
####################################################################################################<br />
def Start():<br />
    Plugin.AddViewGroup("List", viewMode="List", mediaType="items")<br />
    Plugin.AddViewGroup("Pictures", viewMode="Pictures", mediaType="photos")<br />
<br />
    # Set the default ObjectContainer attributes<br />
    ObjectContainer.art = R(ART)<br />
    ObjectContainer.title1 = NAME<br />
    ObjectContainer.view_group = "List"<br />
<br />
    # Default icons for DirectoryObject<br />
    DirectoryObject.thumb = R(ICON)<br />
    DirectoryObject.art = R(ART)<br />
<br />
    # Set the default cache time<br />
    HTTP.CacheTime = CACHE_1HOUR<br />
<br />
####################################################################################################<br />
@handler(PLUGIN_PREFIX, NAME, art = ART, thumb = ICON)<br />
def XKCDMenu():<br />
    archiveURL = 'http://xkcd.com/archive/'<br />
    archiveXPath = '//div[@id="middleContainer"]/a'<br />
    <br />
    oc = ObjectContainer()<br />
    # Get all the elements needed to determine the number of entries and how to sectionize them<br />
    StripsMainPage = HTML.ElementFromURL(archiveURL).xpath(archiveXPath)<br />
    StripsList = [(comic.text, urljoin(archiveURL, comic.get('href'))) for comic in StripsMainPage]<br />
    StripsList.reverse()<br />
<br />
    if len(StripsList) < NB_IMAGES:<br />
        #will never happen but will be implemented later on<br />
        pass<br />
    else:<br />
        #Create subdirectories containing a fixed number of images<br />
        nbdir = ceil(float(len(StripsList))/NB_IMAGES)<br />
        for i in xrange(1,nbdir+1):<br />
            firstelt = (i-1)*NB_IMAGES<br />
            lastelt = i*NB_IMAGES<br />
            refs = StripsList[firstelt:lastelt]<br />
            if i == nbdir:<br />
                refs = StripsList[firstelt:]<br />
                lastelt = len(StripsList)<br />
            name = '%d - %d' % (firstelt+1, lastelt)<br />
            img = GetDirectoryIcon(StripsList[firstelt])<br />
            oc.add(DirectoryObject(key = Callback(StripDirectory, id = i, stripsdata = refs), title = name, thumb = img))<br />
    return oc<br />
<br />
####################################################################################################<br />
@route('%s/dir-{id}' % PLUGIN_PREFIX, method='POST')<br />
def StripDirectory(id, stripsdata, sender = None):<br />
    imgXpath = '//div[@id="comic"]//img'<br />
    # oc = ObjectContainer(view_group = 'Pictures')<br />
    oc = ObjectContainer(view_group = 'List')<br />
<br />
    for (title,comicURL) in stripsdata:<br />
        imgs = HTML.ElementFromURL(comicURL, cacheTime=CACHE_1YEAR).xpath(imgXpath)<br />
        if len(imgs):<br />
            img = imgs[0].get('src')<br />
            resume = imgs[0].get('title')<br />
            oc.add(PhotoObject(url=img, title=title, thumb=img, summary=resume))<br />
    return oc<br />
<br />
####################################################################################################<br />
def GetDirectoryIcon(stripdata):<br />
    imgXpath = '//div[@id="comic"]//img'<br />
    imgs = HTML.ElementFromURL(stripdata[1], cacheTime=CACHE_1YEAR).xpath(imgXpath)<br />
    img = R(ICON)<br />
    if len(imgs):<br />
        img = imgs[0].get('src')<br />
    return img


It is in fact quite simple. My problem is that when I introduce the line to add the photoobject, I run into an internal server error:

The URL it is trying to get http://imgs.xkcd.com/comics/barrel_cropped_(1).jpg corresponds to the first entry http://xkcd.com/1/
Could anyone help me with this?
Thanks
2


It's great to see more people getting involved in plugin development/upgrading. The problem is that when using the "url" parameter for adding a PhotoObject (or VideoObject), PMS tries to find an URL Service to handle the given URL and return a playable/viewable media object. Since no URL Service currently exists for XKCD, the plugin fails to load the objects. The ideal solution to this is to create an URL Service for XKCD. This has the added bonus of allowing support for the PlexIt bookmarklet on the XKCD website as well as other goodies like being able to recommend comics to friends via myPlex. The URL Service for "Lens" is probably the best example at the moment of how to implement an URL Service with PhotoObjects. Once that's done, there's very little else that would need fixing in your code. I've include some commented changes below.


from urlparse import urljoin<br />
from math import ceil<br />
<br />
NAME    = 'xkcd'<br />
PLUGIN_PREFIX   = '/photos/xkcd'<br />
ART      = 'art-default.jpg'<br />
ICON 	= 'icon-default.png'<br />
NB_IMAGES = 200<br />
CACHE_1YEAR = 365 * CACHE_1DAY<br />
####################################################################################################<br />
def Start():<br />
''' ViewGroups are no longer respected by all clients and are being removed from plugins to allow clients to maker "smart" decisions about how to display contents '''<br />
    #Plugin.AddViewGroup("List", viewMode="List", mediaType="items")<br />
    #Plugin.AddViewGroup("Pictures", viewMode="Pictures", mediaType="photos")<br />
<br />
    # Set the default ObjectContainer attributes<br />
    ObjectContainer.art = R(ART)<br />
    ObjectContainer.title1 = NAME<br />
    #ObjectContainer.view_group = "List" '''see above'''<br />
<br />
    # Default icons for DirectoryObject<br />
    DirectoryObject.thumb = R(ICON)<br />
    DirectoryObject.art = R(ART)<br />
<br />
    # Set the default cache time<br />
    HTTP.CacheTime = CACHE_1HOUR<br />
<br />
####################################################################################################<br />
@handler(PLUGIN_PREFIX, NAME, art = ART, thumb = ICON)<br />
def XKCDMenu():<br />
    archiveURL = 'http://xkcd.com/archive/'<br />
    archiveXPath = '//div[@id="middleContainer"]/a'<br />
    <br />
    oc = ObjectContainer()<br />
    # Get all the elements needed to determine the number of entries and how to sectionize them<br />
    StripsMainPage = HTML.ElementFromURL(archiveURL).xpath(archiveXPath)<br />
    StripsList = [(comic.text, urljoin(archiveURL, comic.get('href'))) for comic in StripsMainPage]<br />
    StripsList.reverse()<br />
<br />
    if len(StripsList) < NB_IMAGES:<br />
        #will never happen but will be implemented later on<br />
        pass<br />
    else:<br />
        #Create subdirectories containing a fixed number of images<br />
        nbdir = ceil(float(len(StripsList))/NB_IMAGES)<br />
        for i in xrange(1,nbdir+1):<br />
            firstelt = (i-1)*NB_IMAGES<br />
            lastelt = i*NB_IMAGES<br />
            refs = StripsList[firstelt:lastelt]<br />
            if i == nbdir:<br />
                refs = StripsList[firstelt:]<br />
                lastelt = len(StripsList)<br />
            name = '%d - %d' % (firstelt+1, lastelt)<br />
	   ''' Forcing the plugin to load every page and grab the thumb in order is very slow. Using a Callback to generate the thumb allows them to load asynchronously'''<br />
            #img = GetDirectoryIcon(StripsList[firstelt])<br />
            oc.add(DirectoryObject(key = Callback(StripDirectory, id = i, stripsdata = refs), title = name, thumb = Callback(GetDirectoryIcon, StripsList[firstelt])))<br />
    return oc<br />
<br />
####################################################################################################<br />
''' the @route is much simpler if you make it explicit for each function rather than using variables to define a changeable route'''<br />
@route('/photos/xkcd/stripdir')<br />
#@route('%s/dir-{id}' % PLUGIN_PREFIX, method='POST')<br />
def StripDirectory(id, stripsdata, sender = None):<br />
    imgXpath = '//div[@id="comic"]//img'<br />
    # oc = ObjectContainer(view_group = 'Pictures')<br />
    #oc = ObjectContainer(view_group = 'List') ''' see comments re: View Groups above '''<br />
<br />
    for (title,comicURL) in stripsdata:<br />
        imgs = HTML.ElementFromURL(comicURL, cacheTime=CACHE_1YEAR).xpath(imgXpath)<br />
        if len(imgs):<br />
            img = imgs[0].get('src')<br />
            resume = imgs[0].get('title')<br />
	   ''' Use the actual page url for the url parameter to call the URL Service and have it return the MediaObject. Also,<br />
		use a callback to make thumb loading asynchronous<br />
	      oc.add(PhotoObject(url=comicURL, title=title, thumb=Resource.ContentsOfURLWithFallback(url=img, fallback=ICON), summary=resume))<br />
            #oc.add(PhotoObject(url=img, title=title, thumb=img, summary=resume))<br />
    return oc<br />
<br />
####################################################################################################<br />
''' add a @route for each function '''<br />
@route('/photos/xkcd/getdir')<br />
def GetDirectoryIcon(stripdata):<br />
    imgXpath = '//div[@id="comic"]//img'<br />
    imgs = HTML.ElementFromURL(stripdata[1], cacheTime=CACHE_1YEAR).xpath(imgXpath)<br />
    img = R(ICON)<br />
    if len(imgs):<br />
        img = imgs[0].get('src')<br />
    return img


Here's a good place to do a little reading about URL Services. If you run into any more trouble, just ask.

P.S. I'm moving this thread to the Channel Development sub-forum.
3

Thank you for your answer and your help.

Indeed, I posted in the wrong subforum but I didn’t find how to change it (don’t know if it is even possible for me)



Concerning your comments, I have a few questions. I will look around about the URL service, no problem.



For the routes, I am a little confused: the decorator is only used to give the function a quick way to be accessed by a link with parameters ie like an URL request?

Then what is the interest to use a parameter {x}? I based this part on the National geographic plug-in and I thought this was to get dedicated access for subdirectories.

Did I misunderstand this or was there another reason?



‘’’ Forcing the plugin to load every page and grab the thumb in order is very slow. Using a Callback to generate the thumb allows them to load asynchronously’’’

Basically, what you are saying is that any manageable URL call should be handled by a callback function until the bottom level function.



I will look at service for the last comment.



Thanks again.

4

No worries. It's really not a big deal :)


TBH, I've only recently started using @route decorators. They're not technically "required". The framework will create them begin the scenes as necessary but then it fills up the logs with message about missing Routes. I haven't looked at National Geo since the @routes were added. Perhaps Jam or one of the other plugin devs can chime in with some clarification.


Essentially, the concept is that when building a list, the concept is to minimize the number of HTTP requests because that really slows things down. Since grabbing a thumb image usually requires at least one HTTP request per item, separating that logic into a callback allows the plugin to load and return the list without waiting for every thumbnail to return. The key (if used) and thumb parameters are the (I think) only ones that will take a Callback to allow asynchronous loading.



Glad to help. Always happy to see people taking an interest in plugin development.
5

Ok.

First, I have a working version of the plugin attached xkcd.bundle-v2-working.rar (102 KB) and a not working version I tried to build as Lens with photo album xkcd.bundle-not working.rar (101 KB). I really would love to understand why some things are not working, but nevermind.



Nevertheless, I still have a few questions:


You are totally right and this a pain in the ass for me.

Ok, the main bottle neck in this plugin is metadata retrieval ie the part roughly constructed like this:

<br />
@route('/photos/xkcd/stripdir')<br />
def StripDirectory(stripsstart, stripsstop, sender = None):<br />
    imgXpath = '//div[@id="comic"]//img'<br />
    oc = ObjectContainer()<br />
<br />
    for comicURL in GetStripList()[int(stripsstart):int(stripsstop)]:<br />
        imgs = HTML.ElementFromURL(comicURL, cacheTime=CACHE_1YEAR).xpath(imgXpath)[0]<br />
        img = imgs.get('src')<br />
        resume = imgs.get('title')<br />
        titre = imgs.get('alt')<br />
        oc.add(PhotoObject(url=comicURL, title=titre, thumb=Callback(GetIcon, stripURL=comicURL), summary=resume))<br />
    return oc<br />

The for is making a lot of HTTP calls to get metadata.
First idea was to use Callback to use Plex thread system to speed it -->Not possible
Second idea, use a MetadataFromURL:

test = URLService.MetadataObjectForURL(url=comicURL)<br />
test.add(URLService.MediaObjectsForURL(url=comicURL))<br />
oc.add(test)

This is not working (I would love the code to make it work) and should be bad code, but in this case as you have to call the URL to get metadata, it would in fact be the more efficient method.
URLService.MetadataObjectForURL(url=comicURL) is however slower than the previous code which I don't understand.
Finally, I thought about using PhotoAlbum but don't know exactly what is the interest of this structure.

Anyway, I can nix it by adding threading (I think) but frankly I am not so much interested in going to the big guns if I can do it otherwise.

Another problem which should be easier to solve, is how to put text under the image when you tap it. Previously, I used the summary property but with PhotoObject this doesn't seem to work.
Anyway, I would love to manage the way this is displayed on my tablet but this doesn't easy.
I even have rating under the image in some cases.
Another weird think is that if I don't define a thumb, I don't have any image anymore only a thumb. Hum...

OK, then I think that summ it all! Any help welcome.
6

hi :slight_smile:

any updates? Is this already in the git(AppStore)?

7

I forked the modifications on git. I used a working solution, not exactly what I wanted but it gets the job done.

Concerning new modifications, nothing is planned. I won’t put any more work on this til there is a decent working doc or people answering technical questions.

8

Well, having some time on my hands and a new NAS, I got back to this plugin.

A new working and improved version is on my git account: https://github.com/sengian/xkcd.bundle/

I still have this old problem about service so as this would really be more efficient, I ask the question:
Is there a way to call an url without a service using a PhotoObject? (I would really love to be able to do this)

9

As a bonus (if anyone is interested), my windows batch script to kill, copy and restart the server. Not sure it is necessary but it works!

@echo off
@setlocal
set path="C:\Program Files (x86)\Mozilla Firefox\";%path%
set plugin=Path to your Plugin
set devfolder=Path to your Plugin dev folder
set start="C:\Program Files (x86)\Plex\Plex Media Server\Plex Media Server.exe"
set program="Plex Media Server.exe"

taskkill /IM %program% /T /F

timeout 1

robocopy “%devfolder%” “%plugin%” /s /xf “Update.bat” “.gitignore” /xd “.git”

start “” %start%

timeout 1

firefox http://localhost:32400/web/index.html#!/dashboard
10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.