Welcome to our forums! Please take a few moments to read through our Community Guidelines (also conveniently linked in the header at the top of each page). There, you'll find guidelines on conduct, tips on getting the help you may be searching for, and more!

[REL] RemIdx

dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
edited March 2014 in Plex Channel Plugins

WARNING: There has been reported an issue, that the indexes is out of sync, and breaks if fast forward is pressed.

Until resolved, please discard this tool!

 

 

 

It's a REMOTE INDEXER

 

This means, that you now can do your indexing from a remote workstation, that is cabled with your PMS, without your PMS been drained CPU wise

Making it even more usefull for low powered NAS boxes

 

And even better.....You can at any time stop the Remote Indexer process on the remote workstation, and when started again, it'll pick up where it left.

 

Remote Indexer has currently only been tested on OpenSuse and Windows, but I suspect, that it should run on any flavor of Windows, Linux and Mac.

 

Remote Indexer bundle (The part that runs on your PMS), has currently only been tested on QNAP

 

To grap it, take a peek here: https://github.com/ukdtom/RemIdx

 

Best Regards

 

Tommy

 

P.S. Feedbacks to this thread is more than welcome, both good and bad news

P.P.S. If you are a dev, then please fork away, and check back code....I need help here ;)

 

Edit: 28. feb. 2014: Version 0.0.1.4 released (And this is a major on, if running Windows)

Edit: 26. feb. 2014: Version 0.0.1.3 released

Edit: 25. feb. 2014: Version 0.0.1.2 released

Edit: 23. feb. 2014: Version 0.0.1.1 released

Edit: 22. feb. 2014: Version 0.0.1.0 released

Edit: 22. feb. 2014: Version 0.0.0.7 released

Edit: 22. feb. 2014: Version 0.0.0.6 released

I hate bugs - Tommy Lee Jones, MIB
Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
Support the Samsung Client:Donate
Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

NO Support via PM, unless called by me

«134

Comments

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
    edited February 2014

    Updated documentation with installation notes

    Edit: 21-feb 2014 14.00 GMT+1 Yet another Doc update

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
    edited February 2014

    Sadly, made a slight error, that caused the PMS agent to not pick up new contents on a forced refresh.

    Corrected now in V0.0.0.6

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja

    V0.0.0.7:

    Added a thread limiter for the Remote Indexer, so you now can adjust how many CPU cores to use during screenshot creation

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja

    Happy to say, that I now went from Alpha to Beta :D

    Windows is now a supported platform

    Version is 0.0.1.0

    And find the link in the first post in this thread

    Feedback are welcome

    /Tommy

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja

    Updated to V0.0.1.1

    Fixed issue with missing registration under TV-Shows

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja

    Updated to V0.0.1.2

    Added a configuration wizard to the Remote Indexer, so no longer the need to fiddle with the python file ;)

    Download info here

    https://forums.plex.tv/topic/99413-rel-remidx/

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • s_razers_razer Posts: 114Members, Plex Pass Plex Pass
    edited February 2014

    Tommy,

    I just started testing this.

    I got this:

    2014-02-26 10:03:01,190 (35a0) :  CRITICAL (core:561) - Exception in the update function of agent named 'RemIdx (Movies)', called with guid 'com.plexapp.agents.imdb://tt2294629?lang=en' (most recent call last):
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 970, in _update
        agent.update(obj, media, lang, force)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 41, in update
        GetMediaInfoMovie(media.id, media.title)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 69, in GetMediaInfoMovie
        sections = XML.ElementFromURL(myNewURL).xpath('//MediaPart')
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\parsekit.py", line 344, in ElementFromURL
        method=method,
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\networkkit.py", line 67, in _http_request
        req = self._core.networking.http_request(url, *args, **kwargs)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components\networking.py", line 422, in http_request
        return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components\networking.py", line 147, in __init__
        self.load()
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components\networking.py", line 183, in load
        f = self._opener.open(req, timeout=self._timeout)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 410, in open
        response = meth(req, response)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 523, in http_response
        'http', request, response, code, msg, hdrs)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 448, in error
        return self._call_chain(*args)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 382, in _call_chain
        result = func(*args)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 531, in http_error_default
        raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    HTTPError: HTTP Error 401: Unauthorized
    
    

    This is because I have require authentication on local networks turned on. I had to add my PMS local ip address to the list of networks that are allowed without auth. 

    ex: 192.168.0.21/255.255.255.255

    After that change it seems to be working. Might be worth a note in your readme.

    Chris

    Edit:

    I believe the Indexer PC also needs to be in the allowed without auth list. I already have mine in the list, so I'm not sure. 

  • s_razers_razer Posts: 114Members, Plex Pass Plex Pass
    edited February 2014

    Now I'm getting this in the indexer:

    Traceback (most recent call last):
      File "C:\Users\Chris\Documents\GitHub\RemIdx\Remote_Indexer\RemIdx.py", line 548, in <module>
        main()
      File "C:\Users\Chris\Documents\GitHub\RemIdx\Remote_Indexer\RemIdx.py", line 538, in main
        GenJPGs(sMyDir)
      File "C:\Users\Chris\Documents\GitHub\RemIdx\Remote_Indexer\RemIdx.py", line 236, in GenJPGs
        data = json.load(data_file)
      File "C:\Python27\lib\json\__init__.py", line 290, in load
        **kw)
      File "C:\Python27\lib\json\__init__.py", line 338, in loads
        return _default_decoder.decode(s)
      File "C:\Python27\lib\json\decoder.py", line 365, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
        raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded
    

    I'll dig a little deeper later today hopefully.

    Edit:

    The .bundle file in the Work dir is empty. Not sure how that happened.

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
    srazer wrote on February 26 2014, 4:15 PM: »

    Edit:

    The .bundle file in the Work dir is empty. Not sure how that happened.

    Seen that as well very periodic, and could be trapped -> cleanout workdir

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
    srazer wrote on February 26 2014, 3:37 PM: »

    Tommy,

    I just started testing this.

    I got this:

    2014-02-26 10:03:01,190 (35a0) :  CRITICAL (core:561) - Exception in the update function of agent named 'RemIdx (Movies)', called with guid 'com.plexapp.agents.imdb://tt2294629?lang=en' (most recent call last):
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 970, in _update
        agent.update(obj, media, lang, force)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 41, in update
        GetMediaInfoMovie(media.id, media.title)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 69, in GetMediaInfoMovie
        sections = XML.ElementFromURL(myNewURL).xpath('//MediaPart')
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\parsekit.py", line 344, in ElementFromURL
        method=method,
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\networkkit.py", line 67, in _http_request
        req = self._core.networking.http_request(url, *args, **kwargs)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components\networking.py", line 422, in http_request
        return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components\networking.py", line 147, in __init__
        self.load()
      File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components\networking.py", line 183, in load
        f = self._opener.open(req, timeout=self._timeout)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 410, in open
        response = meth(req, response)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 523, in http_response
        'http', request, response, code, msg, hdrs)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 448, in error
        return self._call_chain(*args)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 382, in _call_chain
        result = func(*args)
      File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 531, in http_error_default
        raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    HTTPError: HTTP Error 401: Unauthorized
    
    

    This is because I have require authentication on local networks turned on. I had to add my PMS local ip address to the list of networks that are allowed without auth. 

    ex: 192.168.0.21/255.255.255.255

    After that change it seems to be working. Might be worth a note in your readme.

    Nice catch....never thought of that, and will do

    srazer wrote on February 26 2014, 3:37 PM: »

    Edit:

    I believe the Indexer PC also needs to be in the allowed without auth list. I already have mine in the list, so I'm not sure. 

    What do you mean here?

    Best regards, and huge thanks for taking the time here

    /Tommy

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • s_razers_razer Posts: 114Members, Plex Pass Plex Pass
    dane22 wrote on February 26 2014, 4:23 PM: »

    Nice catch....never thought of that, and will do

    What do you mean here?

    Best regards, and huge thanks for taking the time here

    /Tommy

    The PC running RemIdx.py I believe has to be in the allowed without auth list.

    For example my allowed without auth list looks like:

    127.0.0.1/255.255.255.255,192.168.0.1/255.255.255.255,192.168.0.21/255.255.255.255

     
    192.168.0.1     Is my pc running RemIdx.py
    192.168.0.21   My PMS
     
     
    Chris
  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
    srazer wrote on February 26 2014, 4:27 PM: »

    The PC running RemIdx.py I believe has to be in the allowed without auth list.

    For example my allowed without auth list looks like:

    127.0.0.1/255.255.255.255,192.168.0.1/255.255.255.255,192.168.0.21/255.255.255.255

     
    192.168.0.1     Is my pc running RemIdx.py
    192.168.0.21   My PMS
     
     
    Chris

    Got it

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja

    Updated to V0.0.1.3

    Fixed issue with bundle file in Work directory sometimes was empty

    Updated doc to reflect the needs to allow the Remote Indexer access without auth

    Both above was discovered by srazer

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja

    Updated to V0.0.1.4

    Before telling what's new here, I would like to tell you, that this project now actually will raise from the dark and into heaven, cuz I'm been blessed with the company of srazer, that is now working with me on this.

    And this version is all his, and here's what he added:

    - Remote Indexer:
    -- Process will now run as low priority to prevent from bogging down the system.
    -- Framework now to build win32 executable using PyInstaller.
    -- ffmpeg's log level can now be set in the config file.
    -- Allow default config values to be set by pressing enter.

    Overall.....If on Windows, simply launch an exe file....

    If on Mac or Linux -> Nerd mode ;)

    /Tommy and Chris

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • s_razers_razer Posts: 114Members, Plex Pass Plex Pass
    edited February 2014

    Oh shoot  B)

    I'm not all that good. I can google and copy and past code fairly well though.... lol  :ph34r:

  • s_razers_razer Posts: 114Members, Plex Pass Plex Pass
    edited February 2014

    Wrong place. lol.

  • benaddersonbenadderson Posts: 110Members, Plex Pass Plex Pass

    This is brilliant, I've been wanting something like this for ages :D

    I've hooked it up on my PMS (0.9.9.5 running on Synology DS412+) with the Remote Indexer running on my desktop (Windows 8.1 Core i7). 

    I can confirm that refreshing an individual show works (subject to the known bug) and refreshing an entire show works as well. When I refresh my whole TV section however, the Remote Indexer doesn't receive any work to do? Obviously I can just queue shows up one at a time (that will still be much faster than having my NAS index them) but I'd rather queue all of the work in the section. The docs suggest that this should work? Any ideas how to troubleshoot?

    I'm just kicking off a section refresh using the button at the top right of the web UI. Should I be doing a Force Refresh instead? Or is there perhaps a requirement that something has changed in the section since the last refresh, in order to trigger your code?

  • benaddersonbenadderson Posts: 110Members, Plex Pass Plex Pass

    OK, so I tried a force refresh of the TV section, and that started generating work for the Remote Indexer :)

    Unfortunately I've now begun seeing errors from the Remote Indexer console :( I've attached a copy of the console output, but the specific error seems to be:

    error: [Errno 10054] An existing connection was forcibly closed by the remote host
    

    Restarting both the PMS and my desktop hasn't helped.

    Any ideas? Should I clear out the Queue folder, and go back to indexing one show at a time?

  • dane22dane22 Posts: 10,195Members, Plex Pass, Plex Ninja Plex Ninja
    benadderson wrote on February 28 2014, 3:45 PM: »

    OK, so I tried a force refresh of the TV section, and that started generating work for the Remote Indexer :)

    Unfortunately I've now begun seeing errors from the Remote Indexer console :( I've attached a copy of the console output, but the specific error seems to be:

    error: [Errno 10054] An existing connection was forcibly closed by the remote host
    

    Restarting both the PMS and my desktop hasn't helped.

    Any ideas? Should I clear out the Queue folder, and go back to indexing one show at a time?

    Please delete the remidx.ini file and start the remidx again.

    When going through the setup, set logging to debug

    Reproduce, and upload the remidx.log file.

    No need to nuke the queue dir

    /T

    I hate bugs - Tommy Lee Jones, MIB
    Join me in developing: epg-dk, str2utf-8, remidx, ExportTools, WebTools
    Support the Samsung Client:Donate
    Guides I use: Media Naming Guide, Local subtitles, Log-Files, QNAP FAQ, The Plex Dance

    NO Support via PM, unless called by me

  • jkenneyjkenney Posts: 19Members, Plex Pass Plex Pass

    This is ingenious, I will have to give this a go this weekend.  I have a slightly huge library (as I use the plex server and plex to distribute compressed tv recordings through the house - my super TiVo if you will), any issues with generating the index files for say ~350 shows with a total of 11176 (as of right now) episodes?  How big do the index files get?

    ----

    Slightly crazy plex user.

    Plex Server: Dell R815 with 4 x 2.3GHz 16-Core Opteron 6376, 128GB DDR3 ECC RAM, 6 x 120GB PNY XLR8 SSD in Raid 5 (H700), 2 x 120GB OCZ RevoDrives in Raid 0, 6gbps SAS card connected to 2 x MSA60 12 Bay SAS Enclosures (total 24 x 3TB Drives)
    Why? 2x TiVo Roamio (Over the air) and 2x TiVo Roamio Pro (cable), shows extracted and compressed via kmttg and distributed via plex through the house.  Recording up to 20 things at once!!!  I love plex...
«134
Sign In or Register to comment.