[REL] RemIdx


#1

WARNING: There has been reported an issue, that the indexes is out of sync, and breaks if fast forward is pressed.

Until resolved, please discard this tool!

 

 

 

It's a REMOTE INDEXER

 

This means, that you now can do your indexing from a remote workstation, that is cabled with your PMS, without your PMS been drained CPU wise

Making it even more usefull for low powered NAS boxes

 

And even better.....You can at any time stop the Remote Indexer process on the remote workstation, and when started again, it'll pick up where it left.

 

Remote Indexer has currently only been tested on OpenSuse and Windows, but I suspect, that it should run on any flavor of Windows, Linux and Mac.

 

Remote Indexer bundle (The part that runs on your PMS), has currently only been tested on QNAP

 

To grap it, take a peek here: https://github.com/ukdtom/RemIdx

 

Best Regards

 

Tommy

 

P.S. Feedbacks to this thread is more than welcome, both good and bad news

P.P.S. If you are a dev, then please fork away, and check back code....I need help here ;)

 

Edit: 28. feb. 2014: Version 0.0.1.4 released (And this is a major on, if running Windows)

Edit: 26. feb. 2014: Version 0.0.1.3 released

Edit: 25. feb. 2014: Version 0.0.1.2 released

Edit: 23. feb. 2014: Version 0.0.1.1 released

Edit: 22. feb. 2014: Version 0.0.1.0 released

Edit: 22. feb. 2014: Version 0.0.0.7 released

Edit: 22. feb. 2014: Version 0.0.0.6 released


#2

Updated documentation with installation notes

Edit: 21-feb 2014 14.00 GMT+1 Yet another Doc update


#3

Sadly, made a slight error, that caused the PMS agent to not pick up new contents on a forced refresh.

Corrected now in V0.0.0.6


#4

V0.0.0.7:

Added a thread limiter for the Remote Indexer, so you now can adjust how many CPU cores to use during screenshot creation


#5

Happy to say, that I now went from Alpha to Beta :D

Windows is now a supported platform

Version is 0.0.1.0

And find the link in the first post in this thread

Feedback are welcome

/Tommy


#6

Updated to V0.0.1.1

Fixed issue with missing registration under TV-Shows


#7

Updated to V0.0.1.2

Added a configuration wizard to the Remote Indexer, so no longer the need to fiddle with the python file ;)

Download info here

https://forums.plex.tv/topic/99413-rel-remidx/


#8

Tommy,

I just started testing this.

I got this:

2014-02-26 10:03:01,190 (35a0) :  CRITICAL (core:561) - Exception in the update function of agent named 'RemIdx (Movies)', called with guid 'com.plexapp.agents.imdb://tt2294629?lang=en' (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 970, in _update
    agent.update(obj, media, lang, force)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 41, in update
    GetMediaInfoMovie(media.id, media.title)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 69, in GetMediaInfoMovie
    sections = XML.ElementFromURL(myNewURL).xpath('//MediaPart')
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\parsekit.py", line 344, in ElementFromURL
    method=method,
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api
etworkkit.py", line 67, in _http_request
    req = self._core.networking.http_request(url, *args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components
etworking.py", line 422, in http_request
    return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components
etworking.py", line 147, in __init__
    self.load()
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components
etworking.py", line 183, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 410, in open
    response = meth(req, response)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Unauthorized

This is because I have require authentication on local networks turned on. I had to add my PMS local ip address to the list of networks that are allowed without auth. 

ex: 192.168.0.21/255.255.255.255

After that change it seems to be working. Might be worth a note in your readme.

Chris

Edit:

I believe the Indexer PC also needs to be in the allowed without auth list. I already have mine in the list, so I'm not sure. 


#9

Now I'm getting this in the indexer:

Traceback (most recent call last):
  File "C:\Users\Chris\Documents\GitHub\RemIdx\Remote_Indexer\RemIdx.py", line 548, in 
    main()
  File "C:\Users\Chris\Documents\GitHub\RemIdx\Remote_Indexer\RemIdx.py", line 538, in main
    GenJPGs(sMyDir)
  File "C:\Users\Chris\Documents\GitHub\RemIdx\Remote_Indexer\RemIdx.py", line 236, in GenJPGs
    data = json.load(data_file)
  File "C:\Python27\lib\json\__init__.py", line 290, in load
    **kw)
  File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I'll dig a little deeper later today hopefully.

Edit:

The .bundle file in the Work dir is empty. Not sure how that happened.


#10

Edit:

The .bundle file in the Work dir is empty. Not sure how that happened.

Seen that as well very periodic, and could be trapped -> cleanout workdir


#11

Tommy,

I just started testing this.

I got this:

2014-02-26 10:03:01,190 (35a0) :  CRITICAL (core:561) - Exception in the update function of agent named 'RemIdx (Movies)', called with guid 'com.plexapp.agents.imdb://tt2294629?lang=en' (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 970, in _update
    agent.update(obj, media, lang, force)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 41, in update
    GetMediaInfoMovie(media.id, media.title)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\RemIdx.bundle\Contents\Code\__init__.py", line 69, in GetMediaInfoMovie
    sections = XML.ElementFromURL(myNewURL).xpath('//MediaPart')
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\parsekit.py", line 344, in ElementFromURL
    method=method,
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api
etworkkit.py", line 67, in _http_request
    req = self._core.networking.http_request(url, *args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components
etworking.py", line 422, in http_request
    return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components
etworking.py", line 147, in __init__
    self.load()
  File "C:\Users\Administrator\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components
etworking.py", line 183, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 410, in open
    response = meth(req, response)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Unauthorized

This is because I have require authentication on local networks turned on. I had to add my PMS local ip address to the list of networks that are allowed without auth. 

ex: 192.168.0.21/255.255.255.255

After that change it seems to be working. Might be worth a note in your readme.

Nice catch....never thought of that, and will do


Edit:

I believe the Indexer PC also needs to be in the allowed without auth list. I already have mine in the list, so I'm not sure. 

What do you mean here?

Best regards, and huge thanks for taking the time here

/Tommy


#12

Nice catch....never thought of that, and will do

What do you mean here?

Best regards, and huge thanks for taking the time here

/Tommy

The PC running RemIdx.py I believe has to be in the allowed without auth list.

For example my allowed without auth list looks like:

127.0.0.1/255.255.255.255,192.168.0.1/255.255.255.255,192.168.0.21/255.255.255.255

 
192.168.0.1     Is my pc running RemIdx.py
192.168.0.21   My PMS
 
 
Chris

#13

The PC running RemIdx.py I believe has to be in the allowed without auth list.

For example my allowed without auth list looks like:

127.0.0.1/255.255.255.255,192.168.0.1/255.255.255.255,192.168.0.21/255.255.255.255

 
192.168.0.1     Is my pc running RemIdx.py
192.168.0.21   My PMS
 
 
Chris

Got it


#14

Updated to V0.0.1.3

Fixed issue with bundle file in Work directory sometimes was empty

Updated doc to reflect the needs to allow the Remote Indexer access without auth

Both above was discovered by srazer


#15

Updated to V0.0.1.4

Before telling what's new here, I would like to tell you, that this project now actually will raise from the dark and into heaven, cuz I'm been blessed with the company of srazer, that is now working with me on this.

And this version is all his, and here's what he added:

- Remote Indexer:
-- Process will now run as low priority to prevent from bogging down the system.
-- Framework now to build win32 executable using PyInstaller.
-- ffmpeg's log level can now be set in the config file.
-- Allow default config values to be set by pressing enter.

Overall.....If on Windows, simply launch an exe file....

If on Mac or Linux -> Nerd mode ;)

/Tommy and Chris


#16

Oh shoot  B)

I'm not all that good. I can google and copy and past code fairly well though.... lol  :ph34r:


#17

Wrong place. lol.


#18

This is brilliant, I've been wanting something like this for ages :D

I've hooked it up on my PMS (0.9.9.5 running on Synology DS412+) with the Remote Indexer running on my desktop (Windows 8.1 Core i7). 

I can confirm that refreshing an individual show works (subject to the known bug) and refreshing an entire show works as well. When I refresh my whole TV section however, the Remote Indexer doesn't receive any work to do? Obviously I can just queue shows up one at a time (that will still be much faster than having my NAS index them) but I'd rather queue all of the work in the section. The docs suggest that this should work? Any ideas how to troubleshoot?

I'm just kicking off a section refresh using the button at the top right of the web UI. Should I be doing a Force Refresh instead? Or is there perhaps a requirement that something has changed in the section since the last refresh, in order to trigger your code?


#19

OK, so I tried a force refresh of the TV section, and that started generating work for the Remote Indexer :)

Unfortunately I've now begun seeing errors from the Remote Indexer console :( I've attached a copy of the console output, but the specific error seems to be:

error: [Errno 10054] An existing connection was forcibly closed by the remote host

Restarting both the PMS and my desktop hasn't helped.

Any ideas? Should I clear out the Queue folder, and go back to indexing one show at a time?


#20

OK, so I tried a force refresh of the TV section, and that started generating work for the Remote Indexer :)

Unfortunately I've now begun seeing errors from the Remote Indexer console :( I've attached a copy of the console output, but the specific error seems to be:

error: [Errno 10054] An existing connection was forcibly closed by the remote host

Restarting both the PMS and my desktop hasn't helped.

Any ideas? Should I clear out the Queue folder, and go back to indexing one show at a time?

Please delete the remidx.ini file and start the remidx again.

When going through the setup, set logging to debug

Reproduce, and upload the remidx.log file.

No need to nuke the queue dir

/T