Proof of Concept: Plex with Kinect

Dev Community Dev/API Corner
1

Hi all,



Recently I’ve been experimenting with controlling Plex via an Xbox Kinect and am happy to present the first working prototype; codename HandyPlex.



See it in action here:

[media]

http://www.youtube.com/watch?v=wFM7pVMLIY4

[/media]



The source code is available at GitHub: https://github.com/jopdeklein/Handyplex



Currently the following gestures/actions are supported:

Global:


<br />
Wave        Start Gesture Recognition<br />


*Navigation Mode:*

<br />
Swipe Left      Navigate Left<br />
Swipe Right     Navigate Right<br />
Swipe Up        Navigate Up<br />
Swipe Down      Navigate Down<br />
Push            Action / Play<br />
Pull            Back<br />


*Playback Mode:*

<br />
Swipe Left      Rewind<br />
Swipe Right     Fast Forward<br />
Swipe Up        Volume Up<br />
Swipe Down      Volume Down<br />
Push            Play/Pause<br />
Pull            Stop<br />


**Background**
I liked what Kinemote was doing but since they've recently closed down the source I feel there's room for an open source solution which also works on the Mac.
My initial goal was to get a working prototype and figure out whether the Kinect can be used as an efficient / fun way of controlling Plex, thus the current feature set is quite basic.
I am not yet convinced that this is the best way to control Plex, but feel it's a decent alternative at least (and it impresses the dinner guests :)).

**Technical Details**
The current solution is written in Python and works with the SensorKinect drivers and OpenNI/NITE framework/middleware.
Skeleton data is retrieved via the OSC protocol provided by OSCeleton and pyOsceleton, on top of which I wrote straightforward gesture recognition algorithms that recognizes simple gestures such as swipes, push/pull and a 'wave' gesture.
I currently use the Plex HTTP API to control Plex, but it should be easy to wrap it in a plug-in that sends key commands so it can be configured using a remote.xml.

I've only tested HandyPlex on Plex 9.3.4 and will have to investigate forward compatibility.

**Future Plans**
I am planning to extend HandyPlex to support multi-hand gestures and support a 'I am done' gesture to signal that you want it to stop recognizing gestures until you re-initialize the session with a 'wave' gesture.
The current OSCeleton-based implementation has quite some layers and dependencies, which I believe I could do without (and could be hampering performance/complicate the distribution process unnecessarily), I'll likely move to a OpenCV-based implementation in the future.

The current state is very much 'Proof of Concept', it works for my daily use but requires significant improvements before I would distribute this as a plug-in.
I would like to take this opportunity to gauge whether there is enough interest from the Plex community to warrant further development.
If so, I am planning to open source the code and provide it via a public repository, so if there's any tech savvy users willing to beta test it would be much appreciated!
2

Very cool project. Congratulations on this proof of concept!

3

That is awesome! I’m with you on weather it’s the best way to control a tv or not but it’s seriously awesome! Well done. Makes me want to go get a kinetic. If I do I’d love to be beta tester.

4

+1 for beta testing, please keep us informed.

5

Hey great work !!

Looks very promising, and it seems to work perfectly already.



I would love to control Plex with my kinect. That’s a very good idea.



If you need some beta testers I can be part of it.

6

This is amazing - if this is released I will buy a Kinect just for Plex!



I hope you will develop this further, so a “beginner friendly” plugin/app can be released :slight_smile:

7

Thanks for all the positive feedback, reason enough to warrant further development.

My next effort will be to consolidate some of the external dependencies to make it easier to distribute.

I will keep you posted here in this thread when the code is available on GitHub.

8

Awesome job…look forward to seeing this progress!

9

So how are you doing the main tracking reference for commands? are you basing it on where the hand was waved in the X, Y, Z area or basing it off relation to the head or shoulder? I am working on voice controll and will be moving to motion control once i get voice working how i want



I just put a voice demo up:

[media]http://www.youtube.com/watch?v=danXUIksXg8[/media]

10


Currently I am using hand tracking with the wave position as a initial starting point, as the full skeleton tracking of NITE currently needs a rather elaborate 'callibration pose' which is too complicated/cumbersome when you just want to watch a movie from your couch. I might move toward full skeleton tracking if the NITE project makes it easier to callibrate user, which I believe they are working on now.

Before each gesture you need to hold your hand in one position for a short time, and that position will be used as the starting point for gesture recognition.

I am very interested in looking whether at some point we can merge our efforts, as voice commands was next on my list :) I'm cleaning up the code and writing some documentation after the weekend, I aim to get the source online somewhere next week.

Your demo looks really good by the way, seems to be working well. Are you using the Mac OS built-in speech recognition?
11

I am actually using the Kinect windows SDK Beta 2 with C# With the actual release of a Kinect for Windows SDK and the PC tuned hardware i have a feeling the partial skeleton tracking will become alot easier sometime early next year (few months from now).



i know what i am working on right now is mainly for windows systems where as you are building a more mac centered version, have you tested your setup on windows yet to do cross platform? I do not have a mac to do any cross development/testing and apple only allows running a Virtual Apple OS on apple hardware, which makes cross development impossible for me.



I would love to see how you are doing most of your control with the API since there is no documentation on the client side (that i have found) right now the only command i am passing via the API is to jump to a section and to display the notifications.

12

“It’s written in Python”.



:slight_smile: You made me smile.



You can grab me on @kaichanvong on twitter or github (same username: kaichanvong (Kai Chan Vong ) · GitHub) - I’d love to help out with this.



Is the repo up somewhere or could I have a play with the code? Hope to hear from you soon.

13

The source code is now available on GitHub: https://github.com/jopdeklein/Handyplex

Please see the supplied README for installation and usage instructions - let me know if you need any help getting started.



I currently tested it on only one machine, I’ll be happy to receive feedback from users with different OS/Plex configurations.



As this is the first version, there are a lot of things I would like to improve over time. I’ll keep you updated on the progress here, suggestions are of course more than welcome.



Enjoy!

14

Thanks jop. I just ordered a kinetic just for this!

15

This is awesome. Going to be keeping an eye on the progress!

16

I received some PM’s with questions regarding the installation procedure and instead of answering them individually I’d rather share them here.



As mentioned before, you’ll need to be build some projects from the command line and have some basic knowledge of cloning projects with Git.

Since you will compile the projects you’ll need a C++ compiler, if you install XCode you’ll be good to go.



Although a bit daunting looking at first, fortunately my experience with installing the drivers and middleware have been good - everything worked out of the box on my Snow Leopard machine.

The following are the steps I took to get it working on Mac OS X Snow Leopard, but in theory it should work on Lion as well.



For each of the following required projects I assume you will clone them in a directory, eg: ~/Kinect and build them from source from the directory you cloned them in.

[list]

[] OpenKinect Drivers

[] OpenNI Framework

[] NITE Middleware

[] OSCeleton

[/list]



Installing Drivers and Middleware

The drivers are used to transfer the Kinect to your computer, and the raw data is translated to skeleton/hand objects via the middleware.

I used the installation instructions as specified in the OpenKinect README: https://github.com/avin2/SensorKinect, here’s a recap of the steps I took to get it working:



[list]

Install the OpenNI framework (unstable version): http://www.openni.org/Downloads/OpenNIModules.aspx


$ sudo ./install.sh


[*] Install the OpenKinect Drivers: https://github.com/avin2/SensorKinect, as per the instructions:
Building Sensor:
1) Go into the directory: "Platform/Linux-x86/CreateRedist".
Run the script: 

$ ./RedistMaker


This will compile everything and create a redist package in the "Platform/Linux-x86/Redist" directory.

2) Go into the directory: "Platform/Linux-x86/Redist".
Run the script: 

$ sudo ./install.sh



[*] Install NITE: http://www.openni.org/downloadfiles/opennimodules/openni-compliant-middleware-binaries/33-latest-unstable

$ sudo ./install.sh

[/list]

Installing OSCeleton
OSCeleton transfers the skeleton/hand objects via the OSC Protocol, which in turn will be picked up by Handyplex.

Install OSCeleton: https://github.com/Sensebloom/OSCeleton

$ ./make


Now we're almost there, you might want to run OSCeleton in window mode to test whether the drivers and middleware have been installed correctly.
You should see a window with the video feed transferred from the Kinect:

$ ./OSCeleton -w


Installing Handyplex
Handyplex receives the hand data via the OSC protocol, for which a couple of dependencies are needed (defined in requirements.txt).
You can either install these by hand, or use pip to install them based on the requirements.txt (recommended).

Assuming pip is installed you can install the dependencies:

$ pip install -r requirements.txt


If all went well you should now be able to use Handyplex! :)

Run Handyplex:

$ python handyplex.py


As said, it looks more daunting than it actually is, feel free to contact me if you need more help - preferably in this thread so we can share experiences.
17

I am really impressed with what you’ve done here. This is the only solution to control PLEX with the Kinect that I can find. I just had a suggestion and a question.



I agree that it is annoying to have no way of turning off the gesture recognition when you are watching something and I had an idea of how to do that. Can your setup be used to recognized both hands? I thought maybe a wave is good to initialize but to turn it off you could wave both hands in a sweeping motion towards each other. If it makes anymore sense it would be a reverse wave with both hands.



I wouldn’t mind trying to add recognition for this but being unfamiliar with this setup (if all goes as planned I will setup your scripts later today) I didn’t know if I would need other config files to properly associate that sort of gesture with a matching command to stop all gestures from being recognized.

18

So I did get HandyPlex to run. A few notes though:



First, since I am not a developer I had to download everything and it seemed as though a few config files or installations were missing from your original instructions. I needed to first install MacPorts and enable ports with these commands:



sudo port install libtool



sudo port install libusb-devel +universal

___________________________________________________________________________________



Then to make NITE work, there was actually a product key needed that was buried on another blog.



This should be it: 0KOIk2JeIBYClPWVnMoRKn5cdY4=



Also, to install and run OSCeleton I needed to get “setuptools” to then install pip. (pip is used to install OSCeleton) [not intuitive to install but available on the Python website]

___________________________________________________________________________________



In case anyone else has trouble installing you may want to add this info to your original post.





Aside from a few install snags it’s running really well. Thanks!

19

AdobeEditor, thanks for being brave and congratulations for getting it all to work. I’ve updated the detailed install instructions with your additions, thanks.



As for the ‘stop’ gesture, I was also thinking about multi-hand gestures, but at the time OSCeleton/pyOSCeleton didn’t support it, and skeleton recognition required a cumbersome calibration pose. I read the latest NITE middleware can now recognize skeletons without calibration pose, which makes it much easier to user skeleton tracking (as opposed to hand tracking). This allows for much more natural interaction by detecting multiple joints - for example you can detect where the hands are in relation to each other, or to the head/torso.



My plan is to move closer towards NITE and eliminate the OSCeleton/pyOSCeleton layer, in order to gain more flexibility when NITE adds new features, and to be able to talk to the Kinect device directly.

In the mean time, feel free to fork Handyplex on Github and see if you can add the stop gesture, I think your best bet is another ‘wave’ gesture but you’ll likely need to update OSCeleton to have it detect this. Another option would be to define a boundary and when the hand leaves it, stop recognition.



I’m on a short vacation from today on, when I’m back I’ll investigate upgrading to the latest NITE libraries and decide on future plans.



Thanks for your insights, hope you have fun with Handyplex!

20

Glad I could add to the effort. I will try to manipulate the existing gestures before I start working on a stop gesture. I didn’t realize till using it that its not difficult to make it lose your hand so I’m not that set on having a stop gesture before tweaking the navigation commands.



I did have a question or two though (I understand if you don’t respond right away because you should enjoy a day or two of vacation!). I was trying to find a way to make a script that could run the proper commands in terminal for OSCeleton without having to change directories and retype the commands every time I want to run it. I know its possible but I am not very familiar with programming on Mac OSX (I built a hackintosh before owning my current Mac Mini but I’m still learning how to program in a new environment).



Is there an easy way to run this and possibly the python file with as single click from a file on the desktop (just to simplify the process).



Let me know what you think when you have time. I’ll update the thread with any new progress…hopefully we can get these voice commands ported to Mac OSX for full control!