Ok, I will PM you with the Log file and if you like I can send you the plugin (send me a message if you want it), the only thing which would not really matter unless you selected a Hosting site that used captcha graphic toying with OCR I need to add code to check for OS and add OCR for Linux and Mac. I also have to go in and break down my ServiceCode.pys into smaller sections right now it's HUGE. I still have to finish the login section for the website adding a section to get your messages and your video posts. You might look at the code and call me crazy... ;)
I am also using Requests to process forms and I am also thinking of using this other guys version of requests that supports Socks5 so I can get to some international sites that are blocked by region and use the Tor Network to access those vids. Just haven't gotten that far yet with my coding...
Ok, I sent you the info you requested zipped up to your messages along with a few questions and thoughts if you didn't mind...
I do have another question going back to the Preferences and setting up the DefaultPrefs.json and ServicePrefs.json. I noticed on Roku you can have sub menues in the Preferences but I can't find in Plex the docs if you can do this with the json file settings? In other words if I want to create a sub section in the Preferences click it then it goes to a new page with more settings/options.
Thanks again for the help and great community!!!!!!
While waiting on the results and thoughts of the other issues and questions...
EpisodeObject I noticed I can't set generes or MPAA TV ratings (TV-G, TV-PG, TV-14, TV-MA, ect...) they are blank.
Any ideas thanks again!!!
Haven’t had a chance to dig into your logs yet.
Re: Actors metadata, I’m trying to get some details on that. I recently ran into a similar situation and just left the list of actors out. Ideally, MovieObjects should allow us to set the cast metadata I just don’t know if/how it’s possible.
I’m pretty sure that the Plex plugin framework does not allow for multi-level prefs. I doubt that it’s in the roadmap for implementation (not that I have access to said roadmap).
Re: EpisodeObject ratings and genres, a few questions.
First, when you say they come up blank, do you mean your values are not displayed in the client or that the values don’t appear in the XML. The two are not the same, some clients may not display all available metadata (properly).
Second, are you setting the rating or the content_rating? The first takes a number (int or maybe float) as a argument. The second takes a string such as “PG”, “G”, etc. I’m not sure if the “TV-” should have any impact or not. If that’s causing issues, it might be a bug or a something that should be implemented.
Third, are you using an array of strings for genres?
I appologise for not making myself more plain on the last post. I was taking about when you see content_rating displayed on the Roku you don't see TV ratings only Movie ratings will display. Such as G, PG, R, NC-17, NR... That is if the value is set for one of those. If set for TV-G, TV-PG, TV-14, ect... will not display any content_rating....
The genres is an array of strings, example genres = ['Scifi', 'Horror', 'Comedy'] what ever the genres are for the TV show or movie. If you try to set genres in the EpisodeObject it give you an error this can't be set. But works great for the MovieObject. The strange thing is it will allow you to set the genres in the URL services for the EpisodeObject but it still will not display the values.
UPDATE: guest_stars = ['Bob Blart', Darth Cooper'] will get you to see actors displayed on the EpisodeObject but not the MovieObject, so getting closer I guess I will use this for now until we find out anything else...
One thing I noticed still with this plugin the new plex media server I still get an error using any web browser trying to go to the section where it displays the over all content/description of the TV Show/Movie/Video page before the video player page. I have looked at my error logs and it does not denote a darn thing that I see but works fine on the Roku box.
I still need help with getting Request Headers and not the Responce Headers of a site. Like I said I am using the requests.post to process forms and some forms want the request header cookies sent. I have tried using session = requests.session() then session.post(...) Only cookies that are stored are the Responce Header Cookies. This worked ok on designing a login function for the website and designed a loop that put the session.cookies into the HTTP.Headers['Cookie'] and I also stored them in the Dict but not sure to do with the Dict. The only thing even though it did store the cookies when I go to a my user page such as inbox example:
inbox = HTML.ElementFromURL(MYMESSAGES_PAGE).xpath('//div[@id="maincontent4"]/form')[3]
The site acts like I am not logged in and gives an index error because it kicks me back to the login page. I have checked and my login function does log into the site.
I haven't made any headway regarding the actors/cast yet. I think that for TV, genres and perhaps content_rating are only accessible at the TVSeriesObject level. I don't really use them, but essentially they're just a special type of DirectoryObjects. If your TV menus follow a predictable hierarchy and the metadata is available up front, you should be able to implement it as DirectoryObject (list of available series) > TVSeriesObject (specific series) > [SeasonObject (optional) >] EpisodeObject. Or some variation of that.
You can specify whatever headers you want to send, including cookies, as a dict and pass it to any framework function that makes an HTTP request as an extra 'headers' arg. Like so:
session_headers = {'Cookie':HTTP.GetCookiesForURL(MYMESSAGES_PAGE)}
inbox = HTML.ElementFromURL(MYMESSAGES_PAGE, headers=session_headers).xpath('//div[@id="maincontent4"]/form')[3]
Regarding the errors (without logged errors) when using Plex/Web, I would try watching the http traffic when navigating the menus. That might reveal the source of the glitch whether it's in your code or the Web client.
The site does break it down into Season and Next Episode and then Player/Host Links Page. But Genre is not listed until Player/Host Links Page. I take that back I could use the genre in the section the person chooses from and see if it will display it then and change my cdoe from DirectoryObject to SeasonObject. See if I can pass genre metadata there... I will have to go and look up the Values the SeasonObject can take.
Ok, I tried your idea on:
session_headers = {'Cookie':HTTP.GetCookiesForURL(MYMESSAGES_PAGE)}
inbox = HTML.ElementFromURL(MYMESSAGES_PAGE, headers=session_headers).xpath('//div[@id="maincontent4"]/form')[3]
I even tried this from the Dict I stored the cookies in the login:
session_headers = {'Cookie': Dict['_movie2k_uid']}
inbox = HTML.ElementFromURL(MYMESSAGES_PAGE, headers=session_headers).xpath('//div[@id="maincontent4"]/form')[3]
The site must be looking for more Cookies than what is stored. If you look at the Request Headers using firebug plugin for firefox it has what I have stored plus a ton of other cookie values. When I tried your version to GetCookiesForURL it returned nothing. This is exactly like what I have run into with other hosting sites I can't get all the cookies that are set by the site. Unless there is a piece of python code that can do this I am at a loss.....
Any ideas and again thanks for your help!!!
Try using HTTP.CookiesForURL rather than GetCookiesForURL. I keep forgetting that GetCookiesForURL is deprecated and the docs are out of date.
Ok tried HTTP.CookiesForURL still returned None... I bet HTTP.CookiesForURL is only looking a the Responce Headers and not the Request Headers. Need a python script that looks at the request headers.
A lot of the hosting sites want information from Request Headers passed back to the page when you do a post or get to process the page correctly. Which I know this is what we are trying to do with setting headers and passing them back in to get the page. Key here is reading the Request Headers. Any ideas????
AFAIK, any site that requires a specific cookie in the request headers will provide the cookie in the response headers. For headers other than session-specific cookies, you should be able to sniff those joy by watching HTTP traffic in your browser, then hardcode them in your plugin. I’m not aware of any code that grabs Request headers. Generally speaking, you have control over the request headers but will often need to grab the response headers.
I was able to get the inbox to pull up doing it this way: using Requests:
session_cookies = Dict['_movie2k_uid']
session_headers = {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.3", "Accept-Encoding": "gzip,deflate,sdch", "Accept-Language": "en-US,en;q=0.8", "Connection": "keep-alive", "Host": "www.movie2k.to", "Referer": "http://www.movie2k.to", "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31"}
inbox = requests.get(MYMESSAGES_PAGE, headers=session_headers, cookies=session_cookies)
You have to sepperate the cookeis from the header and pass it in to make it return the page correctly.
This will return the page I want if I use the requests add on code, only thing it returns the page as text if you do inbox.content. Does Plex Media Server have a python script that will allow you to pass the content into it and parse it like you do with HTML.ElementFromURL instead of passing the URL in pass it the page content? I know this is 3rd party python code.
So basically I did as you said it created the session cookies from the login function but I still had to use session = requests.session() to set up a session to get the Resonce Header cookies.
I wish they had a python siffer code for the request headers. I did find today this one kewl website that will show you that info love to know how they coded it:
http://www.askapache.com/online-tools/http-headers-tool/
It will show you the cookies that are set that are not in the responce headers I might try and use it for some of my other hosting sites that have weird cookies set that you don't get in the resonce headers. If you use firebug plugin for firefox or google chrome's web browser content viewer pressing ctrl-shift-i on the PC it will pop up then you go to the Network section on both of them and select the webpage and it will show you Headers (responce & request), Post, ect...
I really do thank you for your help caused me to think more about how to solve these problems with cookies!!!
I think I might have answered my own question by using:
HTML.ElementFromString(string)
Converts string to a HTML element object.
I will test this out when I get back, have to run for a bit...... Oh and you think this will work let me know too and thanks again for all your help!!!!
I'm not quite following what you are trying to accomplish, but sending certain headers or cookies can be done with HTML.ElementFromURL, XML.ElementFromURL and JSON.ObjectFromURL in one single go, like so:
url = "http://www.blah.com/"
headers = {
'User-Agent': 'Blah/1.0',
'Cookie': 'test=true;another_cookie_value=thisone;omg=wtfbbq'
}
html = HTML.ElementFromURL(url, headers=headers)
I have tried exactly what you stated html = HTML.ElementFromURL(url, headers=headers) or if I do it exactly the same way for html = requests.get(url, headers=headers) it will not process the page cookies correctly and send me back to the login page... But, if is do html = requests.get(url, headers=headers, cookies=cookies) this will process the cookie data and go into the webpage requested. So however requests.get() is processing the cookie data gets me into the page. I will try to hurry up and send you a copy of the plugin so you can see I am not crazy.. :blink:
I tried this to see if I could process the returned page contents:
html = requests.get(url, headers=headers, cookies=cookies)
inbox = HTML.ElementFromString(html.content).xpath('//div[@id="maincontent4"]/form')[3]
When I try to use the HTML.ElementFromString the plugin error log generates three pages of these lines and stop processing:
013-04-29 16:48:19,512 (5048) : WARNING (runtime:1052) - Generating a callback path for a function with no route:
2013-04-29 16:48:19,513 (5048) : WARNING (runtime:1052) - Generating a callback path for a function with no route:
2013-04-29 16:48:19,515 (5048) : WARNING (runtime:1052) - Generating a callback path for a function with no route:
2013-04-29 16:48:19,516 (5048) : WARNING (runtime:1052) - Generating a callback path for a function with no route:
2013-04-29 16:48:19,517 (5048) : WARNING (runtime:1052) - Generating a callback path for a function with no route:
Have you ever used the function HTML.ElementFromString?
Looks like it should work from this post:
http://forums.plexapp.com/index.php/topic/48790-channel-development-setup/#entry365385
What is this html = requests.get you're talking about? It doesn't look like standard Plex framework functions. Do you import urllib or some other module(s)?
Yes I am importing Requests.
http://docs.python-requests.org/en/latest/index.html
The reason a lot of the Hosting sites and including the site Movie2k.to that points to the hosting sites I can't get the Plex's built url processing for cookies or forms to work because it will just send you back to the login or main page. The only thing that I have found that will process these sites more correctly is using Requests not saying 100% full proof. And, it is not; there are still about 4 or 5 sites that still give me issues even using Requests and it goes back to dynamic cookies that are generated for the Request Headers so even using request.session() will not get all the cookies that are set because they are not listed in the Responce Headers. I am thinking about using a branch of the Requests that support socks v5 proxy so I can get to couple more sites that are blocked to the USA. I am not knocking Plex developers hard work just issues I have ran into and this was the fix. I know this is 3rd party code and is not supported by your devs.
Requests does return a fully formed webpage why I was trying to use HTML.ElementFromString to process it so I can use xpath. Way cleaner than using a ton of splits like I have had to do. If I can get HTML.ElementFromString to work I will go back and clean up my code and us it to process the Hosting Sites.
Here is the example to process Socks procy branch of Requests then what you can do is use the Tor Network:
https://github.com/kennethreitz/requests/pull/478
Example of the page that is returned when Requests is processed correctly:
<div style="position:absolute; margin-top:59px; margin-left:30px; color:#FFF; font-family: Arial, Helvetica, sans-serif; font-size: 10px;">
Your movie, tv shows and blockbuster database! </div>
<div style="position:absolute; margin-top:65px; margin-left:950px;"><a href="http://www.movie2k.to/index.php"><img src="http://img.movie2k.to/img/us_flag.png" border="0" alt="Watch free english cinema movies online" title="Watch free english cinema movies online" /></a></div>
<div style="position:absolute; margin-top:90px; margin-left:950px;"><a href="http://www.movie2k.to/index.php?lang=de"><img src="http://img.movie2k.to/img/ger_flag.png" border="0" alt="Schau kostenlose deutsche Kinofilme online" title="Schau kostenlose deutsche Kinofilme online" /></a></div>
<div style="position:absolute; margin-top:125px; margin-left:936px;"><a href="http://www.movie2k.to/index.php?lang=fr"><img src="http://img.movie2k.to/img/french.png" border="0" alt="Watch free french cinema movies online" title="Watch free french cinema movies online" /></a></div>
<div style="position:absolute; margin-top:125px; margin-left:963px;"><a href="http://www.movie2k.to/index.php?lang=es"><img src="http://img.movie2k.to/img/spain.png" border="0" alt="Watch free spanish cinema movies online" title="Watch free spanish cinema movies online" /></a></div>
<div style="position:absolute; margin-top:145px; margin-left:936px;"><a href="http://www.movie2k.to/index.php?lang=it"><img src="http://img.movie2k.to/img/italia.png" border="0" alt="Watch free italian cinema movies online" title="Watch free italian cinema movies online" /></a></div>
<div style="position:absolute; margin-top:145px; margin-left:963px;"><a href="http://www.movie2k.to/index.php?lang=jp"><img src="http://img.movie2k.to/img/japan.png" border="0" alt="Watch free japanese cinema movies online" title="Watch free japanese cinema movies online" /></a></div>
<div style="position:absolute; margin-top:165px; margin-left:936px;"><a href="http://www.movie2k.to/index.php?lang=tr"><img src="http://img.movie2k.to/img/turkey.png" border="0" alt="Watch free turkish cinema movies online" title="Watch free turkish cinema movies online" /></a></div>
<div style="position:absolute; margin-top:165px; margin-left:963px;"><a href="http://www.movie2k.to/index.php?lang=ru"><img src="http://img.movie2k.to/img/russia.png" border="0" alt="Watch free russia cinema movies online" title="Watch free russia cinema movies online" /></a></div>
|
<span style="font-size:10px; color: #FFF; font-family: Arial, Helvetica, sans-serif;">Search</span><br />
<div class="days3">
<div id="menue">
<div class="aussen">
<a class="test123" href="index.php?lang=en"><span class="menutag">Movies</span></a>
<a class="innen-1" href="index.php?lang=en">Cinema movies</a>
<a class="innen" href="./movies-updates.html">Latest updates</a>
<a class="innen" href="./movies-all.html">All movies</a>
<a class="innen" href="./genres-movies.html">Genres</a>
<a class="innen" href="./rss-movies.html">RSS feed</a>
</div>
<div class="aussen">
<a class="test123" href="tvshows_featured.php"><span class="menutag">TV shows</span></a>
<a class="innen-1" href="./tvshows_featured.php">Featured</a>
<a class="innen" href="./tvshows-updates.html">Latest updates</a>
<a class="innen" href="./tvshows-all.html">All TV shows</a>
<a class="innen" href="./genres-tvshows.html">Genres</a>
<a class="innen" href="./rss-tvshows.html">RSS feed</a>
</div>
<div class="aussen">
<a class="test123" href="xxx-updates.html"><span class="menutag">XXX movies</span></a>
<a class="innen-1" href="./xxx-updates.html">Latest updates</a>
<a class="innen" href="./xxx-all.html">All movies</a>
<a class="innen" href="./genres-xxx.html">Genres</a>
<a class="innen" href="rss-xxx.html">RSS feed</a>
</div>
<div class="aussen">
<span class="menutag">User
<span id="newmessage"></span>
<SCRIPT>
$('#newmessage').load('http://www.movie2k.to/newmessage_check.php');
</SCRIPT>
</span>
<a class="innen-1" href="./ui.php?ua=movie">Add a movie</a>
<a class="innen" href="./ui.php?ua=tvshow">Add a tvshow</a>
<a class="innen" href="./ui.php?ua=myuploads">My uploads</a>
<a class="innen" href="./modify.php">Edit account</a>
<a class="innen" href="./ui.php?ua=messages_inbox">Messages
<span id="newmessage2"></span>
<SCRIPT>
$('#newmessage2').load('http://www.movie2k.to/newmessage_check.php');
</SCRIPT>
</a>
<a class="innen" href="./ui.php?ua=logout">Logout</a>
</div>
</div>
</div><!-- menue -->
<FORM method=POST action="movies.php?list=search" style="display:inline; font-family: Arial, Helvetica, sans-serif; z-index:3;">
<INPUT name="search" onKeyUp="showAutoComplete(this.value)" onBlur="window.setTimeout('closeAutoComplete()',300);" autocomplete="off"> <INPUT type=submit value="GO">
<!--<INPUT name="search"> <INPUT type=submit value="GO"> -->
<div id="searchAutoComplete" style="display:none;position:absolute;left:30px;top:146px; z-index:3;"></div>
</FORM>
|
td { font-size:12px; }
Inbox
| |
From |
Date |
Subject |
|
joecowboy |
04/17/13 16:20 |
Movie2k |
<INPUT type=submit name="del" value="DELETE MARKED" style="width:150px">
<INPUT type=submit name="read" value="MARK AS READ" style="width:150px">
<INPUT type=submit name="unread" value="MARK AS UNREAD" style="width:150px">
</FORM>
</div>
<div style="display:none">
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-24229017-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</div>
<SCRIPT>
$.get("http://forum.movie2k.to/setlangcookie.php?lang=en");
</SCRIPT>
<BR><BR><BR><BR><BR>
Your log snippet doesn’t include any info about the actual failure in HTML.ElementFromString(). I have run into issues before when the webpage includes some broken formatting or unusual encoding but it almost always throws an exception that includes a pointer to the portion of the string that caused the error.
I said errors out which it's more of a time out on the Roku box and the Plex Web. I got to noticing it didn't just fill up 3log files worth of:
2013-04-29 22:42:22,704 (5908) : WARNING (runtime:1052) - Generating a callback path for a function with no route:
Looked more like an infanant loop of filling up log file page after log file page. It kept going on even after Roku box errored and the Plex Web errored. Only way to stop it from writing the log files was to reset the Plex Media Server. How I was able to get that out put was to Log it before it did the HTML.ElementFromString(). Fun fun testing got to love it....
Your thoughts?
UPDATE:
I used the code else where in the plugin to process another page that needed static cookies and it worked:
req = requests.get(url, headers=headers, cookies=cookies)
NUMPAGES = len(HTML.ElementFromString(req.content).xpath('//div[@id="maincontent4"]/div[@id="boxgrey"]')) + 1
SOOOOOO...
I guess you are right something is malformed in the return of that inbox page... I will have to go through it figure it out and fix it to be able to use the HTML.ElementFromString()
I will test this else where........
I tested the TVShowObject() to set the genres in and it still didn't display it on the next to the page right before the player. What's the name of that page that displays all the meta data about a tv show, video or movie?? So I know what to call it ;)
Just an update...
I'm not sure what to call it either, so I go with the "summary page". the info displayed there is whatever you return from the MetadataObjectForURL in your URL Service code
