Hi I am trying to work out the video URL but the site is using javascript to create an Iframe then inside the iframe more javascript creates the video url so with using ElmentFromURL its not even creating the iframe and not executing any of the javascript. I can easily work out what the Iframe url should be but if I try and link to this URL it detects that it has not been loaded from the previous page so don’t display anything. so if I can get it to run all the javascript and build the page correctly getting the video URL will be easy.
If your final video URL requires the Referer value set within the request header, then you are out of luck. Plex Framework does not support the Referer input when playing streams. It only allows the User-Agent and Cookies to be set. Note: I said Final video URL.
Sometimes a site doesn’t need the Referer. Many times you can get away with only specifying a browser like User-Agent that is consistent between calls.
A good practice is to test what headers you’ll need is by using curl in a terminal or cmd (-I so only response headers are read).
curl -I "final_video_url"
Normally I setup a test script outside Plex Framework to get the video URL (as-well-as other channel functions before they are Plexified). Then I test the final video URL with curl and then with curl again but within https://codeanywhere.com so my IP is different. If it fails, then I start adding in headers one-by-one so I know which are absolutely needed. Also by changing my request IP, I find out if the final video URL has a generate hash tied to my IP. If so, then the URL can only be direct play/stream within the Plex Media Server’s home network.
As-far-as JavaScript, Plex is built off of a modified version of Python 2.7 and uses no JavaScript locally. If you really need to use JavaScript, then look into PyExecJS. Note you will need to specify to your user base that a valid JavaScript Runtime Engine will be need while the server is running.
Please note I’m no official developer, these have been my observations over the past year of Plex channel development. If anyone has corrections to what I’ve said, then please let me know, as the info would benefit me as well.
@Twoure said:
If your final video URL requires theReferervalue set within the request header, then you are out of luck. Plex Framework does not support theRefererinput when playing streams. It only allows theUser-AgentandCookiesto be set. Note: I said Final video URL.Sometimes a site doesn’t need the
Referer. Many times you can get away with only specifying a browser likeUser-Agentthat is consistent between calls.A good practice is to test what headers you’ll need is by using
curlin a terminal or cmd (-Iso only response headers are read).curl -I "final_video_url"Normally I setup a test script outside Plex Framework to get the video URL (as-well-as other channel functions before they are Plexified). Then I test the final video URL with
curland then withcurlagain but within https://codeanywhere.com so my IP is different. If it fails, then I start adding in headers one-by-one so I know which are absolutely needed. Also by changing my request IP, I find out if the final video URL has a generate hash tied to my IP. If so, then the URL can only be direct play/stream within the Plex Media Server’s home network.As-far-as JavaScript, Plex is built off of a modified version of Python 2.7 and uses no JavaScript locally. If you really need to use JavaScript, then look into PyExecJS. Note you will need to specify to your user base that a valid JavaScript Runtime Engine will be need while the server is running.
Please note I’m no official developer, these have been my observations over the past year of Plex channel development. If anyone has corrections to what I’ve said, then please let me know, as the info would benefit me as well.
Hi Twoure, Thanks for the help, the end url is actually a google video so the stream dont need to pass referer over, the problem I was getting is this.
the main page uses javascript to load an Iframe with a an id of the video. the iframe’s page javascript takes this and has a hash value in the javascript the var name changes daily but its all ways the 3rd one in the javascript, a md5 hash is then made of the video var and then this other hash value from the javascript is then put in front of this a new md5 hash is created but with some elements of its location shifted around, and then the original video var and now this new secirity var are passed to a different web site and this then returns a google video.
I can now get the video var to pass over and I have managed to re write what the javascript is doing to make the security var, the problem I have at the moment is the Iframe page that contains the javascript with the hash in the javascript wont fetch with http.request or html.elementfromurl, from testing with firefox, the only requirement for the page to load is I need to set the referer as the correct page that loads it, no cookies or other values are needed to be passed.
I tried to add , headers={‘Referer’: ‘http://TheUrlOfCallingPage’} in the HTTP.Request but it dont seem to do anything.
considering writing a php file on one of my web servers that I can call to handle this and pass the value back, unless someone can think of a better solution
@robyice said:
I can now get the video var to pass over and I have managed to re write what the javascript is doing to make the security var, the problem I have at the moment is the Iframe page that contains the javascript with the hash in the javascript wont fetch with http.request or html.elementfromurl, from testing with firefox, the only requirement for the page to load is I need to set the referer as the correct page that loads it, no cookies or other values are needed to be passed.I tried to add , headers={‘Referer’: ‘http://TheUrlOfCallingPage’} in the HTTP.Request but it dont seem to do anything.
Just to be sure (sorry), did you test the URL in question via urllib2 or requests in a separate script outside the Plex Framework (i.e. a script separate from your channel and PMS altogether)? Plex uses urllib2 to build its requests, but requests can be easier for a quick test (just depends on your preference).
Doing a simple test with pure python (no Plex) should give you a better idea of what headers are needed. Not sure how you’re testing for headers, but Firefox and Chrome both cache header values and do not always show the info (or indicate that info was cached) while using the developer tools.
Some other methods for retrieving video URLs:
- If page has an embed address, sometimes can skip obfuscation and
POSTrequests when navigating to an embed link - If the page has an app released with it, sometimes the way the app retrieves video links is different and may be more straight forward.
- Change the
User-Agentto look like a mobile device. Many websites serve mobile devices different stream information as-well-as bypass some of the obfuscation
Otherwise there is not much more I can do without seeing an example webpage/script.
If you need to catch the redirected URL, then a modified version of my code here can be used as follows:
import urllib2
import contextlib
def get_url_redirect(url, http_headers):
class NoRedirection(urllib2.HTTPErrorProcessor):
def http_response(self, request, response):
return response
https_response = http_response
opener = urllib2.build_opener(NoRedirection)
opener.addheaders = [(k, http_headers[k]) for k in http_headers.keys()]
try:
with contextlib.closing(opener.open(url)) as resp:
h = resp.headers
return h['Location'] if 'Location' in h.keys() else (h['location'] if 'location' in h.keys() else url)
except urllib2.HTTPError, e:
Log.Error('* get_url_redirect: %s' %str(e))
return url
Original code taken from 2nd part of the answer here.
Make sure the User-Agent and Referer are both set (urllib2 will need at least the User-Agent set to work properly), as-well-as any other headers that may be needed to access the redirected URL.
If you know you can expect a redirect URL and you want the value of that URL (instead of simply following it and grab the contents), you can also do it with framework only functions (without urllib):
try:
page = HTTP.Request(url, follow_redirects=False).content
except Ex.RedirectError, e:
if 'Location' in e.headers:
new_url = e.headers['Location']
I know this is resolved now, but I’d like to mention, for anybody who might be still looking to execute Javascript, I’ve stumbled upon Js2Py library which can parse and interpret (almost) any Javascript code without any other dependencies, which, I think, is just amazing! That said, I didn’t try to run the whole web page with it, but it worked perfectly with just a portion of it where the page decodes the video URL using some weird algorithm.
