Help finding form element with xpath

Dev Community Dev/API Corner
1

I think I picked a hard project to cut my teeth on…



I’m working on a plugin for rohwrestling.com, to view the episodes they stream from there. The site does require a login in order to be able to watch episodes, but they have an odd method of logging in. Their login form has a hidden form element with some kind of hash value it uses to authenticate, and both the element ID and the value are set to the hash. Since the ID is different each time the page loads, I’m having a hell of a time trying to get the element.



Please see the attached image for a DOM inspector view of the element I need to get.



I’ve tried the following code to get the value of the element:


	loginpage = HTML.ElementFromURL('https://www.rohwrestling.com/user?destination=node')<br />
	buildid = loginpage.xpath("//div[@id='content-area']/input[@type='hidden']")[0].get('value')


As well as the following:

	loginpage = HTML.ElementFromURL('https://www.rohwrestling.com/user?destination=node')<br />
	buildid = loginpage.xpath("//div[@id='content-area']/input[@name='form_build_id']")[0].get('value')


Unfortunately, neither works. It seems like the xpath function isn't finding the element in either case; both times, it returns an empty list, causing an "IndexError: list index out of range" exception. Are there any other methods I can try to get the element? I'm kind of new to page scraping, so be gentle.

If you're curious, the page I'm trying to scrape is at https://www.rohwrestling.com/user?destination=node
2

Hi godkarmachine!

The input element with name ‘form_build_id’ is not a direct descendant of the div with id ‘content-area’. That’s why //div*/input won’t get you any results, but //div//*input will:



loginpage = HTML.ElementFromURL('https://www.rohwrestling.com/user?destination=node')<br />
buildid = loginpage.xpath('//div[@id="content-area"]//input[@name="form_build_id"]")[0].get('value')


http://www.tizag.com/xmlTutorial/xpathdescendant.php

Hope this helps!
3

So I was chasing the wrong problem all this time. Like I said, I don’t have much experience with scraping (or Python, for that matter).



I tried your code, and it worked (well, after I tweaked the quotes a bit; that last double-quote should be a single-quote). Thanks for the help!

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.