So, I recently made my first channel. It was working well until this week. Now the element returned by HTML.ElementFromUrl are different from the element I see when I inspect the page in my browser. The Elements are essentially the same except that it missing the ones I need to make it work.
I tried to add this in the start function but it doesn't change anything
HTTP.Headers['User-Agent'] = user-agent
Is there a way to fix this problem?
Now the element returned by HTML.ElementFromUrl are different from the element I see when I inspect the page in my browser.
Make sure to not use the "Inspect Element" options from your browser, but look at the site's source with the "View Source" option. "Inspect Element" will give you a look at the rendered DOM, and all kinds of things may have been changed in the DOM after the initial HTML page has been loaded (through javascript most likely).
It sounds like the elements you are looking for might be being processed/added by javascript after the page load (which doesn't happen within the framework).
I suggest using something like curl or wget on the command line to retrieve the URL in question and then looking at that source, that's basically what you're going to have to work with within the channel. As far as I know there's no way to have the framework process any javascript based stuff for you unfortunately. I say to use curl or wget because then there's no chance that any script will execute -- depending on the browser sometimes I've seen the page source view be different from the raw source you get with curl and the like.
Thank you for your quick response.
So, if I understood correctly, I have to find the script that process this element and try to include in my channel some code that will do this process to retrieve that element.
