C# .NET - C# Scraping a website - Asked By vimalkumar panneerselvam on 06-Feb-15 05:41 AM

website as it navigates through each page it depends on client side scripts after the page loads .

As we are scraping a secure webpage the HTML from the server C# doesn't execute any client scripts like ajax or javascript as it not a real web browser.

I dn't know how complex to call these AJAX calls from the website into our C# application. I have analysed through Fiddler or IE to see how the requests are made but there is no simple solution for this problem am afraid.

Any suggestions welcome ?

Robbe Morris replied to vimalkumar panneerselvam on 06-Feb-15 08:06 AM

You will have to load the page into some sort of browser web page processing system. Absent that, you are out of luck. You could try the webbrowser control but I've often had issues getting it to run under a certain version that supports JSON and/or AJAX calls. It is possible that there are other open source browser engines to load the pages into but I'm not aware of the specific one's that support JSON or AJAX.