C# screen scraping windows application

You'll need to update your code each time the source website changes their markup structure. Screen scraping doesn't play well with Javascript. If the target website is using any sort of dynamic script to manipulate the webpage you're going to have a very hard time scraping it.

It's easy to grab the HTTP response, it's a lot harder to scrape what the browser displays in response to client-side script contained in that response. Make it as easy as possible to change the patterns you look for. If possible, store the patterns as text files or in a resource file somewhere. Make it very easy for other developers or yourself in 3 months to understand what markup you expect to find.

Validate input and throw meaningful exceptions. In your parsing code, take care to make your exceptions very helpful. The target site will change on you, and when that happens you want your error messages to tell you not only what part of the code failed, but why it failed. Mention both the pattern you're looking for AND the text you're comparing against.

Write lots of automated tests. You want it to be very easy to run your scraper in a non-destructive fashion because you will be doing a lot of iterative development to get the patterns right. Automate as much testing as you can, it will pay off in the long run. Consider a browser automation tool like Watin. If you require complex interactions with the target website it might be easier to write your scraper from the point of view of the browser itself, rather than mucking with the HTTP requests and responses by hand.

Use Html Agility Pack. It handles poorly and malformed HTML. It lets you query with XPath, making it very easy to find the data you're looking for.

One thing you have to consider about scraping web sites is that they are beyond your control and can change frequently and significantly. If you do go with scraping the fact of change ought to part of your overall strategy. Just one thing to note, a few people have mentioned pulling down the website as XML and then using XPath to iterate through the nodes.

From a practical perspective I have written dozens of "web-interactive" apps over the years , I finally settled on Watin combined with CSQuery. Watin provide the basics of browser automation interacting with buttons etc , while CSQuery lets you use jQuery style syntax to parse the page content.

New Post. What do I do? Follow Post Reply. Well, the first thing to do is post to the correct newsgroup. This is the C language group. Aside from that, screen scraping is done pretty much the same way regardless of whether from a windows app or a web app. Ashot Geodakov. What do you mean by "scraping text"? Getting all dialog's child control's captions? That'll do it Good advice.

I would expect screen scraping a win form to be very different. This discussion thread is closed Start new discussion. Similar topics. GetDefaultProxy ; request. GetResponse ; if response. The Regex used is not a complete solution to the Email pattern but works with most email addresses.

It is only used to demonstrate the technique. Groups[ "emails" ]. Add c. Below is a snapshot of the Win Email Extractor tool I have built to demonstrate the technique.

As an example, we are scraping Email addresses of US Senators from a public site that hosts those Email addresses.

Sometimes issues like Proxy, firewall can cause the tool to not work. View All. Screen Scraping using System. Shantanu Updated date Sep 30,