PPRuNe Forums - View Single Post - Automated fare query engine
View Single Post
Old 18th March 2009 | 00:54
  #3 (permalink)  
davidjohnson6
 
Joined: Sep 2008
Posts: 5,691
Likes: 24
From: Blighty
bnt - many thanks for your comment.

I was thinking of writing an app in something like C#, using the HttpWebRequest class to simulate the action of a user filling in a form and pressing the 'Submit' button. I've already done some sniffing of network packets - I can understand some of the data but not all of it. The HTML that comes back can then be handled as a stream, either saved to disk for later analysis, or parsed on-the-fly to extract the relevant bits of data (flight times and prices) and the results sent to a SQL Server database.

The point about the screen scraping, is that airlines have a habit of changing their websites in small ways from time to time. While humans are usually quite good at handling this change, this often necessitates a complete rewrite of the code that posts and parses the data to/from the airline website, making a screen scrape process potentially less maintenance.

Any more thoughts would of course be welcome
davidjohnson6 is offline  
Reply