PPRuNe Forums - View Single Post - Automated fare query engine
View Single Post
Old 19th March 2009 | 03:08
  #7 (permalink)  
Jofm5
 
Joined: Jan 2008
Posts: 525
Likes: 0
From: LONDON
OK, I think I got the wrong end of the stick - My impression was that you were wanting to screen scrape with a view to having your own offering.

From what I think now you just wish to compare results from the given query from multiple sites which would require the http post to provide the parameters.

To perform a http post from c# is quite easy, just google http post c# and you will find numerous examples. Unfortunately its not going to be so easy to simulate a user posting from code.

The big problem you will encounter is that not all the session information that will be used in the query will be contained within the posted variables (quite often to deliberately avoid what your trying to do). Session information may be stored in session variables, cookies and hidden controls on the form.

The hidden controls on the form are easy as they are parseable and so long as you you provide the content of hidden form elements in your post as well as visible ones then it will work. Session variables and cookies are much harder as if it is an ASP website some may be stored server side and others you just wont have access to unless you know specific cookie information to interrogate locally and manipulate.

This is not to say it cannot be done, all you need to do is emulate the functionality of a browser - but you are getting into the territory of manipulating the raw http feed yourself which will probably render what you are trying to do being more trouble than it is worth.


There is no real easy answer to what you want to do other than either trial and error or investigating whether a publish API (Web Service) is available - being such a highly competitive market I would not be surprised that even if the sites use XHTML rather than straight HTML they changed the tags to scupper bots scraping the sites.

Sorry for not being optimistic - its a cool challenge to write for but I think most sites will be trying to make your job as hard as possible.
Jofm5 is offline  
Reply