I asked this question on stackoverflow and was told "Use Wireshark". I'm a noob, and see nothing on your site I recognize as related to my question. I'm using perl to scrape data from a website. Now the website has changed, so it only loads the data via AJAX in response to entry into a textbox and clicking a submit button. How can my perl program simulate that user input and capture the resulting new page? Is this something that Wireshark can help me with? asked 16 Mar '14, 00:32  hsfrey |
3 Answers:
The guys at StackOverflow probably meant that you could use Wireshark to capture the traffic of a normal website visit (via manually browsing to it), and than capture what your perl script does and compare the requests and answer content. At least that's what I would do to make sure my script mimics the manual way of accessing the site. Very often a site not only looks at the URL that you pass in the request, but also the referrer, user agent, cookies and other details carried in the HTTP header of the request. So Wireshark can show you what they are when surfing on the site with a browser. Use these values and mimic them in your script, and it should work. answered 16 Mar '14, 04:58  Jasper ♦♦ |
well, what was you question on stackoverflow !?!
how did you figure out the structure of the web site in the first place? Just repeat that step for the new site structure
It will show you requests and responses to/from the site. However, in your case any browser developer tool (like Firebug for Firefox) would be much better. Regards answered 16 Mar '14, 05:12  Kurt Knochner ♦ edited 16 Mar '14, 13:15 |
You might also want to take a look at an intercepting web proxy like Burpsuite (http://portswigger.net/burp/). You can intercept and examine the packets flowing in each direction in real time, and use that information to make adjustments in your perl script. The free version of Burpsuite should be sufficient for the task. answered 16 Mar '14, 07:10  griff |
