Capturing traffic to any host within a specific domain

Question

I need to capture HTTP GET and POST requests to (responses from aren't of interest) any host that is within .example.com. This will be a long running capture and the machine is fairly heavily used, so I'm keenly interested in controlling what is captured rather than just what is displayed. There are probably hundreds of hosts within .example.com, some I won't even know about. So a wildcard is key. These requests may come from a browser or other type of application, so I need to use a lower level approach. My initial impression was that this isn't possible via a capture filter, but I thought I'd ping the experts before giving up. Is it possible with Wireshark at all?

Note: although I'm currently interested in just HTTP requests, if there is a way to do a similar thing without limiting it to HTTP (which conveniently provides the HOST header) or a specific direction, I would be interested in the more general case too :)

Thanks in advance!

Answer 1

Capture filters can't work with wildcards nor can they handle re-assembly. Your best bet is to use dumpcap using the "-b filesize" option to split data accross files. You can then use tshark with a display filter to extract the packets of interest.

Answer 2

-1

Check out these examples from http://wiki.wireshark.org/CaptureFilters and tailor to your situation...

Capture traffic to a range of IP addresses: dst net 192.168.0.0/24 or dst net 192.168.0.0 mask 255.255.255.0

I haven't researched it, but I'm guessing 0/24 means all IPs ending ".0" up to and including ".24", but it might mean twentyfour IPs starting at 0 (ie, 0-23).

The mask should capture anything going to 192.168.0.* where * is from 0 to 255.

Also in the above capture examples: Capture HTTP GET requests. This looks for the bytes 'G', 'E', 'T', and ' ' (hex values 47, 45, 54, and 20) just after the TCP header. "tcp[12:1] & 0xf0) >> 2" figures out the TCP header length. (From Jefferson Ogata via the tcpdump-workers mailing list.)

port 80 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420

So all you have to do is find your IPs for .example.com and combine that with the GET and POST filters.

answered 03 Apr '12, 11:41

PReinie
15●2●2●4
accept rate: 0%

Thank you for your time and reply. A point I failed to explicitly mention (sorry) is that *.example.com aren't neatly tucked into known blocks of IP Addresses. I don't know what the specific hosts are and they are surely spread across many providers/blocks.

At first I thought that approach to identifying a GET wouldn't work in some (especially POST) cases where the HTTP request is spread across multiple packets. However, your comment made me dig a bit deeper and I've just come across some discussions regarding the reassemble options. I'm not familiar with Wireshark at that level yet, but perhaps now that I have something to search for and read I'll figure out some way to implement a Host: pattern match. Note: This was too big to fit into a "comment", so I was forced to make it an "answer".

(03 Apr '12, 13:06) TheBitsCometh

So maybe you have to make it a find/discover phase, first capturing "many" of the known IPs from a list of many-more, and then capturing of the known set those GET and POST communications you need. (Rome wasn't built in a day.)

Add on more IPs as you deem fit. Have you ever whittled? You start with a large piece (big chunk), and end up with a much (usually) smaller piece (also a chunk, of lesser dimensions), the end being what you want. If it's not, repeat (with a new piece, not gluing it back together). Granted, not many people whittle now, but it's a concept used in, shall we say, "artisan crafting", and sometimes, that's what we (as engineers) have to do, in a different world). Hell, Jobs wouldn't have given it another thunk! (Thunk being the past-tense of thought, in my words. ;) )

(05 Apr '12, 22:02) PReinie

I admire and acknowledge the benefits of iterative approaches :) The domains/systems of interest are large and dynamic (think global ad/content delivery networks). So although I am hesitant to rule out any IP blocks, it indeed may prove useful to refine things once I have some sense of what they are. I'm still mulling over approaches. In addition to what we've been discussing, I'm also considering:

1) For HTTP, route through an HTTP proxy that supports logging based on HTTP header pattern matching. 2) For HTTPS, consider the possibilities of setting up a MITM HTTPS proxy with similar logging capability. Note: Entirely my computers and my local network. No other users would be affected by the MITM.

While thinking about this, one approach did come to mind which I'll share for fun. Use a DNS proxy to redirect "hostnames of interest" traffic to local private IP Addresses corresponding to a proxy that captures the data of interest and then forwards it to the real destination IP Address that was saved by the DNS proxy.

(05 Apr '12, 23:25) TheBitsCometh