Extracting SOAP XML Payload

Here's Lua that extracts the XML fields to a file (with a dotted line in between fields). I tested it against your pcap in tshark and Wireshark. For tshark, run: tshark -r sethwilson.pcap -R "tcp and data"

Note that the file is written in append mode to:

• UN*X: ~/.wireshark/temp.xml
• Windows XP: C:\Documents and Settings\your_username\Application Data\Wireshark\temp.xml

-- tap uses dfilter for tcp data and ignores retransmissions
local tap       = Listener.new(nil, "tcp && data && !tcp.analysis.retransmission")
local xml_field = Field.new("data")
local file      = nil

-- #######################################################################
-- # If not already open, this opens a file for writing (append mode)
-- #######################################################################
local function open_file()
    if not file then
        local path = USER_DIR .. "/temp.xml"

        print("opening file:", path)
        file = assert(io.open(path, "a"), 
                    "Can't open file for writing")
    end
end

local HTML_REQ = {
    ["HTTP"] = 1,
    ["GET "] = 1,
    ["PUT "] = 1,
    ["POST"] = 1,
}

-- #######################################################################
-- # Extracts the XML field from the buffer and writes the field to file
-- #######################################################################
local function handle_xml(pinfo, tvb)
    if not file then
        print("no file...ignoring packet")
        return
    end

    local fieldinfo = xml_field()
    local xmldata   = tvb(fieldinfo.offset):string()

    -- some of these packets start w/HTTP header...skip to XML
    local starts = xmldata:sub(1,4)
    if HTML_REQ[starts] ~= nil then
        local pos = string.find(xmldata, "<%?xml version")
        if not pos then
            return
        end

        xmldata = xmldata:sub(pos)
    end

    print(xmldata)
    print("\n\n-- #"..pinfo.number.." ---------------------------------------------------\n\n")

    file:write(xmldata)
    file:write("\n\n-- #"..pinfo.number.." ---------------------------------------------------\n\n")
end

-- #######################################################################
-- # tap.packet() is called to notify the Listener of a packet that
-- # matches its filter rule ("xml" in this case). This can be called
-- # multiple times before tap.draw().
-- #######################################################################
function tap.packet(pinfo, tvb)
    print("tap.packet", "#"..pinfo.number)

    -- XXX: Compensate for no tap.reset() in tshark
    if not gui_enabled() then open_file() end

    -- wrap the handler in a pcall() in case an error occurs
    local ok, msg = pcall(  function() 
                                handle_xml(pinfo,tvb) 
                            end )

    -- print any error and bow out
    if not ok then
        print("wtf!", msg)
    end

end

-- #######################################################################
-- # tap.draw() is called to notify the Listener to "draw" its results
-- # that were accumulated in tap.packet(). This is normally called after
-- # tap.packet(), based on "Preferences > Statistics > Tap update interval".
-- #######################################################################
function tap.draw()
    print("tap.draw")

    -- flush toilet (NOTE: When $file is garbage collected, it's 
    -- automatically flushed and closed...that doesn't mean we
    -- can't do it sooner to free resources.)
    if file then
        print("closing file")
        file:close()
        file = nil
    end
end

-- #######################################################################
-- # tap.reset() is called to notify the Listener to reset any variables
-- # or counters in preparation for a packet (passed to tap.packet()).
-- # This can be called multiple times before a packet is even seen.
-- #
-- # XXX: tshark doesn't call this function, but Wireshark does. Bug?
-- #######################################################################
function tap.reset()
    print("tap.reset")
    open_file()
end

For some reason, I had problems combining the proto variable function call along with the tostring() call So I broke them up and added a check in between.

On your line:
  local s_xml_cdata = tostring(xml_cdata_f())
try changing to
  local l_xml_cdata = f_xml_cdata()
  if l_xml_cdata == nil then return end
  s_xml_cdata = tostring(l_xml_cdata)

XML CDATA is often encoded with &lt; (<) &gt; (>) and other codes. There may be a better way to do all that follows with an XML parser, but I spend enough time just getting Wireshark to behave as advertised. I don't have time to dive into Lua nuances. So I manually transfer the CDATA to a string as shown above. Then I use string.gsub() to do searches and replaces:

s_xml_cdata = string.gsub(s_xml_cdata, "&lt;", "<")
s_xml_cdata = string.gsub(s_xml_cdata, "&gt;", ">")
s_xml_cdata = string.gsub(s_xml_cdata, "&#xA;", "")
s_xml_cdata = string.gsub(s_xml_cdata, "&#xD;", "")
s_xml_cdata = string.gsub(s_xml_cdata, "\n", "")
local la_cdata = BreakUpLongLines(s_xml_cdata, ">")

This last line is a call to my function BreakUpLongLines(string_xml_cdata, string split_character). Pass in your string with the reformed XML data, and the character you want to do splits on. In this case, it's basically the greater than symbol. Here is the BreakUpLongLines() function.

--utility to break up long lines into smaller ones
function BreakUpLongLines (str, sep)
    --Array to return each of the elements
    vals = {}; valindex = 1; word = ""
    local strSize = #str
    local offset =0
    while strSize > offset do
        svar, evar=string.find(str, sep, offset)
        if evar == nil then return vals end
        if evar > strSize then return vals end
        vals[valindex] = string.sub(str, offset,evar)
        valindex = valindex+1
        offset=evar+1
    end
    return vals
end

So this function will take any long line and break it by the sep character, which for XML happens to be the greater than symbol. It returns an array of all your lines. Use a simple For loop to print your lines.

for kvar,vvar in next, la_cdata do
    g2stree:add(kvar, la_cdata[kvar])
end

Comments and improvements welcome!

To dump XML documents from a tap/listener:

local tap = Listener.new(nil, "xml")
local xml_field = Field.new("xml")

function tap.packet (pinfo, tvb)

    -- Extract the XML field (which contains the full XML document).
    --
    -- NOTE: `xml_field()` returns a `FieldInfo` object. See this link for more info:
    -- http://www.wireshark.org/docs/wsug_html_chunked/lua_module_Field.html#lua_class_FieldInfo

    local fieldinfo = xml_field()

    -- `print()` refuses to print the field because it "may contain invalid
    -- characters". The particular subfield that's causing the problem is
    -- `fieldinfo.label` (not sure why). One would think that `fieldinfo.value`
    -- would contain the full XML and `fieldinfo.len` would be the document's
    -- byte length, but not true (bug?).
    -- 
    -- No worries. We can parse the string from the `tvb` ourselves. `fieldinfo.offset`
    -- tells us where in the `tvb` the XML document begins. So, all we need to do
    -- is convert the bytes from the offset to the end of the buffer into a string.
    -- (This assumes that the XML document is the last field in the packet.)
    --
    -- See http://wiki.wireshark.org/LuaAPI/Tvb#tvbrange:string.28.29

    print( tvb(fieldinfo.offset):string() )
end