This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

Why does my Lua dissector cause Wireshark to consume all memory?

0

My Lua dissector works well if I have only a few packets within a .pcap file. The reassembly is working, and the bytes are as expected once the last PDU is reached. The more packets there are, the faster (and larger) the memory consumption of Wireshark becomes (until it runs out and bails on me). I am not a software developer, so please be gentle. Your assistance is appreciated.

SOMETHING_TCP_PORT = 30003
SOMETHING = Proto("SOMETHING","SOMETHING")

function SOMETHING.init() end

function SOMETHING.dissector(buffer, pinfo, tree) if buffer:len() > 3 then if string.upper(tostring(buffer(0, 4))) == "0401F0CE" then tvbLength = 20728 elseif string.upper(tostring(buffer(0, 4))) == "0501F0CE" then tvbLength = 12024 elseif string.upper(tostring(buffer(0, 4))) == "2201F0CE" then tvbLength = 7028 elseif string.upper(tostring(buffer(0, 4))) == "2B01F0CE" then tvbLength = 7028 else tvbLength = buffer:len() end end

pinfo.cols.info = "SOMETHING " pinfo.cols.protocol = "SOMETHING"

local n = pinfo.desegment_offset or 0 while true do local nextPDU = n + tvbLength

if nextPDU > buffer:len() then
    pinfo.desegment_len = nextPDU - buffer:len()
    pinfo.desegment_offset = n
    return
end

if string.upper(tostring(buffer(n, 4))) == "2201F0CE" or string.upper(tostring(buffer(n, 4))) == "2B01F0CE" then
    if string.upper(tostring(buffer(n, 4))) == "2201F0CE" then
        pinfo.cols.info:append("- Some Message ")

        subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

        subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
        n = n + 4
    elseif string.upper(tostring(buffer(n, 4))) == "2B01F0CE" then
        pinfo.cols.info:append("- Some Other Message ")

        subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

        subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
        n = n + 4
    else
        return
    end

    subtree:add(buffer(n, 4), "  Message size = " .. buffer(n, 4):le_uint() .. " bytes")
    if buffer(n, 4):le_uint() ~= 7028 then
        subtree:add(buffer(n, 4), "  Message size is incorrect!")
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buffer:len() then
        return
    end
end

if string.upper(tostring(buffer(n, 4))) == "0401F0CE" then
    pinfo.cols.info:append("- Some Message ")

    subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

    subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
    n = n + 4

    subtree:add(buffer(n, 4), "  Message size = " .. buffer(n, 4):le_uint() .. " bytes")
    if buffer(n, 4):le_uint() ~= 20728 then
        subtree:add(buffer(n, 4), "  Message size is incorrect!")
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buffer:len() then
        return
    end
end

if string.upper(tostring(buffer(n, 4))) == "0501F0CE" then
    pinfo.cols.info:append("- Some Message ")

    subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

    subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
    n = n + 4

    subtree:add(buffer(n, 4), "  Message size = " .. buffer(n, 4):le_uint() .. " bytes")
    if buffer(n, 4):le_uint() ~= 12024 then
        subtree:add(buffer(n, 4), "  Message size is incorrect!")
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buffer:len() then
        return
    end
end

end end

tcp_table = DissectorTable.get("tcp.port")

tcp_table:add(SOMETHING_TCP_PORT,SOMETHING)

asked 01 Apr ‘15, 13:12

JustSomeNoob's gravatar image

JustSomeNoob
6113
accept rate: 0%

edited 01 Apr ‘15, 13:13


One Answer:

0

I believe what's happening is wireshark is getting stuck in your while-loop, and never returning from it. This can happen because in certain cases your code will never have nextPDU be greater than buffer:len(). For example, if your dissector sees a TCP segment that has the first 4 bytes not match one of the hex strings you expect to see, then it will set tvbLength = buffer:len(), enter the while-loop with the variable n equal to 0, and thus have a nextPDU equal to buffer:len()... but that's equal-to, not greater-than, so the while loop won't return... ever. And since the if-then statements within the while loop allocate memory (by creating multiple TvbRanges), the memory usage will keep growing and growing.

So you're going to have to change your code logic to prevent being stuck in that while-loop forever. Like for example verify the nextPDU value changes on each loop, or something.


Also, your code is fairly inefficient - it creates a bunch of TvbRange objects and Lua strings multiple times instead of once.

For example, at the top of the dissector function:

if buffer:len() > 3 then
    if string.upper(tostring(buffer(0, 4))) == "0401F0CE" then
        tvbLength = 20728
    elseif string.upper(tostring(buffer(0, 4))) == "0501F0CE" then
        tvbLength = 12024
    elseif string.upper(tostring(buffer(0, 4))) == "2201F0CE" then
        tvbLength = 7028
    elseif string.upper(tostring(buffer(0, 4))) == "2B01F0CE" then
        tvbLength = 7028
    else
        tvbLength = buffer:len()
    end
end

There are three things in that if-then block that cause memory to be allocated:

  1. buffer(0,4) creates a TvbRange object in C-code and Lua.
  2. tostring() of the TvbRange creates a string in Lua.
  3. string.upper() creates another string in Lua.

You call those three things up to four times in the above if-then block, depending on whether the elseif clauses get executed. For numbers 2 and 3 above, they will only create one string each in Lua for all four if/elseif clauses because they result in the same string and Lua internally only stores one copy of a given string; but still it's extra processing that isn't necessary. Instead, you should do this:

if buffer:len() > 3 then
    local msgType = string.upper(tostring(buffer(0, 4)))
    if msgType == "0401F0CE" then
        tvbLength = 20728
    elseif msgType == "0501F0CE" then
        tvbLength = 12024
    elseif msgType == "2201F0CE" then
        tvbLength = 7028
    elseif msgType == "2B01F0CE" then
        tvbLength = 7028
    else
        tvbLength = buffer:len()
    end
end

That both reduces the memory allocated, and improves performance.

By the way, what if buffer:len() is not greater than 3? Then tvbLength isn't set to anything.


Next you have this:

    if string.upper(tostring(buffer(n, 4))) == "2201F0CE" or string.upper(tostring(buffer(n, 4))) == "2B01F0CE" then
        if string.upper(tostring(buffer(n, 4))) == "2201F0CE" then
            pinfo.cols.info:append("- Some Message ")
        subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

        subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
        n = n + 4
    elseif string.upper(tostring(buffer(n, 4))) == "2B01F0CE" then
        pinfo.cols.info:append("- Some Other Message ")

        subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

        subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
        n = n + 4
    else
        return
    end

    subtree:add(buffer(n, 4), "  Message size = " .. buffer(n, 4):le_uint() .. " bytes")
    if buffer(n, 4):le_uint() ~= 7028 then
        subtree:add(buffer(n, 4), "  Message size is incorrect!")
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buffer:len() then
        return
    end
end</code></pre><p>Again, you're creating multiple <code>TvbRange</code> objects and strings for them and so on, even though they're identical. So instead do this:</p><pre><code>    local msgTypeBuf = buffer(n, 4)
local msgType = string.upper(tostring(msgTypeBuf))
if msgType == &quot;2201F0CE&quot; or msgType == &quot;2B01F0CE&quot; then
    if msgType == &quot;2201F0CE&quot; then
        pinfo.cols.info:append(&quot;- Some Message &quot;)

        subtree = tree:add(SOMETHING, buffer(), &quot;SOMETHING - Some Message&quot;)

        subtree:add(msgTypeBuf, &quot;Found - &quot; .. msgType)
        n = n + 4
    elseif msgType == &quot;2B01F0CE&quot; then
        pinfo.cols.info:append(&quot;- Some Other Message &quot;)

        subtree = tree:add(SOMETHING, buffer(), &quot;SOMETHING - Some Message&quot;)

        subtree:add(msgTypeBuf, &quot;Found - &quot; .. msgType)
        n = n + 4
    else
        return
    end

    local msgLengthBuf = buffer(n, 4)
    local msgLength = msgLengthBuf:le_uint()
    subtree:add(msgLengthBuf, &quot;  Message size = &quot; .. msgLength .. &quot; bytes&quot;)
    if msgLength ~= 7028 then
        subtree:add(msgLengthBuf, &quot;  Message size is incorrect!&quot;)
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buflen then
        return
    end
end</code></pre><p>And similarly in the later if-then statements.</p></div><div class="answer-controls post-controls"></div><div class="post-update-info-container"><div class="post-update-info post-update-info-user"><p>answered <strong>29 Jun '15, 09:30</strong></p><img src="https://secure.gravatar.com/avatar/d02f20c18a7742ec73a666f1974bf6dc?s=32&amp;d=identicon&amp;r=g" class="gravatar" width="32" height="32" alt="Hadriel&#39;s gravatar image" /><p><span>Hadriel</span><br />

2.7k2939
accept rate: 18%