Why does my Lua dissector cause Wireshark to consume all memory?

I believe what's happening is wireshark is getting stuck in your while-loop, and never returning from it. This can happen because in certain cases your code will never have nextPDU be greater than buffer:len(). For example, if your dissector sees a TCP segment that has the first 4 bytes not match one of the hex strings you expect to see, then it will set tvbLength = buffer:len(), enter the while-loop with the variable n equal to 0, and thus have a nextPDU equal to buffer:len()... but that's equal-to, not greater-than, so the while loop won't return... ever. And since the if-then statements within the while loop allocate memory (by creating multiple TvbRanges), the memory usage will keep growing and growing.

So you're going to have to change your code logic to prevent being stuck in that while-loop forever. Like for example verify the nextPDU value changes on each loop, or something.

Also, your code is fairly inefficient - it creates a bunch of TvbRange objects and Lua strings multiple times instead of once.

For example, at the top of the dissector function:

if buffer:len() > 3 then
    if string.upper(tostring(buffer(0, 4))) == "0401F0CE" then
        tvbLength = 20728
    elseif string.upper(tostring(buffer(0, 4))) == "0501F0CE" then
        tvbLength = 12024
    elseif string.upper(tostring(buffer(0, 4))) == "2201F0CE" then
        tvbLength = 7028
    elseif string.upper(tostring(buffer(0, 4))) == "2B01F0CE" then
        tvbLength = 7028
    else
        tvbLength = buffer:len()
    end
end

There are three things in that if-then block that cause memory to be allocated:

1. buffer(0,4) creates a TvbRange object in C-code and Lua.
2. tostring() of the TvbRange creates a string in Lua.
3. string.upper() creates another string in Lua.

You call those three things up to four times in the above if-then block, depending on whether the elseif clauses get executed. For numbers 2 and 3 above, they will only create one string each in Lua for all four if/elseif clauses because they result in the same string and Lua internally only stores one copy of a given string; but still it's extra processing that isn't necessary. Instead, you should do this:

if buffer:len() > 3 then
    local msgType = string.upper(tostring(buffer(0, 4)))
    if msgType == "0401F0CE" then
        tvbLength = 20728
    elseif msgType == "0501F0CE" then
        tvbLength = 12024
    elseif msgType == "2201F0CE" then
        tvbLength = 7028
    elseif msgType == "2B01F0CE" then
        tvbLength = 7028
    else
        tvbLength = buffer:len()
    end
end

That both reduces the memory allocated, and improves performance.

By the way, what if buffer:len() is not greater than 3? Then tvbLength isn't set to anything.

Next you have this:

    if string.upper(tostring(buffer(n, 4))) == "2201F0CE" or string.upper(tostring(buffer(n, 4))) == "2B01F0CE" then
        if string.upper(tostring(buffer(n, 4))) == "2201F0CE" then
            pinfo.cols.info:append("- Some Message ")
        subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

        subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
        n = n + 4
    elseif string.upper(tostring(buffer(n, 4))) == "2B01F0CE" then
        pinfo.cols.info:append("- Some Other Message ")

        subtree = tree:add(SOMETHING, buffer(), "SOMETHING - Some Message")

        subtree:add(buffer(n, 4), "Found - " .. string.upper(tostring(buffer(n, 4))))
        n = n + 4
    else
        return
    end

    subtree:add(buffer(n, 4), "  Message size = " .. buffer(n, 4):le_uint() .. " bytes")
    if buffer(n, 4):le_uint() ~= 7028 then
        subtree:add(buffer(n, 4), "  Message size is incorrect!")
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buffer:len() then
        return
    end
end</code></pre><p>Again, you're creating multiple <code>TvbRange</code> objects and strings for them and so on, even though they're identical. So instead do this:</p><pre><code>    local msgTypeBuf = buffer(n, 4)
local msgType = string.upper(tostring(msgTypeBuf))
if msgType == &quot;2201F0CE&quot; or msgType == &quot;2B01F0CE&quot; then
    if msgType == &quot;2201F0CE&quot; then
        pinfo.cols.info:append(&quot;- Some Message &quot;)

        subtree = tree:add(SOMETHING, buffer(), &quot;SOMETHING - Some Message&quot;)

        subtree:add(msgTypeBuf, &quot;Found - &quot; .. msgType)
        n = n + 4
    elseif msgType == &quot;2B01F0CE&quot; then
        pinfo.cols.info:append(&quot;- Some Other Message &quot;)

        subtree = tree:add(SOMETHING, buffer(), &quot;SOMETHING - Some Message&quot;)

        subtree:add(msgTypeBuf, &quot;Found - &quot; .. msgType)
        n = n + 4
    else
        return
    end

    local msgLengthBuf = buffer(n, 4)
    local msgLength = msgLengthBuf:le_uint()
    subtree:add(msgLengthBuf, &quot;  Message size = &quot; .. msgLength .. &quot; bytes&quot;)
    if msgLength ~= 7028 then
        subtree:add(msgLengthBuf, &quot;  Message size is incorrect!&quot;)
    end
    n = n + 4

    -- DECODE HERE

    n = nextPDU
    if nextPDU == buflen then
        return
    end
end</code></pre><p>And similarly in the later if-then statements.</p></div><div class="answer-controls post-controls"></div><div class="post-update-info-container"><div class="post-update-info post-update-info-user"><p>answered <strong>29 Jun '15, 09:30</strong></p><img src="https://secure.gravatar.com/avatar/d02f20c18a7742ec73a666f1974bf6dc?s=32&amp;d=identicon&amp;r=g" class="gravatar" width="32" height="32" alt="Hadriel&#39;s gravatar image" /><p><span>Hadriel</span><br />

2.7k●2●9●39

accept rate: 18%