How can I fix this Glib-ERROR?

Question

Using an unmodified verion of wireshark 1.4.9 that I build from source I am getting the following error when I attempt to load a capture file with a proprietary protocol:

Glib-ERROR **:gmem.c:176: failed to allocate 2516584916 bytes aborting

After this I get a MSVC error and wireshark closes.

I get the same error when I run the same version of wireshark with a custom plug-in to decode the proprietary protocol.

Any pointers on possible causes or where to begin troubleshooting?

Answer 1

I assume that you control the code that actually decodes your protocol (if not, there won't be much you can do other than contact the maintainer of the decoder). That said, my guess is that the dissector for your protocol is attempting to allocate a buffer for inflated/decrypted/etc data based on a size field that is not bounds-checked and either incorrectly extracted or incorrectly set in your capture file. Put differently, something like this is in the dissect_PROTONAME function:

guint32 inflated_size;
guint8 *inflated_data_buffer;
...
inflated_size = tvb_get_ntohl(tvb, OFFSET, ENCODING);
inflated_data_buffer = g_alloc(inflated_size);

Realistically, it is impossible to say what is causing the problem without seeing some dissector code, but I assume you have access to that. Since you can compile Wireshark yourself, the best thing to do will be to use the debugger to see what's going on. At the very least, a stacktrace will help you pinpoint the problem, even if it is ultimately out of your control.