string variable not interpreted right by unpack()

Question

I'm trying to convert a hex string to decimal string byte by byte, and then pass it to a Struct.unpack function to convert to double. However, I get the wrong number even though the exact same string value in a string literal will get the right number.

Here is the code:

str = "3f91df0b2b89dd1e"
decimalStr = ""
for i = 1, string.len(str), 2 do
decimalStr = decimalStr .. "\"
onebyte = string.sub(str, i, i + 1)
num = tonumber(onebyte, 16)
decimalStr = decimalStr .. num
end
print(decimalStr)
print(Struct.unpack(">d", decimalStr))
print(Struct.unpack(">d", "\63\145\223\11\43\137\221\30"))

result:

\63\145\223\11\43\137\221\30
1.6136274310717e+136        9
0.017452406437284       9

I don’t understand! decimalStr is “\63\145\223\11\43\137\221\30”, but it is giving out the wrong value after unpack. As you can see, if you use the same actual string literal it works.

What’s going on?

Answer 1

0

Your for-loop is constructing a Lua string, containing the characters exactly as printed. In other words, you've constructed a string of the character "\" followed by the character "6" followed by the character "3", etc.

But this:

local foo = "\63\145\223\11\43\137\221\30"

Does not construct that same string. Why? Because those "\" in this case are escape codes, telling the Lua interpreter that the digits following the "\" are to be interpreted as the decimal value of a character - in this case whose decimal value is 63, not the raw characters "6" followed by "3".

In other words, before Struct.unpack() even gets the Lua string, Lua has already converted it to a string containing the binary byte whose decimal value is 63 (hex 0x3F) followed by one whose decimal value is 145 (hex 0x91), etc. That's a very different string, obviously.

Try doing this and you'll see what I mean:

print("\63\145\223\11\43\137\221\30")

You'll probably see gibberish on your screen, because it's a string of bytes which may or may not be printable characters. But it's not the ascii character "\" followed by "6" followed by "3", etc.

This is just like in most any programming language I can think of - if you escape things inside a string literal in the source code/script, the compiler/interpreter knows you mean to handle them as something else.

answered 20 Mar '14, 12:31

Hadriel
2.7k●2●9●39
accept rate: 18%

BTW, if you can tell me what it is you're trying to accomplish - the bigger picture - perhaps I can help you get there faster. But if you'd like to get there on your own that's cool too. :)

(20 Mar '14, 12:35) Hadriel

Well, the big picture is I need to convert bytes of different lengths to various numbers (unsigned int, int, float, double, etc).
When I have 4 bytes or less it's more or less straight forward. When I have more that 4 bytes, I've been using the UInt64/Int64 to get the value from the bytes, pack it first and then unpack to convert to the type I want. However, when converting 8 bytes to double, it doesn't work right in Windows. Following your post to my question, I decided to skip the pack step and go directly to unpack, passing to it a string of the bytes' values in decimals. Obviously I thought it's a normal string of numbers separated by "\". Now I realize it's not the case. How did you get the decimal number sequence with escape characters from hex? If you have an easier way to do what I need to do, please share.

(20 Mar '14, 14:27) YXI

"How did you get the decimal number sequence with escape characters from hex?"

I used my calculator to convert the hex value to a decimal value. Mac's built-in calculator in programmer view mode is quite useful. :)

But obviously you wouldn't do that in a Lua script - you'd use string.byte() in a for-loop or as an argument to string.gsub(). But in wireshark you could just use Struct.fromhex().

But that's assuming you started out with a string of hex-ascii characters. Is that the case for you? Where did this string come from? If it's from a packet's contents, why do you need to use Lua to convert them instead of using the provided tvb/tvbrange or tree functions?

(20 Mar '14, 14:51) Hadriel

I cannot use tvb/tvbrange functions because my values are stored in disjoint bytes. For example, a 8-byte value can be stored in offset 0-4 and then 16-20. I could use byteArray:tvb() to create a new tvb using these bytes. However, whenever I create a new tvb this way, a new tab is added to the packet bytes pane in the wireshark GUI. And since I have lots of these tabs, I believe it is the reason why my wireshark is crashing whenever I click on a tree item that refers to a newly created tvb. So now what I do is to get the value by this code: Let's say byteList have the byte locations I need from the payload. byteList = { 4, 5, 6, 7, 12, 13, 14, 15}

    allBytes = ByteArray.new()
for i = 1, #byteList do

    nextByte = buffer(offset + byteList[i], 1):bytes()

    allBytes:append(nextByte)

end

value = UInt64.fromhex(tostring(allBytes))</code></pre><p>I do some bit shifts and masking that is some other extra requirement, and then I get the string again by str = value:tohex()</p><p>Now from here I now plan to use unpack to finally get the value I need that will work in Windows as well.</p></div><div id="comment-31018-info" class="comment-info"><span class="comment-age">(20 Mar '14, 15:37)</span> <span class="comment-user userinfo">YXI</span></div></div><span id="31026"></span><div id="comment-31026" class="comment"><div id="post-31026-score" class="comment-score"></div><div class="comment-text"><p>For the first part of your comment: Wireshark shouldn't crash, so please either submit a bug or provide enough of a sample script here for me to reproduce the crash. Crashes are bad mojo.</p></div><div id="comment-31026-info" class="comment-info"><span class="comment-age">(20 Mar '14, 17:48)</span> <span class="comment-user userinfo">Hadriel</span></div></div><span id="31027"></span><div id="comment-31027" class="comment not_top_scorer"><div id="post-31027-score" class="comment-score"></div><div class="comment-text"><p>For the second part of your comment: that seems like a hard way to go about it - why not just use <code>Tvb:raw()</code> to get a Lua binary string of the portion that has your bytes, and then operate on that string in Lua, to get the substring(s) your need (for example using <code>string.sub()</code> or even <code>string.match()</code>), concatenate the substrings and then feed that resulting new string to <code>Struct.unpack()</code>?</p></div><div id="comment-31027-info" class="comment-info"><span class="comment-age">(20 Mar '14, 17:53)</span> <span class="comment-user userinfo">Hadriel</span></div></div></div><div id="comment-tools-31009" class="comment-tools"><span class="comments-showing"> showing 5 of 6 </span> <a href="#" class="show-all-comments-link">show 1 more comments</a></div><div class="clear"></div><div id="comment-31009-form-container" class="comment-form-container"></div><div class="clear"></div></div></td></tr></tbody></table>



    
    
      
        
        
        
        
      
      
        
        
        
        Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.