where can i see the decoding log

Look at the RTMPT_DEBUG macro. It either expands to rtmpt_debug or to a bit of a hack that results in nothing being printed. rtmpt_debug() uses vprintf(), which prints to the standard output, so the "log file" is wherever Wireshark's or TShark's standard output goes. If you run Wireshark in a terminal, then, unless you redirect its standard output, it'll show up on your terminal; your terminal is probably an xterm or Konsole or gnome-terminal or other such terminal-emulator program; it might have scrolling enabled, so you can see more than one screenful of output.