Only Able to Capture One Side of RTP

When I capture RTP Packets from my IP phone I only seem to get one side of the conversation. The source and destination stay the same all the way through the conversation. I never see the switch during the call. So when I go to play back the call I only get one side. The source always seems to be the person who initiated the call.