Determine the direction of a conversation (client, server)

I'm writing a dissector for a TCP-based protocol in C.

Is it possible to determine whether a packet goes in the client-to-server or in the server-to-client direction?

I have set up a conversation state, but I'm not sure what to put into it. I can recognize the first packet in the conversation (client-to-server), and I use that to store pinfo->srcport and pinfo->destport in the conversation state, and in future packets I can compare pinfo->srcport against the stored value in the conversation. But that will break if the source and destination ports are the same.

I noticed that *pinfo contains source and destination addresses, too, but those are fixed for the entire conversation, i.e. they don't change between requests and responses.

What's the idiomatic way to track the direction of the conversation?