Is there a way to group ProtoFields In Lua?

Maybe, depending on how you mean the question.

Basically the foo_proto.fields member is a Lua table array of ProtoFields, as you said; but it's real purpose is to bind the internal wireshark representation of protocol fields to Lua. When you set the ProtoFields into the foo_proto.fields member, which you don't show in your example but is presumably being done, then internally wireshark remembers those fields, and once your script is loaded, they're each registered internally for your protocol. That internal structure isn't changeable by Lua code. You can certainly store those fields in other tables/variables in your Lua script, but it won't change how wireshark processes them.

So, for example, you could do this:

-- my new proto
local foo_proto = Proto(...)
-- create some protofields
local field1 = ProtoField.new(...)
local field2 = ProtoField.new(...)
local field3 = ProtoField.new(...)
-- sets the protofields to the proto
foo_proto.fields = { field1, field2 }
-- have a nicer way of getting my protofields
local myfoo = { message = {header = field1, data = field2}, keepalive = field3 }
function foo_proto.dissector(tvbuf,pktinfo,root)
…
tree:add(myfoo.message.data, tvbuf:range(2,2))
…
end

You can do that, but it’s not like Wireshark is going to know that field2 or myfoo.message.data is at that buffer range/offset or anything. All you’re doing is creating a Lua table, with member variables that happen to point to the same ProtoField userdata objects that you set into the Proto.field member. I.e., since userdata objects are handled as pointers and copied by reference, the above works; wireshark knows nothing about it… as far as wireshark knows, your myfoo.message.data is identical to using field2.

What you cannot do, is do that type of thing using foo_proto.messages, because foo_proto represents a wirshark object, and that class type has no member variable named messages which you could be getting/setting from/to. In the past I think it might have silently allowed you to it but not actually done the action; but more recent versions will error if you try setting a member that doesn’t exist like that.