The trick is to separate the collision from the visible geometry. So what you do is create a client context hierarchy/tree that contains the visible geometry/objects, and then use a separate hierarchy/tree for much simpler 3D objects (primitives like cubes, cylinders etc) that are only as complex as they need to be to surround the visible objects as "bounding boxes". These would be networked, but the idea is they are much simpler and fewer in number since all they need to is provide collision, and then you just set them to be invisible.
Thanks for the response. I suspected this was the case but wanted to make sure there wasn't another option before reimaging my templates for efficient collision detection.
If you want to keep separation, use the static context.
Static context is a wrapper-like folder, which stores static objects like meshes with collisions.
You can spawn a static-context at runtime to make it have complex collisions.
Of course it will cost you 1 networked object count to your hierarchy, but only 1,the children of the static-context doesn't count, like server-context and client-context.