If you have an idea, read the spoilers, if you'd rather wait for someone else to guess so you can enjoy the book once it is found, do not!
This takes place in some kind of hive in intersellar space. Humans have discovered this huge organic structure inhabited by many many different species, some of them sentient, other very primitive animals. They have determined that it is in fact a hive of one of these species, with all the others being commensals (just as there are a lot of commensal species in an ant-hill, for instance) but they cannot determine which it is. The species they interacted with, the most advanced
At the end the human explorers find the explanation.