Polyamory, whether it is harem or multi-directional, is a difficult issue for people raised in the modern era in first-world countries. While it is beginning to be normalized in some countries, it has been demonized for so long that it will probably be a long time before the stigma associated with it fades away, if ever.
In VNs, I think the one that takes polyamory to its ideal end is Draculius, one of my favorite VNs. While Jun, the protagonist, is central to things, each of the heroines has a defined role within the family that can't be played by any of the other girls. Belche is the mother, Xeno is the protector, Rika is the attacker, and Lian is the representative.
I do agree that in most cases a harem should stop at three (one primary and two secondaries with a relationship aside from that possessed by the primary). To be blunt, there is only so much time one person can put into other people, and expanding to the ridiculous size some VN harems have is just undoable.
Traditionally, there were any number of reasons for a harem to exist. One was in the case of royals and high nobles, where numerous heirs were necessary to solidify power and ensure passage down to the next generation. Then there were rich merchants and leaders of settlements. In both cases, this prevented too much wealth being permanently settled into one bloodline and reduced resentment from the population at large. It also provides a larger supporting family for the raising of children, reducing individual burdens for child-rearing and making things more flexible without going outside the family group.
Generally speaking, early human civilizations didn't place emphasis on and in some cases didn't even have the concept of romance. That is why some cultures retain arranged marriages and defacto polyamorous relationship setups. (for JVN fans, the example would be the way it is tacitly understood that powerful Japanese men - like high-ranking politicians and corporate leaders - will have lovers outside of their wives).