Polygamy tends to work practically when only one gender practices it. A single man with multiple wives or a single woman with multiple husbands can still be a cohesive family unit. A man with multiple wives, each of whom is also married to multiple men who probably also have other wives, etc no longer is. Marriage would cease to be a useful social construct in that case.

Since there would be no way the government could legally get away with only allowing one gender to marry multiple partners and not the other, you'd risk large multiple marriages where any one person could have as many other spouses as they wished, at which point the societal benefits of marriage would be impossible to apply practically.