Could someone explain to me why it matters whether or not Joseph had sex with his plural wives?
It doesn't. Some will point to the argument between straight plural marriage and dynastic sealings but so what? I personally think a combination of the two is possible. Some will point to the notion of being married within the laws of the land and the commandment to be obedient to such but so what? If plural marriage is God authorized, there are instances in the scriptures where God has authorized or justified the breaking of the law often by setting up a tension between conflicting laws or principles (starting with the partaking of the forbidden fruit and the command to multiply and replenish the earth).
So it all comes down to testimony of this Church as God's work and therefore it does not matter if Joseph Smith had sex with his wives or not. Critics start with things like plural marriage as if it were the foundation and ignore the actual foundation this is all built on.