Not really, it's true or false regardless.
But it does drastically affect his credilbity in assessing the validity of his claims.
This credibility depends to a large degree on the a priori assumptions we hold about the nature and morality of God (e.g., does God really devalue women so) and the expected morality of God's emmissaries (e.g., using a position of power to extract sex from young girls and others who accept you as an authority figure).
It depends also on whether we apply the same standards or morality, propriety, etc. to Joseph Smith as we would to others in his position and the degree to which we're willing to relax those standards in his case.
The credibility issues have become one of my major points within what i see as issues of Mormon base structure. When we have a man that was regularly shown to be dishonest or to have a very strong lack of credibility. It should then likewise put everything he does under an understandable level of scrutiny to make sure he isnt leading someone with a lie. There are many people today who claim to do great and holy things in the name of one god or another. Only that when you look at their lives as a whole, they can be shown in nearly every aspect to be dishonest. To me, just because you claim to be doing holy work, does not excuse you from what should be considered very standard societal practices of judging that persons credibility.
As one such example, if you hired a person that had a long history of drug use and had been convicted several times of felony distribution of controlled substances. Would you be readily willing to hire that person for a job that would allow him to have little supervision for much of his shift? Im not saying that people don't deserve a chance, but if they had not shown credibilty in any area as of yet, what reasoning would you have to give them a good level of trust for something as simple as a job? If we where to liken this to Joe and the happenings of his life, we are taking the claims of someone that has a long history of being a professional con-man. All for the claims of something that could potentialy be even more important than our individual lives! wouldn't you want to make sure that such a person and their credibilty stand up to make sure you're not following someone who just wants power?
The more i learned about Joseph Smith's history, the events of his life, and the ties they had to the invention of Mormonism. The more i see a steady continuation of his old behavior. Possibly even amplified due to his new stature of power. As such, even after his claims of being a prophet, i see little to give him credit to be attributed to someone that could be considered for such an important posistion. Lies, desceit, misdirrection, manipulation, ect. that continued through his life.
Im going to use a phrase that will make me want to wash my mouth out with soap...but here goes...WWJD? Jesus though the entirety of the Bible would teach and practice personaly. But at no time was he credited with lying to anyone. Take the issue with the money changers in the temple. If someone would have come up to him afterwards, would be have lied or twisted the facts? He led an open and honest life on what he did and practiced. He kept his story straight even when pressed by dissenters. Even those that where within his inner circle. (im getting a bit o/t, ill try to get back)
Within this entire argument it stands a bit more on not whether or not we could call Joseph Smith a pedophile (which personaly i would say he was) but much more on his credibility and his use of controll that he had over people for use of his own purposes. I think it was Bob Mccue that said that within cult groups, such controll over sex is common place. But generaly it is understood openly within such groups that when the leader calls "so and so" to be with him. That its what is generaly expected. Sexual favors and controll over sexual acts could be considered as payment for following the group tenants. Even if Joseph Smith was trying to get someone else (such as the issue originaly with FA's mother), that he would take such a young girl in place of her mother shows just how far he was willing to go for sexual gratification. That he took several other equaly young girls supports this.