Mike Quinn (The Mormon Hierarchy: Origins of Power, 32-33) wrote about it years ago, in fact:
The visits of various figures after that would then be related to the conferral of specific keys, but priesthood was something Joseph believed he already possessed because he had inherited it from his father. Once one understands Mormon priesthood in this light, then many other apparent problems and mysteries come into clearer view. Of course, this is why the Presiding Patriarch was such an important figure. Of course, the Smith family believed that the leader of the Church needed to be a son of Joseph Smith, Jr. Of course, the Smiths continued to hold the office of Patriarch until Eldred G. Smith.Smith inherited lineal priesthood without ordination. A revelation stated this in December 1832: "Therefore, thus saith the Lord unto you, with whom the priesthood hath continued through the lineage of your fathers--For ye are lawful heirs, according to the flesh, and have been hid from the world with Christ in God." (D&C 86:8-9). In 1847 Brigham Young preached: "Joseph Smith was entitled to the Keys of the Priesthood according to Blood. Still He was the fourth son." Decades later, Apostle Franklin D. Richards published that "Joseph Smith, Sen., inherited the Patriarchal Priesthood, by right from the fathers over the house of Israel in this dispensation. However, early Mormon doctrine was not simply that Smith inherited the "right" to the priesthood, but that he actually possessed the priesthood at birth. For example, in January 1845 Young blessed a newborn "son of Promise . . . on whoom [sic] the Preasthood [sic] shall rest from his birth to all Eternity even so Amen."
Joseph Smith, Jr., was the fourth-born son in his family, and the third to survive. Like Lehi's fourth son Nephi in the Book of Mormon (1 Ne. 2:5), Smith was the "son of promise." For the Smith family, literal ordination to the office of patriarch was merely a formality of church consistency of those who were lineal heirs of Patriarchal priesthood.
Of course, polygamy was intended to attach other families in the Church to the one family that truly had the priesthood by right through the sealing power that came with the patriarchal priesthood. That's right: the sealing power was believed to accompany the patriarchal priesthood.
Suddenly Abraham chapter 1's description of Abraham and his priesthood makes a whole lot more sense. In it, Joseph Smith reveals a lot about his priesthood and its origins that he did not spell out explicitly in any other place.