I believe that Oliver wrote that in a letter to his relative. And at that time is was out of or leaving the church. And I don't think that Oliver caught them at it. He went by the rumor, I believe. I would need to reread the process of events.
And least we forget, he did come back to the fold and bore his testimony of the Book of Mormon. Quite a guy!!
Since he never lost belief, apparently, that ought to provide even more compelling evidence that there was serious evidence something significant occurred between Fanny and Joseph - something more than being alone in a room.
But, of course, the most compelling evidence that "something" happened is the fact that Fanny is recognized as Joseph Smith's first plural wife by apologists. There would be no reason for apologists to accept that if there were not good evidence that there was an intimate relationship between the two.
The second most compelling evidence that Fanny and Joseph had an intimate relationship is that there can be no reasonable doubt that Joseph had intimate relations with his later spiritual wives. So why would he have not done so with his first spiritual wife?
This reminds me of the Martha Brotherton story. Apologists insist that Martha fabricated the event. Yet, Brigham Young had Martha sealed to him as his "wife" after his death. I find that the most compelling piece of evidence that she was telling the truth - combined with the fact that the details of her story matched details of other spiritual wives of the period.
There can also be no reasonable doubt that the church was plagued with serious rumors about intimate relationships, hence their repeated denials. The later open practice of polygamy proved that those earlier rumors were based in fact. Joseph was plagued by serious rumors about Fanny Alger. The later open recognition of her as his "wife" proves that those rumors were based in fact.