As usual, "fact" here refers to a subjective impression. But if Monygham hasn't questioned Nostromo's faithfulness "as a fact," he certainly has questioned the motives behind it. The implication is that Monygham's ideal -- the strength of his desire -- has corrupted his judgement.