Why does Herzog begin to question his trust in Aunt Zelda in the novel, Herzog?

Aunt Zelda is Madeleine's aunt, and she has taken Madeleine's side in the divorce, so Herzog no longer trusts her. He still cares for Zelda and her husband, Herman, but he sees during their conversation that Madeleine has influenced Zelda to "her side."