Who is the antagonist in The Color of Water by James McBride?

It's difficult to pinpoint a single antagonist. Ruth's father would serve as an antagonist, as well as racial prejudice.