Who is the antagonist in The Remains of the Day by Kazuo Ishiguro?

I don't really think there is an antagonist in this novel. Mr. Stevens has his own internal struggles with greater society and intimacy, especially with women.