Who is the antagonist in Musui's Story: The Autobiography of a Tokugawa Samurai by Katsu Kokichi?

Musui interacts with many people who might be considered antagonists over his lifetime. His grandfather imprisoned him in a cage, his grandmother was partially responsible for his runnning away, and Kobayashi attacks Musui repeatedly after losing to him in a competition.