Why does Dave feel uncomfortable when he takes Toph to Little League games in the memoir, A Heartbreaking Work of Staggering Genius?

Dave goes to Toph's Little League games and sits with the parents of the other children. He can sense them watching him, wondering who he is to Toph, but he never attempts to talk to any of them. Their attention makes him uncomfortable.