Why was James so tormented by the way he looks?

James' appearance tormented him because he found there was no one with whom he could identitfy. He didn't look like his mother...... she was white, and he didn't look like his step-father, who was black. Although, we would likely see him as unique, James saw himself without a true identity.