Why does the narrator say he is invisible in the novel, Invisible Man?

The narrator says that he is invisible because he is not a ghost, and yet, people refuse to see him with their inner eyes. That is, they refuse to acknowledge him. He is like a street lamp; everyone knows it is there, but people don't spend any time thinking about it.