Why does Sedaris fear he is considered the village idiot in Normandy?

Sedaris fears he is the village idiot because he can only speak in random nouns he has memorized, and this makes his conversation hilarious and nonsensical.