The reader is never really told what it is exactly that Chillingworth sees on Dimmesdale's chest, but one could possibly infer that he had somehow placed the letter A on his chest to match Hester's. Or he could have been wearing a necklace with a lock of Hester's hair in it or Pearl's. We just don't really know.
