Mr. Hooper was ostracized from society after he decided to wear the black veil permanently. Everyone feared him especially the children, and he was not invited into any social gatherings. Lastly, his wife let him because of the anxiety he causes her about him.
