The youngsterwas called Frank as a child. He had two older siblings, Eliza and Henry; a younger sister, Margaret, was born in 1838. Harte's father added an "e" to the family surname when Harte was a young boy, apparently to distance himself from his father, with whom he did not have a good relationship. Harte's father was a teacher, but many schools closed during financial hard times that began in 1837. The family moved frequently to wherever Henry Harte could find work as a teacher. When he died in 1845, the family was left in dire conditions and moved to New York City. By the time he reached thirteen, Harte quit school to work as a clerk. He later joined a local military company.
Around the time Harte was sixteen and already living on his own, he accompanied his sister on a boat ride from New York to San Francisco, California. The boat sailed from New York to Nicaragua. The Hartes crossed Nicaragua to reach a boat on the Pacific Ocean and sailed to San Francisco. Harte's mother had settled there after marrying Andrew Williams, who became the first mayor of Oakland, California.
Soon after arriving in Oakland, Harte headed north to a mining district near Sonora, California.
