Harvey was born in East Haddam, Connecticut, moving with his family to Ohio in 1828. He attended Western Reserve College and Preparatory School and worked as a teacher for a time, and eventually moved to Kenosha, Wisconsin, then named Southport, where he founded an academy. In Southport he associated with the Whig Party and edited a Whig newspaper, the Southport American (1843–1846). Lieutenant Governor Edward Salomon succeeded Harvey.