H. Henry Baxter, the son of Judge Henry Baxter, was born in Saxtons River, Vermont on January 8, 1818. He attended schools in Windham and Windsor Counties and at age 15 he became a clerk in a Boston dry goods commission warehouse. He worked in Boston for two years and became a supervisor of employees as the warehouse's head bookkeeper.