Sherman was born in Phelps, Ontario County, New York on May 28, 1836. He studied law after his family moved to Iowa in 1855 and was admitted to the bar in 1859. Sherman then practiced law in Vinton, Iowa in 1860, but only until 1861, when he signed up with the 13th Iowa Volunteer Infantry Regiment for the Union Army. He retired as a captain in mid-1863 due to injuries suffered at Shiloh. In 1885, he was awarded an LL.D. from the University of Iowa. He was also a freemason who was especially active in Scottish Rite Masonry and helped found the Des Moines Scottish Rite Consistory.