Lovell was born in Boston, Massachusetts, the son of James S. and Deborah (Gorham) Lovell. His father attained the grade of major in the Continental Army during the Revolutionary War and his grandfather, James Lovell, was an active member of the Whig organization in Boston before the Revolution, and was a member of the Continental Congress from 1777 to 1782. He was one of the prime movers in the scheme to supplant General Washington as commander-in-chief by General Horatio Gates. He was an original member of the Massachusetts Society of the Cincinnati.