The youngest son of General Jonathan Hunt of Vermont and the former Jane Maria Leavitt, and brother to architect Richard Morris Hunt and painter William Morris Hunt, Leavitt Hunt was born in Brattleboro, Vermont, but grew up in Paris following the early death of his father Jonathan Hunt, a Vermont Congressman whose father had been the state's Lieutenant Governor.