Sargent was the second son of Henrietta (Gray) and Ignatius Sargent, a Boston merchant and banker who grew wealthy on railroad investments. He grew up on his father's 130-acre (53-ha) estate in Brookline, Massachusetts. He attended Harvard College, where he graduated in Biology in the class of 1862. Sargent enlisted in the Union Army later that year, saw service in Louisiana during the American Civil War, and was mustered out in 1865. He traveled in Europe for three years.