He was born in Roxbury, Massachusetts, the younger son of Frederick Dan and Hannah Huntington. While he was a child his father, a Unitarian minister, converted to the Episcopal Church, and in quick succession was ordained deacon and priest, and then consecrated Bishop of Central New York. James went to Harvard, as his father did, and later studied at St. Andrew’s Divinity School in Syracuse. He was ordained priest around 1880, and began work among working-class immigrants on Manhattan's Lower East Side.