Kemper was born in Mountain Prospect, Madison County, Virginia, the son of William and Maria E. Allison Kemper and brother of Frederick T. Kemper (the founder of Kemper Military School). His father was of German ancestry, which had been in Virginia since the colonial era. His grandfather had served on the staff of George Washington during the American Revolutionary War, but he himself had virtually no military training. He graduated from Washington College (now Washington and Lee University) in 1842, becoming a lawyer.