Brockman was born in South Carolina, the eldest son of Colonel and Senator Thomas Patterson Brockman and the granduncle of Tallulah Brockman Bankhead. He was also descended from John Brockman, Jr., and was the great-nephew of Major Brockman, both of whom were American Revolutionary War veterans in the English Brockman family, a cavalier family that settled in Virginia in the late 17th century.