Robertson was born in Washington-on-the-Brazos, Texas, to Mary (Cummins) and Jerome B. Robertson, who would also serve as a Confederate general. He attended Baylor University and went to West Point in 1857, but left before graduation to serve the Confederacy.