Charles Robb was born in Phoenix, Arizona, and grew up in the Mount Vernon area of Fairfax County, Virginia. He earned a B.A. from the University of Wisconsin–Madison in 1961, where he was a member of the Chi Phi Fraternity. A United States Marine Corps veteran and honor graduate of Quantico, Robb became a White House social aide. It was there that he met and eventually married Lynda Johnson, the daughter of then U.S. President Lyndon B. Johnson. Robb went on to serve two tours of duty in Vietnam, where he commanded a rifle company in combat, and was awarded the Bronze Star.