Anthony Shirley was the second son of Sir Thomas Shirley of Wiston, Sussex, and Anne Kempe, the daughter of Sir Thomas Kempe (d. 7 March 1591) of Olantigh in Wye, Kent. He had an elder brother, Sir Thomas Shirley, and a younger brother, Sir Robert Shirley, and six sisters who survived infancy.