Annabel's was founded in 1963, after Birley's friend John Aspinall decided that they needed somewhere to party after an evening's gambling. As a result, Birley turned the basement of Aspinall's casino, the Clermont Club, into a nightclub, and named it after his then wife, Annabel, who went on to marry Birley's friend, the late Sir James Goldsmith. Birley later blocked off the private staircase linking the Clermont above.