Glover was born in Hampstead, London, the son of Honor Ellen Morgan (née Wyatt), a BBC journalist and a close friend of novelist Barbara Pym, and Claude Gordon Glover, a BBC radio producer. His younger half-brother is the musician Robert Wyatt. Glover has been twice married to actresses: Eileen Atkins and Isla Blair, with whom he has a son, actor Jamie Glover. In 2006, he held the role of Edward Alleyn Club President.