Hanger was born into a prosperous family in Gloucestershire, the third son in a family of seven children. His father was Gabriel Hanger, a member of parliament who in 1762 was created Baron Coleraine in the Peerage of Ireland. He declined a seat in the House of Commons, even though his father and two of his brothers had sat there, and in 1814 he refused to take his seat in the Irish House of Lords when he succeeded to the family title. He died in London in 1824, at the age of 74, when his Irish peerage became extinct.