Willoughby was born in Bath, the son of Harriet Susan (nee Boys) and George Thomas Railton Willoughby. His mother, the daughter of a naval officer, was principal of a seminary for young ladies who she instructed in her home in Camden Place. He was educated privately by the Reverend W. Williams and later at Addiscombe Military Seminary.