He was born at Brent Eleigh, Suffolk and was cousin to Salisbury MP Richard Colman. He attended Trinity College, Cambridge receiving an MA in 1659. Colman converted to Roman Catholicism in the early 1660s. Colman acted independently of Charles in trying to obtain French financial assistance to reduce the king's dependence on the anti-Catholic parliament. When money was eventually secured, it was not through any of Colman's efforts.