He was the son of Victurnien-Jean-Baptiste de Rochechouart, duc de Mortemart (1752–1812), and his second wife, and Adélaïde de Cossé-Brissac (1765–1820), only daughter of Louis-Hercule-Timoléon de Cossé, duc de Brissac (1734–92), and Adélaïde-Diane-Hortense-Délie Mancini (herself daughter of the duc de Nivernais). He and his family left France in 1791 and Casimir grew up in England, returning to France with his mother in 1801. Despite belonging to one of the old noble families of France, he backed the First French Empire, joining the gendarmes d'ordonnance (commanded by the count of Ségur) in September 1803. He moved to the 1st Dragoon Regiment on 10 February 1806 and took part in the campaigns in Prussia (1806), Poland (1807), Austria (1809) and Russia (1812), fighting at Golymin (1806),