Around 1005, Fulk Nerra, count of Anjou, chose a rocky overhang dominating the valley of the Indrois as the site to have a powerful fortress built by his captain Roger le Petit Diable ("Little Devil"). Montrésor had one of the first keeps built out of stone, similar to that at Loches, and two circular walls, of which today only the west wall remains. In the 12th century, Montrésor fell into the hands of Henry II of England and the imposing towers at the entrance were built, as well as a part of the northern curtain wall. In 1188, King Philip Augustus of France retook Montrésor from the English. André de Chauvigny, returning from the Third Crusade with Richard the Lionheart, became the new lord of Montrésor, before having to cede the castle for almost two centuries to the Palluau family.