The original castle was built by the Byset family. The castle came into the hands of the Frasers in the late 13th century. English forces besieged the castle in 1303. The castle sits on the site of several previous castles, including Castle Dounie, which was burned to the ground by Oliver Cromwell's forces during the Wars of the Three Kingdoms in the late 1640s and then by Government forces under the command of the Prince William, Duke of Cumberland shortly after Culloden, at the end of Jacobite rising of 1745, as Simon the Fox had been a Jacobite leader.