The original Z-plan castle was built in 1567 by Clan Brodie but destroyed by fire in 1645 by Lewis Gordon of Clan Gordon, the 3rd Marquess of Huntly. It was greatly expanded in 1824 by the architect William Burn who turned it into a large mansion house in the Scots Baronial style. The Brodie family called the castle home until the late 20th century. It's widely accepted that the Brodies have been associated with the land the castle is built on since around 1160, when it is believed that King Malcolm IV gave the land to the family.