It was built originally in the 13th century as a tower house and enlarged and fortified in the 14th century. At this time the castle was owned by Gerald Widdrington and, although it was still owned by the Widdringtons in the early 14th century, the Swinburns were living in it By the 16th century the castle fell into disrepair and ruin and it was attacked by Border reivers. A survey of 1541 reported the roof and floors to be "decayed and gone". The property was acquired by the Smith family in about 1640, but in 1715 a further survey stated the building to be ruinous.