The buildings were originally a 13th-century stone fortified manorial complex, founded by the Augustinian Priory of Haverfordwest.