The castle dates back to 1389, when the Wallsee family commanded the construction of a strong fortress. Over time, the Wallsee family disappeared and the castle, after having been used as a prison, became the residence of the Luogar and Hofer. At the end of the 19th century it became the property of Prince Alexander Johann Vincenz Rudolf Hugo Karl Lamoral Eligius von Thurn und Taxis from the Czech branch of the House of Thurn and Taxis. In the 1950s the prince's family had rune into financial difficulties and sold it to the state. Within a year the house was ready to be occupied as the Prime Minister's residence, which previously was room 604 at the Grand Hotel.