The hermitage is believed to have been founded by Phrin las rgya mtsho (d. 1667), who was the regent of Tibet from 1665 to his death. He was a student of the Fifth Dalai Lama and requested his permission to build a hermitage for eight to sixteen monks in the foothills above his native Nyangbran and invited the Fifth Dalai Lama to perform a “site investigation” (sa brtag) to determine the most auspicious location on which to build the monastery. The Dalai Lama made the treasure (gter) discovery of the self-arisen stone image of the Buddha that is still located in Chupzang’s lower temple. However, the initial hermitage fell into ruin and the official founding of the monastery is credited to Phrin las rgya mtsho's nephew, Sde srid sangs rgyas rgya mtsho, in around 1696.