Born into a family of mandarins from Hà Tĩnh Province, Phan continued his ancestors' traditions by placing first in the metropolitan imperial examinations in 1877. Phan quickly rose through the ranks under Emperor Tự Đức of the Nguyễn Dynasty, gaining a reputation for his integrity and uncompromising stance against corruption. Phan was appointed as the Imperial Censor, a position that allowed him to criticise his fellow mandarins and even the emperor. As the head of the censorate, Phan's investigations led to the removal of many incompetent or corrupt mandarins.