Albani was born in Rome, the son of Carlo Albani, Duke of Soriano; his grand-uncle was Pope Clement XI (Gianfrancesco Albani). Furthermore, two of his uncles Annibale Albani and Alessandro Albani were cardinals, and was himself uncle of cardinal Giuseppe Albani (with whom he was, for two years, concurrently cardinal).