Flandrau's father was Thomas Hunt Flandrau of New Rochelle, New York, a law partner of Aaron Burr. His mother was Elizabeth Maria Macomb, a daughter of Alexander Macomb, the wealthy New York merchant, and half-sister of General Alexander Macomb, hero of the war of 1812 and afterward head of the United States Army. The Flandraus were descendants of Jacques Flandreau, a Huguenot who came to New Rochelle in the 1690s.