Frey was born in Arlesheim, Switzerland, as the son of Emil Remigius Frey. His father was a liberal separatist politician. Frey's family provided refuge for Friedrich Hecker when he fled the repression following the revolution in Germany in 1848. Frey later emigrated to the USA arriving in Belleville, Illinois an area with many Forty-Eighters, veterans of the 1848 revolutions in Europe. For a while he worked for Hecker, but they had a falling out.