Fittingly, Frei was born in the small Wisconsin town of Oregon. He spent his early years in Brooklyn, Wisconsin, then moved with his family to Stoughton, near Madison. He graduated from Stoughton High School in 1941, shortly before his 17th birthday. He was a classmate there of Marian Benson, whom he later married in 1945. Frei was inducted into the Stoughton Hall of Fame after his death.