Born Joseph Iannuzzi II in Port Chester, New York. He was the son of prominent Westchester County bookmaker and policy numbers racket, Joe Iannuzzi, Sr. His parents were originally from Palermo, Sicily. At a young age, Joe's father would take him along to make his collections for the numbers game, horse and sports bets.