Born and raised in Southern California, Norton graduated from Fullerton Union High School in 1956 and played college football for head coach Skip Stahley at Idaho, alongside future pros Jerry Kramer, Wayne Walker, and Jim Prestel in the Pacific Coast Conference. Norton played defensive back and tight end, and also punted and returned kicks for the Vandals. Although selected by the Detroit Lions in the 1960 NFL Draft (7th round, 75th overall) and the Dallas Texans in the AFL's inaugural draft 1960, he signed as an original Houston Oiler.