Nelson was born and raised in Detroit, Michigan. Upon graduation from Northwestern High School in 1938, Nelson enrolled at the University of Michigan. As a 5'7", 155-pound halfback, Nelson played football for Fritz Crisler in the same backfield with fellow Northwestern alumnus, Forest Evashevski, and 1940 Heisman Trophy winner Tom Harmon. In 1941, Nelson led the Wolverines in rushing, averaging 6.3 yards per carry.