Baker earned varsity letters at USC in 1929, 1930, and 1931. He played in the 1930 and the 1932 Rose Bowl, kicking five points after touchdowns. He kicked the winning 33-yard field goal with one minute to go in USC's game against the Notre Dame in 1931. It was USC's first victory in Souh Bend. Baker was an all-conference first-teamer in 1930 and 1931 and was invited to participate in a demonstration game of American football at the 1932 Summer Olympics, but he declined. At USC, Baker was initiated as a member of Phi Kappa Tau fraternity.