Schwartz was of Jewish heritage, and was a graduate of Saint Stanislaus College prep school in Bay St. Louis, Mississippi. From 1929 to 1930, he led Notre Dame, coached by Knute Rockne, to a 19–0 record and consecutive national championships. In a game against Carnegie Tech in 1931, he rushed for 188 yards, including touchdown runs of 58 and 60 yards.