He started out in junior in the Montreal Canadiens organization with the Kitchener Canucks, Peterborough Petes, and the Ottawa-Hull Canadiens from 1954-1958. After a season with the Montreal Royals Spencer was claimed by the New York Rangers in 1959. Spencer debuted in the NHL with the Rangers in 1959-60 as a part-timer and became a regular for the next two seasons. The Boston Bruins claimed him in 1962 and he spent a full season there and was then claimed by the Detroit Red Wings in 1963. In the NHL Spencer played in a total of 230 games and scored 12 goals and 38 assists.