Harrison was born in Torquay, Devon, and educated at Brockhurst Preparatory School, where he is remembered every Armistice Day and at Dover College. At school Harrison was a tremendous all-round games player and, whilst in the Navy, he played rugby union and was capped several times for England. He is the only England rugby international to have been awarded the VC.