Pooley dropped out of high school to fight in World War I. He earned a scholarship to play football at Princeton University but arrived too late for the entrance exams. He tried a few other schools before ultimately enrolling at the University of Alabama as a 20-year-old freshman. By the time he was a 24-year-old senior, his teammates had begun calling him "Papa Pooley" because he was so much older than them.