Ellington grew up in Wadley, Georgia. He began working in cotton fields at age 5 to help his family, which included eight brothers and sisters. He left home for good by the age of 12, hitching a ride on a freight train to Savannah, Georgia where his sister lived.