Serrell was born in London, England on November 5, 1826. He was the tenth of eleven children of William and Ann Serrell. The family immigrated to New York in 1831, and as a result, Edward received his early education in the schools of New York City. He later took up civil engineering under the direction of his father and an older brother. On April 6, 1848, he married Jane Pound, an English girl two years his senior. They had four children.