Brooks was born in Boston, Massachusetts in 1835. Through his father, William Gray Brooks, he was descended from the Rev. John Cotton; through his mother, Mary Ann Phillips, a very devout woman, he was a great-grandson of Samuel Phillips, Jr., the founder of Phillips Academy, Andover, Massachusetts. Four of the couple's six sons -- Phillips, Frederic, Arthur and John Cotton -- were ordained in the Episcopal Church.