Born in England around 1707, Smith was the illegitimate son of Sir Thomas Lyttelton and a woman of whom details are unknown. He was raised a member of the Lyttelton family, who provided for Smith's education and aided him in the beginnings of his career in the Royal Navy.