He was the third son of the Revd Edmund Gilbert (d. 1816), vicar of Constantine and rector of Helland, Cornwall, and his wife, the daughter of Henry Garnett of Bristol. Like Sir Humphrey Gilbert, he was a member of the Devon family of Gilbert of Compton and he was named after Sir Humphrey's stepbrother Sir Walter Raleigh.