He grew up in Wiltshire in the town of Fullerton. He is presumably the eldest of his nine siblings, and has two younger brothers before Catherine and Sarah Morland, the fourth and fifth respectively of the Morland children. He attended school away from home and spent Christmas one year with the Thorpe family, as he was friends with John Thorpe, brother of Isabella Thorpe. The Thorpes were under the impression that James was the heir to a lot of wealth than he actually was, although not because he deceived them in any sort of way as they deceived themselves.