The contract to build James Madison was awarded to Newport News Shipbuilding and Dry Dock Company in Newport News, Virginia on 20 July 1961 and her keel was laid down there on 5 March 1962. She was launched on 15 March 1963, sponsored by Mrs. A.S. "Mike" Monroney, and commissioned on 28 July 1964 with Commander Joseph L. Skoog, Jr. in command of the Blue Crew and Commander James D. Kearny in command of the Gold Crew. She was launched with sixteen Polaris A-3 ballistic missiles.