Scotto grew up in the Red Hook Carroll Gardens section of Brooklyn and studied pre-law and political science at Brooklyn College for two years. In 1957, Scotto married Marion Anastasio, whose father was capo Anthony Anastasio of what was the Anastasia crime family. Marion's uncle was boss Albert Anastasia. As an officer of the International Longshoremen's Association (ILA) Local 1814 in Red Hook, Anastasio used his position to control the Brooklyn waterfront. Scotto eventually joined Anastasio at the union local and became an ILA officer.