Kline was born near Newton, New Jersey in 1858 and studied at Phillips Academy in Andover, Massachusetts but did not attend college. In 1876 and 1877, he started working for a men's clothing company in New York City and joined the New York National Guard as a private. When the Spanish-American War of 1898 began, he was named a Lieutenant-Colonel, and in 1901 a Brevet (honorary or acting) Brigadier-General.