Koller was born in Glonn in Bavaria. An exemplary officer, he graduated valedictorian at the Air War Academy. He was the Chief of Staff for Hugo Sperrle during the Blitz. For Operation Sea Lion, the planned invasion of the United Kingdom by the Wehrmacht, Oberstleutnant Koller was to serve as the Operations Officer of Luftflotte 3, in coordination with the German 9th Army.