In 1181, he was defeated at the Battle of Sunomatagawa by Taira no Tomomori; he fled, and tried to take a stand by tearing up the bridge over the Yahagi-gawa, and forming a shield wall. His forces were defeated once more at this Battle of Yahagigawa, but the pursuit was called off when Tomomori fell ill.