During the year before the fall of the Azai, Yoshiharu had corresponded with members of the Takeda clan, and joined them after the Azai were destroyed. When the Takeda clan was annihilated, Yoshiharu took refuge at a Buddhist temple called Erin-ji in Kai Province. Afterwards, Yoshiharu joined Toyotomi Hideyoshi, and became a retainer of Toyotomi Hidetsugu. The last thing that Yoshiharu did in history was to become an instructor of archery for Toyotomi Hideyori at Osaka Castle.