"You're the only one" is a rhetoric used to silence complaints by implying or outright claiming that the person is the only one who is bothered by the state of things. Combined with other rhetorics, it can take on a gaslighting vibe to imply that the complainer is overly sensitive, irrational, paranoid or delusional, or having a victim mentality.