Immediately after you have chosen Refuse to Drink, the Am Fah around you will become hostile. This battle isn't too hard, but there are quite a few of them. Am Fah in the rest of this area will become hostile too. All you must do is return to Kaineng Center and give the chalice to Imperial Agent Hanjo, so you could just skip the fight and map travel back to Kaineng Center.