Firstly, go talk to Harriet. Then talk to O'Connell, he will give you a Brass Medallion. If you choose to continue exploring in the day, talk to the villagers and they'll give you a few more cryptic clues on the past, none of which shed any real light on the situation. Go down the orangey stairs (south), there's some rebel denizens you can talk to. At the end of the path, a denizen will give you a clue about what to do next. You can talk to him again, and he will scare you by saying scary things about puzzles, and no mention of a flower portal.