You can holster weapons by pressing Unequip Key (Default Y) if you are holding a rifle of some sort, it will be strapped onto your back. Smaller weapons such as pistols will be put into your inventory unless it is full, in which case it will be hostered by your side.