It's true that it leaves open the possibility of a conserved quantity that is not associated with a symmetry. But the kinds of conservation laws we are thinking about, like conservation of energy, do originate from a symmetry. So if the symmetry is broken it is very reasonable to assume that the conservation law would be broken as well.