This is not a reason to avoid it, but to be very careful when shopping and to buy only something that is not adulterated with anything, for instance pure whey protein concentrate or isolate (when produced correctly, those are made by just filtering the whey, without adding anything, though even some of the companies that do not add sweeteners still add lecithin as emulsifier and an anti-caking agent, to be able to market the powder as "instant"; however these additives are far more benign and in far smaller quantities than sweeteners and fragrances).