Being surprised or upset that there are "illegal numbers" is like being upset that there are illegal configurations of matter and energy. "It's not murder, it's just a configuration of atoms!" doesn't fly.

If you believe the universe is discrete, this becomes stronger than a mere analogy - configurations of matter and energy are in fact numbers.