I think a major thing with viruses is access to filesystem.

If programs asked for permission to access folders or we gave the program their own volume to work within then they couldn't access, run or modify sensitive/important files.

This is how linux systems are used, when an higher level of security is needed.

The security levels look like this(lowest to highest):

Standard installation > AppArmor > Selinux > selinux with default-deny(everything not explicitly allowed is denied)

There's also fapolicyd as well, but the problem is these rule sets are often updated automatically by the package installation manager such as RPM. So while this can stop rogue binaries from running in unapproved locations anyone who has access to the box to install a rogue binary and have it run at elevated permissions would be able to install a package with the package manager. Most of the default rule sets trust things that are installed with the package manager in order to make the operating system usable.

Windows has an built in method, but make backups, since you can deny yourself access to critical things.

Hypothetically, you'd rename the malicious payload after a known whitelisted file, like svchost.exe.

And if you used a hashing algorithm, RC4 or the like, what fun that would be when binaries were updated!

What problems will occur?

You would need to use a hashing algorithm that is not public and cryptographic in some way. Otherwise, a malicious actor can "pad" malware until it reaches a desired checksum and passes the whitelist.

This would also make it extremely difficult for software to be distributed and updated in the way it is currently on Mac and Windows.

Of course hash is the key, not name.

It's been used for years and does work. There are products that try to do this, Google experimented with it for a while. And it's actually the default on macOS of course, Apple whitelists all apps by notarizing them and you can apply custom policy on top.

One problem is that on Windows signing is optional and only about 30% of software is signed, but the only way to handle software updates and changes is to whitelist publishers. Or sometimes signing is done wrong, like only signing the installer but not the installed files.

Another is that it requires incredibly responsive IT staff. If they don't whitelist something fast enough it can slow down the entire organization.

Yet another is that code signing is hard to do in a leak proof manner. Any program that loads code as data, for example any interpreter, can become a code signing "hole" and whitelisting it is equivalent to whitelisting everything. A good example of problem cases: plugins.

Yet another is that to whitelist something requires the integrity of the whitelisted thing to be enforced otherwise malware can just be injected into it. Windows has a mechanism to do this called MSIX but not much uses it partly due to lack of good tooling. My company makes a product that fixes that to some extent but because MSIX is a package manager not an installer framework it can require re-engineering to adopt, for example, you can't customise install components or show other UI at install time.

Apple is slowly moving macOS towards solving all of these problems and there support for binary whitelisting is shipped out of the box, although most people don't realize it and it's not well documented. I assume there are edr vendors who have used those APIs to create products though.