There are simply lacking of such strong requirements in language standards. C/C++ even have the specific "linkage" concept to abstract the binary details under the source form away. And you may know, many libraries are distributed by binaries.
The standards implying binary compatibility rules are about ABI (application binary interface), which usually depend on the ISA (instruction-set architecture) or the OS (if any) being used. You cannot have the unique one once there are multiple ISAs/OSes supported. Even when you only want to rely on some external "exchanging" representations not tied to specific ISAs, there are already plenty of candidates: CLI, JVM, WebAssembly... Plus there are more than one executable (and mostly, runtime loadable) image formats widely used (PE/COFF, ELF, Mach-O ...). You will not have the unique combination, and any attempts to ensure it "work across compilers" in that way will likely finally just add a new instance not fully compatible to existing ones, making it more fragile.
Standards are often incomplete, or full of "implementation specific" behaviour, since AIUI standards often end up catering to implementations, instead of the other way around (For example C/C++ standards, you can read a plethora of blog posts about the experience of people trying to contribute to them, and some of the hurdles are related to how strongly tied to existing implementations they are). That means you can often have 2 "standards compliant" compilers that are wildly different. Another reason is compiler extensions. Sometimes a compiler is "standards compliant", but also implements a superset of the standard (Sometimes by default, sometimes under a flag) which means code gets written for that "superset" instead of according to a "standard" (for example, the linux kernel and gcc extensions to C).
Typically, some library features contain things that require cooperation with a specific compiler to work correctly. Consider something like std::is_standard_layout in C++, or java.lang.Object in Java, or std::panic::catch_unwind in Rust.