It seems that by using the PCIe interface for the M.2, once it's being used for the main disk you cannot share it with other peripherals for example GbE otherwise it won't boot if the OS is being accessed from the M.2 interface.
There's a sibling comment that wished for two M.2 slots for the HAT+, perhaps this the reason why.
Could you expand on that?
Are you saying that if I get this HAT and set my rpi5 to use only the that drive without any sd card, I won't be able to use Ethernet port ?
My understanding is that the PCIe expansion is being used for the HAT+ only works for one disk peripheral if it's being used for booting the OS.
For example, since for HAT+ you can have a maximum 2 stacks of HAT+ boards expansion. Let's say you have one M.2 HAT+ and another is GbE board available from PineBerry [1]. The new additional GbE port on the HAT+ (not the original on-board GbE on the RPi 5) will interfere with the M.2 disk on the HAT+ if it's being used for booting.
[1] PineBerry Pi launches five additional HAT+ boards for the Raspberry Pi 5 with 2.5GbE, GbE, M.2 NVMe, and more:
There's a sibling comment that wished for two M.2 slots for the HAT+, perhaps this the reason why.