Apple has their own custom conventions and ABI (https://developer.apple.com/documentation/xcode/writing-arm6...). I believe you are right that they have their own custom instructions too. The asahi Linux project has good documentation on their wiki about the hardware.
Also important here is the bespoke ASIC functionality on their CPUs - as it's a system on chip, you'll likely need to emulate the full SoC, including all the peripherals the system expects to be there. Unlike x86_64, ARM isn't really designed to do device discovery on extensible buses in the same way. Your peripherals should all be in the same place, many being on the chip on internal interconnects.
Emulating anything would be possible with enough effort. But you'd likely need to emulate a whole host of extra bespoke silicon functionality on the SoC to get it working. For example, given every ARM Mac has a hardware neural engine, you might find unmodified Mac OS assumes it's there and usable for core functionality (or does so in future). It would probably be a lot of work to emulate all of the extra SoC stuff to the point you could boot the OS unmodified. But nothing is impossible - I think iOS has been run in third party emulators etc.
ABI, which calling conventions are one part of, only affects how different pieces of code within the operating system talk to each other. It’s not uncommon to customize it - for instance, Windows does too. [1] Having a custom ABI does not present any obstacle to emulation.
Yeah, agreed. I think of every ARM SoC as more like its entire own architecture, for OS compat purposes. A Raspberry Pi SoC will require an entirely different level of compatibility to an M1 Mac, or a Snapdragon based Chromebook.
I think we've been somewhat spoilt by x86 and how easy loads of different machines will boot the same OS image. It's really really different in ARM land.
To ease execution of x86₆₄ applications on Apple Silicon the Apple processor team implemented the x86 memory model with a series of additional experiences.
Also important here is the bespoke ASIC functionality on their CPUs - as it's a system on chip, you'll likely need to emulate the full SoC, including all the peripherals the system expects to be there. Unlike x86_64, ARM isn't really designed to do device discovery on extensible buses in the same way. Your peripherals should all be in the same place, many being on the chip on internal interconnects.
Emulating anything would be possible with enough effort. But you'd likely need to emulate a whole host of extra bespoke silicon functionality on the SoC to get it working. For example, given every ARM Mac has a hardware neural engine, you might find unmodified Mac OS assumes it's there and usable for core functionality (or does so in future). It would probably be a lot of work to emulate all of the extra SoC stuff to the point you could boot the OS unmodified. But nothing is impossible - I think iOS has been run in third party emulators etc.