The Xbox 360 was based on the same weird, in-order PowerPC 970 derived CPU as the PS3, it just had three of them stuck together instead of one of them tied to seven weird Cell units. The TL;DR of how Xbox backwards compatibility has been achieved is that Microsoft’s whole approach with the Xbox has always been to create a PC-like environment which makes porting games to or from the Xbox simpler.
The real star of the show here is the Windows NT kernel and DirectX. Microsoft’s core APIs have been designed to be portable and platform-agnostic since the beginning of the NT days (of course, that isn’t necessarily true of the rest of the Windows operating system we use on our PCs). Developers could still program their games mostly as though they were targeting a Windows PC using DirectX since all the same high-level APIs worked in basically the same way, just with less memory and some platform-specific optimisations to keep in mind (stuff like the 10MB of eDRAM, or that you could always assume three 3.2GHz in-order CPU cores with 2-way SMT).
Xbox 360 games on the Xbox One seem to be run through something akin to Dolphin’s “Übershaders” - in this case, per-game optimised modifications of an entire Xenon GPU stack implemented in software running alongside the entire Xbox 360 operating environment in a hypervisor. This is aided by the integration of hardware-level support for certain texture and audio formats common in Xbox 360 games into the Xbox One’s CPU design, similarly to how Apple’s M-series SoCs integrate support for x86-style memory ordering to greatly accelerate Rosetta 2.
Microsoft’s APIs for developers to target tend to be fairly platform-agnostic - see Windows CE, which could run on anything from ARM handhelds to the Hitachi SH-4 powered Sega Dreamcast. This enables developers who are mostly experienced in coding for x86 PCs running Windows to relatively easily start writing programs (or games) for other platforms using those APIs. This also has the beneficial side-effect of allowing Microsoft to, with their collective first-hand knowledge of those APIs, create compatibility layers on an x86 system that can run code targeted at a different platform.
Funnily enough, one of the few legitimately impactful non-enterprise uses of AVX512 I’m aware of is that it does a really good job of accelerating emulation of the Cell SPUs in RPCS3. But you’re absolutely right, those things are very funky and implementing their functions is by far the most difficult part of PS3 emulation.
Luckily, I think most games either didn’t do much with them or left programming for them to middleware, so it would mostly be first- and second-party games that would need super-extensive customisation and testing. Sony could probably figure it out, if they were convinced there was sufficient demand and potential profit on the other side.