The major challenges for our team was uBoot and DeviceTree - kernel, rootfs, and userland was pretty easy to sort out - userland was generally ARMv7A with vfpv4/NEON - hence the narrow selection of SOC's supported
The cool stuff was that we had two partitions for the kernel and core stuff - so for OTA updates, there, if sig's didn't match, the commit for the update would keep things on a known good.. Matching things up between our stuff and containers - UnionFS, and there, we took care of system files, userfiles, and ephemeral things log /var/log and /tmp which was in tmpfs.