SurPlus

A fix for the race condition that MacOS 11.3+ exhibits on unsupported Macs

26 September 2021

It took me six months, a lot of sleepless nights, and I ended up having to write my own debugger to get the answers that I needed, but I finally found the source of the race condition that's been plaguing older Mac Pros (and others) using Big Sur 11.3+ and Monterey. ~~I still don't know why this problem doesn't seem to appear on newer systems; my focus has been on finding the problem and creating a solution, and the question of newer systems' success seems like an academic exercise for someone with more time on their hands.~~ (Thanks to insight from @vit9696, it seems that newer Macs don't suffer from this problem because those CPUs support the rdrand instruction, meaning they don't need floating-point access during early boot.)

The patch posted here is intended to be incorporated into an OpenCore config.plist file. If you need assistance with this task, please use one (or more) of the following resources:

The patch itself appears at the bottom of this page, in a format suitable for cutting-and-pasting into the appropriate location in an OpenCore config.plist file.

Questions, comments, or discussion about the "race condition" bug or this patch should be directed to the SurPlus thread on MacRumors.

If this information or the patch extends the usable life of your classic Mac, or you find other value in them, donations are gratefully accepted. (To be clear, I am not a tax-exempt organization, charity, or non-profit, and there is no tax benefit to you should you choose to donate - just my heartfelt thanks.)

<b>Update 30oct21</b> - Monterey 12.1b1 was seeded to developers this week, and adds a new wrinkle to this ongoing mess. See this MacRumors thread for more details (a writeup similar to this one will eventually appear here, either as a separate README or a separate repository). Also, I have updated the MaxKernel values in the enclosed SurPlus patches to 21.2.0, allowing them to be applied to 12.1b1.

<b>Update 20dec21</b> - It appears that the released version of Monterey 12.1 does not require the SurPlus patch. While this may change in a subsequent release, it's certainly welcome news. Also, note that the MonteRand patch referenced in my 30oct21 update is only required for 12.1b1; other betas and releases do not exhibit the problem that MonteRand addresses.

<h2>The problem and solution for non-programmers:</h2>

MacOS consists of many separate parts working together. Two of those parts, cryptography and kernel memory management, are interdependent (each makes calls to the other). Starting with 11.3, there is a circular dependency between them - crypto needs something from memory management, and that same part of memory management needs something from crypto. If that dependency is encountered at the wrong time during boot, a deadlock occurs - crypto is waiting on memory management, and memory management is waiting on crypto. The boot process is then hung until the system is forcibly stopped.

There are several possible solutions, some better than others. I've refined the solution I'm now posting to where it only modifies three bytes of code, and it doesn't impair the functionality or security of the system.

<h2>The problem and solution for more technical folks:</h2>

First, a bit of relevant background:

Intel does something interesting with floating point. "Floating point" (FP) includes not only the obvious (floating point math), but also all of the vector and SIMD instructions (SSE, AVX, basically anything that touches the XMM/YMM/ZMM registers). When an operating system does a context switch (where the CPU pauses execution of one thread to continue execution of another), it has to save and restore all the registers so nothing gets lost. When you add in all the SIMD registers, that's a lot of overhead - so Intel lets the OS assume that FP is unlikely to happen, and will throw a #NM exception if any FP operations occur. That way, context switches can just deal with the "normal" registers and ignore the SIMD registers unless a thread actually uses them; if an FP operation occurs, the system catches the #NM exception, allocates some memory to save the SIMD registers, and saves/restores those registers when switching contexts for that thread.
Since at least El Capitan (and probably earlier), the MacOS kernel has utilized "zone allocation" (zalloc) for internal dynamic memory allocation. zalloc has evolved slowly over the years, but it changed rapidly during Big Sur's various releases; Apple added both refinements and security. While earlier versions of zalloc simply worked with contiguous chunks of memory, in 11.3 Apple introduced random gaps in the memory pages (presumably for security reasons, although it's a bit unclear to me what vulnerabilities this scheme mitigates - if an attacker has access low enough for that to matter, you've already lost the battle).
The corecrypto kext handles most cryptography tasks for MacOS, including random number generation. Like many other kexts, it tests the hardware it's running on and uses the most advanced instructions the hardware will support (e.g. AVX2 on newer machines, AVX1 or AES-NI or SSE3 on older machines). I was surprised at the extent to which corecrypto is used - many/most MacOS subsystems touch corecrypto at some point.

OK, so here's the gist of the problem. At startup, MacOS launches multiple threads to initialize the system - to discover and configure the hardware, etc. At some point, a thread will call IOLockAlloc() (to allocate a lock group for IOKit), which will in turn call zalloc to allocate memory. At this early stage of booting, zalloc has not yet initialized, so it does that now. Remember those random memory gaps I mentioned? zalloc initialization needs random numbers to make the gaps random, so it calls early_random() to get some random numbers.

So far, so good. Now, at this point in the boot, corecrypto may or may not have been initialized yet. If it has not yet been initialized, early_random() will just use its own SHA1 random number generator. This is the "success" case, because that generator doesn't use any floating point instructions, so early_random() will return a random number, zalloc will finish initializing, and everybody's happy.

Unfortunately, corecrypto's initialization code is short and sweet, so most of the time, corecrypto is already initialized when that call to early_random() occurs. In that case, early_random() hands the request to corecrypto, which chooses the best instructions to use (on the Mac Pro 5,1-era machines, that's SSE3 and AES-NI, both of which are floating-point as far as the CPU is concerned). corecrypto then acquires a thread lock (the necessity of which is not entirely clear to me), then it starts generating the random number by executing a floating point/SIMD instruction - which throws a #NM exception so MacOS knows to keep track of the SIMD registers. That exception calls an OS routine called fpnoextflt(), which (among other things) allocates a buffer to save the SIMD registers for this thread. Where does it get that buffer? zalloc, of course.

As you'll recall, the only reason we're executing anything in corecrypto at all is because zalloc needed some random numbers to initialize itself. Therefore, when fpnoextflt() wants to allocate a buffer, zalloc is still uninitialized, and it tries to request some random numbers from corecrypto. (Sound familiar?)

This unhappy set of circumstances would lead to an infinite loop of zalloc requesting random numbers and corecrypto's FP instructions triggering #NM exceptions that call zalloc, except for the corecrypto thread lock I mentioned. The second time through, corecrypto tries to acquire that same lock again. The lock mechanism doesn't care that the same thread already holds that lock, it just knows that the lock is held, and the current request has to wait for the lock to be released, which effectively puts the thread to sleep while it waits for a lock it can never acquire.

The "race" here is between the zalloc and corecrypto subsystems getting initialized (or, more accurately, between zalloc getting initialized and the execution of any SIMD or floating-point instruction, which is most likely to occur in corecrypto). If a floating-point/SIMD instruction occurs at any point before an attempt to use zalloc, fpnoextflt() will invoke zalloc, which will invoke corecrypto, which will deadlock (as described above). If no floating-point/SIMD instructions occur before zalloc is invoked/initialized, the boot proceeds normally, and everybody's happy. For all the current problematic MacOS versions (11.3-11.6 and every 12.x Monterey beta to date), the solution is to delay using corecrypto until the zalloc subsystem is initialized.

After quickly rejecting a kext-based solution (because any kext would just become part of the race condition), I developed a "blunt instrument" solution that just skipped over the zalloc code that introduces random gaps

SurPlus

Install / Use

README

SurPlus

A fix for the race condition that MacOS 11.3+ exhibits on unsupported Macs