Repository navigation
Error compiling HACL* Blake2 support for macOS universal binaries #123748
Description
Activity
macOS universal builds are a special form of cross-compile builds that take advantage of the built-in support in Apple's
XcodeorCommand Line Tools for Xcodefor automatic multi-architecture building and linking into multi-archfatbinaries. One potential gotcha of this approach is when there are architecture-dependent configure tests (inconfigure) that depend on the execution of test code on the build machine. Of course, standardcross-compilebuilds on macOS and other platforms have the same gotcha. For macOS universal builds, this kind of a problem has occasionally arisen in the past and one solution that was adopted was to provide an additional header file, Include/pymacconfig.h, whose purpose is to move "some of the autoconf magic to compile-time when building on macOS", so overriding problematic configure-time build tests with conditional code. If it turns out that this is that case here (and it seems likely that it would be), it may be possible to replace a problematic autoconf test with code in thisincludefile.Note, that this is a potential release blocker issue as the macOS installer binaries we provide with each release are built as a universal2 build. The first 3.14 release, 3.14.0a1, is currently scheduled for 2024-10-15.
Reacted by Gregory P. SmithRight, except that here it's made even more complicated by the fact that some files should be included in the Intel builds but not the ARM builds. I'm not sure how to achieve that except to special-case the list of files that go into the build with an Apple-specific bit of logic...?
What kind of files and at what point are they used: configuring, building, installing, executing?
The files Modules/_hacl/Hacl_Hash_Blake2s_Simd128.c and Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c should only be compiled on x64/x86. They are then linked into the python executable. They may be used at runtime if the CPU that python is then executed on has the right support.
So, in more detail:
- at configure-time, we assess whether the compiler supports -msse... -mavx... -mavx2... for the chosen target (if yes: add those two files to the list of files to be built and linked in; if not: leave those two files out)
- at build-time, we compile those files if they were added
- at install-time, these files are simply linked into python.exe
- at execution, we probe the cpu and may execute the code in those files if the right CPU instructions are available
Let me know if I can provide more details? Thanks!
The files Modules/_hacl/Hacl_Hash_Blake2s_Simd128.c and Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c should only be compiled on x64/x86. They are then linked into the python executable. They may be used at runtime if the CPU that python is then executed on has the right support.
Based on this, it sounds like there's no reasonable prospect of getting the Simd128 and Simd256 files to compile on ARM64. AFAIK it's not possible to compile a file for one architecture and link into a universal build - I might be wrong on that, but if I am, I'm going to guess the configure script is going to be messy.
On that basis, it sounds like universal builds won't be able to support those options. That's easy enough to override in the configure process - an extra if block checking for universal on Darwin can disable the option.
I agree it's a little weird that a pure x86-64 build will have support when ARM and universal won't, but given the new platform that the majority of macOS users are on won't support it, I don't think many users will notice the discrepancy.
How about putting an #ifdef in those files that makes them empty when on arm64? Would that help? Like:
#if define(HACL_CAN_COMPILE_SIMD128) ... previous contents of the file ... #endifknowing that HACL_CAN_COMPILE_SIMD128 is not defined on ARM64.
@gpshead might have further thoughts
How about putting an #ifdef in those files that makes them empty when on arm64? Would that help? Like:
#if define(HACL_CAN_COMPILE_SIMD128) ... previous contents of the file ... #endifknowing that HACL_CAN_COMPILE_SIMD128 is not defined on ARM64.
That's the thing though - universal builds are implemented as a single compilation passes, so HACL_CAN_COMPILE_SIMD128/256 is defined. These are turned on because the compiler flags that enable them are legal compiler options when x86_64 is one of the compiler targets, so the constant is defined by the single-pass configure script.
If we were going down the
#definepath, it would need to be based on#if defined(__APPLE__) && defined(__arm64__), or something like that. However, I think catching this at theconfigurelevel makes more sense, even though it does mean the blake2b simd128/256 implementations won't be available for universal builds running on x86_64, where they otherwise would be.A potential fix based on an improved autoconf check: #123927
Does macOS really require all files in a universal2 build to be the same? that seems silly. it is common to separate arch specific code into its own files.
editoral comments about their toolchain choices aside (clearly they channeled practicality vs purity there), if that is true, just making the simd files have C preprocessor checks that effectively make them empty when the aarch64 side of compilation is running makes sense. the x86_64 side of the compilation (surely there are two independent compiler passes running behind the scenes of their
cc -arch x -arch ycommand line) will still be happy and compile useful code instead of an empty object file for those.If we were going down the #define path, it would need to be based on
#if defined(__APPLE__) && defined(__arm64__), or something like that. However, I think catching this at the configure level makes more sense, even though it does mean the blake2b simd128/256 implementations won't be available for universal builds running on x86_64, where they otherwise would be.I'd actually prefer the #define path. A configure test cannot understand the dual compilation and would unnecessarily leave Intel performance on the table. There are a ton of Intel mac's out there and I assume they'll probably be supported until 2030. It becomes more important in the future if/when we get arch specific accelerated HACL* SHA implementations so that hashlib doesn't need OpenSSL to be performant. (blake2 is less important vs those "Standard"s)
Reacted by Ned DeilyDoes macOS really require all files in a universal2 build to be the same? that seems silly. it is common to separate arch specific code into its own files.
Not unless you're compiling multiple architectures in a single pass, as CPython does. If we compiled for x86_64, then compiled for ARM64, then merged the two binaries into a universal binary, the problem wouldn't exist. However, the single-pass autoconf-based build determines the modules to be compiled, and the flags to be passed in to that compile, based on a single pass compiler check.
I'd actually prefer the #define path. A configure test cannot understand the dual compilation and would unnecessarily leave Intel performance on the table. There are a ton of Intel mac's out there and I assume they'll probably be supported until 2030. It becomes more important in the future if/when we get arch specific accelerated HACL* SHA implementations so that hashlib doesn't need OpenSSL to be performant. (blake2 is less important vs those "Standard"s)
Ok - I'll take a look and see what I can make work. My first attempt at doing this failed, but I didn't look too closely at why - I'm probably missing something obvious.
Thanks, I agree that the pass with the #defines sounds better. This can probably be done with a stub file
Hacl_Hash_Blake2b_Simd256_Universal.cthat does#if defined (...) #include Hacl_Hash_Blake2b_Simd256.c #endif
so as to leave the ingestion of upstream HACL* unchanged, without having to hack _hacl/refresh.sh in cpython.
Reacted by Ned Deily and Russell Keith-MageeQ: do we have a macOS universal2 buildbot?
Q: do we have a macOS universal2 buildbot?
Not that I'm aware of. It's easy enough to add the appropriate options to CPython
configureto trigger a universal2 build. What currently isn't so straightforward is the availability of universal builds of the external third-party libraries that are needed by the standard library and that are not already supplied by macOS: mainlylibsslandlibcryptofrom OpenSSL,liblzma,libmpdec,Tk,gdbm(if GPL-licensing isn't an issue), and potentially newer versions of a few others (sqlite3, ncurses, etc). Homebrew does not provide universal builds (and its decision to use totally different install prefixes for Intel and Apple Silicon builds complicates this). MacPorts does support universal builds of most/all of these but does not provide pre-built universal binaries. At some point soon, we may be able to leverage builds needed for the macOS installer and, possibly, for iOS binaries, as well.- added a commit that references this issue
on Sep 16, 2024 This patch does not work for standalone builds on arm64 macOS. This is because the --with-universal-archs=universal2 parameter is not passed to the standalone build of arm64:
checking for --enable-universalsdk... no checking for --with-universal-archs... nothe statement
if test "$UNIVERSAL_ARCHS" == "universal2"; thenwill failed
Attached is a dirty fix for those of you who need to compile on arm64 macOS systems.
@naizhao I believe arm64 builds on MacOS are being tested already, so how come this was not caught? Is the standalone build you refer to something that is not currently under CI?
Use the following options to compile on an M2 machine to build packages that arm64 only:
./configure --prefix=/Applications/ServBay/package/python/3.14/3.14.0a5 --enable-ipv6 --enable-loadable-sqlite-extensions --with-openssl=/Applications/ServBay/package/common/openssl/3.2 --enable-optimizations --with-system-expat --with-system-libmpdec --with-readline=editline --with-lto --enable-framework=/Applications/ServBay/package/python/3.14/3.14.0a5 Python 3.9+ detected checking build system type... aarch64-apple-darwin23.5.0 checking host system type... aarch64-apple-darwin23.5.0 checking for Python interpreter freezing... ./_bootstrap_python checking for python3.14... no checking for python3.13... no checking for python3.12... python3.12 checking Python for regen version... Python 3.12.5 checking pkg-config is at least version 0.9.0... yes checking MACHDEP... "darwin" checking for --enable-universalsdk... no checking for --with-universal-archs... no checking for --with-app-store-compliance... not patching for app store compliance checking for xcrun... yes checking macOS SDKROOT... /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether the compiler supports GNU C... yes checking whether gcc accepts -g... yes checking for gcc option to enable C11 features... none needed checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /usr/bin/grep checking for a sed that does not truncate output... /usr/bin/sed checking for egrep... /usr/bin/grep -E checking for CC compiler name... clang // blablabla
If you get
$UNIVERSAL_ARCHS, you will get32-bitbecauseUNIVERSAL_ARCHS=“32-bit”inconfigurewithout passing in--enable-universalsdk. But when compiling this local host, there is no need to pass in--enable-universalsdkto get the universal version.
So it's not a good idea to use $UNIVERSAL_ARCHSWhile this works to enable universal2 builds on arm64 macOS systems, unfortunately it does not work to enable universal2 builds on x86_64 systems. The fact that the build works on arm64, together with the fact that the macOS Python distribution is built on arm64, means that this fix does solve the important problem and allow the release to build.
But shouldn't it be possible to do a universal2 build on either platform?
Here is the error I got when attempting a standard (i.e. universal2) build of Python 3.14.0 on an x86_64 macOS 15 system:
In file included from ./Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:26: ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:42:32: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 42 | Hacl_Hash_Blake2b_Simd256_init(Lib_IntVector_Intrinsics_vec256 *hash, uint32_t kk, uint32_t nn); | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:47:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 47 | Lib_IntVector_Intrinsics_vec256 *wv, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:48:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 48 | Lib_IntVector_Intrinsics_vec256 *hash, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:57:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 57 | Lib_IntVector_Intrinsics_vec256 *wv, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:58:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 58 | Lib_IntVector_Intrinsics_vec256 *hash, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:69:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 69 | Lib_IntVector_Intrinsics_vec256 *hash | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:74:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 74 | Lib_IntVector_Intrinsics_vec256 *st, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:81:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 81 | Lib_IntVector_Intrinsics_vec256 *st | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:84:1: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 84 | Lib_IntVector_Intrinsics_vec256 | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:89:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 89 | Lib_IntVector_Intrinsics_vec256 *s, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:97:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 97 | Lib_IntVector_Intrinsics_vec256 *s, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:105:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 105 | Lib_IntVector_Intrinsics_vec256 *src, | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:106:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 106 | Lib_IntVector_Intrinsics_vec256 *dst | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:111:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 111 | Lib_IntVector_Intrinsics_vec256 *fst; | ^ ./Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:112:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 112 | Lib_IntVector_Intrinsics_vec256 *snd; | ^ ./Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:38:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 38 | Lib_IntVector_Intrinsics_vec256 *wv, | ^ ./Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:39:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256' 39 | Lib_IntVector_Intrinsics_vec256 *hash, | ^ ./Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:57:3: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256' 57 | Lib_IntVector_Intrinsics_vec256 mask = Lib_IntVector_Intrinsics_vec256_zero; | ^ ./Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:76:3: error: use of undeclared identifier 'mask' 76 | mask = | ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated.@culler "But shouldn't it be possible to do a universal2 build on either platform?"
Yes, it definitely should. FWIW, I'm able to do so using the current head of the main (3.15) branch (macOS 26.1, Xcode 26.1.1, --enable-universalsdk=/ --with-universal-archs=universal2). If you are unable to build successfully, please open a new issue with the relevant configuration info.
I was able to build successfully on an M4 mac mini. The failure was on an Intel mac mini
whose CPU did not make the cut for macOS 26, so is still running macOS 15. I only specified --prefix and some configure args for openssl. I will try using the configure args that you suggest. If that fails I will open a new ticket.By the way, I was able to do the build by modifying the configure script so that it would
not build the Blake2 module on either platform.
Bug report
Bug description:
#99108 tracks the addition of a native HACL implementation to CPython. #119316 added an implementation of Blake2 to
hashlib.This compiles fine on single architecture macOS builds (as verified by CI); but universal2 builds running on an ARM64 laptop generate a compilation error:
To reproduce the problem: on a macOS machine, configure the build with:
This will eventually yield the compilation error:
From what I can make out, the error comes from the detection of
-mavx2support. On a bare configure on an ARM64 machine,-mavx2support is apparently unsupported:and as a result, the
Hacl_Hash_Blake2b_Simd256.cmodule isn't compiled. However, when universal support is enabled,-mavx2is supported:and the module is included. Based on recent configure logs for x86_64 macOS builds, it appears that
-mavx2is supported on x86_64.I'm not sufficiently familiar with the subject matter to comment on whether the fix here is to fix the autoconf detection to disable the problematic module on universal builds, or to correct the implementation so that it can compile for universal builds.
Tagging @msprotz @R1kM as the authors of the recent HACL* changes.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs