Visitar URL original
Add opt-in non-reflective object comparer for NativeAOT by hydraxman · Pull Request #2308 · MessagePack-CSharp/MessagePack-CSharp · GitHub
Skip to content

Add opt-in non-reflective object comparer for NativeAOT - #2308

Merged
AArnott merged 5 commits into
MessagePack-CSharp:masterfrom
hydraxman:user/Shaopeng.Bu/messagepack-nativeaot-comparer
Oct 12, 2026
Merged

AArnott merged 5 commits into
MessagePack-CSharp:masterfrom
hydraxman:user/Shaopeng.Bu/messagepack-nativeaot-comparer

Conversation

@hydraxman

@hydraxman hydraxman commented Oct 8, 2026 •

Copy link
Copy Markdown

Summary

Add an explicitly opt-in, non-reflective object-key comparer for NativeAOT while leaving the default security behavior unchanged.

The existing ObjectFallbackEqualityComparer.GetHashCode(object) dynamically invokes the virtual generic comparer factory for the key's runtime type. NativeAOT diagnoses that MakeGenericMethod path with IL2060 and IL3050. Preserving arbitrary existing generic overrides requires keeping the legacy behavior available; replacing it automatically on NativeAOT would change an existing contract.

This PR adds:

  • A default-off MessagePack.Security.UseNonReflectiveObjectComparer feature switch, annotated on the net9.0 library asset.
  • A separate object comparer selected by that switch, allowing the legacy implementation to be eliminated at publish time.
  • One protected overload, IEqualityComparer GetHashCollisionResistantEqualityComparer(Type type), for statically written custom comparer dispatch.
  • Contract regression tests and a standalone JIT/native executable.

The diff is limited to four files: one production file, the existing unit test file, and the standalone fixture's project and program. No README, workflow, or solution changes are included. The standalone native fixture is run explicitly; the existing CI is unchanged and does not run it automatically.

Related: #1666 (broader trimming compatibility) and #1202 (preserving the legacy extension contract). This PR addresses only the security object-comparer path; it does not claim whole-library NativeAOT compatibility or resolve the broader scope of #1666.

Explicit opt-in and compatibility

<ItemGroup>
  <RuntimeHostConfigurationOption
      Include="MessagePack.Security.UseNonReflectiveObjectComparer"
      Value="true"
      Trim="true" />
</ItemGroup>

This is an application-wide publish-time choice, not a per-instance setting. Native elimination requires the annotated net9.0 library asset and a .NET 9 or later NativeAOT toolchain. Consuming an older asset or setting AppContext only at runtime does not provide that guarantee. Configure the switch before MessagePack initializes; runtime toggling is unsupported.

With the switch off, the existing implementation remains unchanged, including JIT generic overrides and NativeAOT applications that already make the needed native instantiations available.

With the switch on:

  • Primitive keys, Guid, DateTime and DateTimeOffset retain their keyed hashes and normalization. Primitive generic overrides still participate, with per-type caching.
  • Boxed enums retain keyed hashing of the full underlying integer, not ordinary Enum.GetHashCode.
  • Arbitrary custom types are rejected unless the new Type overload explicitly returns a comparer.
  • Boxed enum-specific generic overrides also require an explicit branch in that overload. Typed generic calls are unchanged.
  • Object equality, both comparer interfaces, the parameterless factory, and Clone remain available.

Custom security subclasses can connect an existing collision-resistant generic comparer through a statically known type:

protected override System.Collections.IEqualityComparer
    GetHashCollisionResistantEqualityComparer(Type type)
{
    return type == typeof(MyKey)
        ? (System.Collections.IEqualityComparer)this.GetHashCollisionResistantEqualityComparer<MyKey>()
        : base.GetHashCollisionResistantEqualityComparer(type);
}

Use the same pattern for an enum-specific comparer. The returned comparer must implement non-generic IEqualityComparer, as the legacy object fallback also requires. User-supplied comparers retain the existing obligation to provide collision resistance and equality consistency.

No collision protection is disabled. Applications must still select MessagePackSecurity.UntrustedData when appropriate. No serializer payload format, library dependency, target framework, release workflow, or tracked version is changed. No warning suppressions, linker roots, or dependency annotations are added.

Validation

The following local results were obtained on Windows using SDK 10.0.401 at commit 56fafcdd02b33dbf43599d4fde9b482b321c1c60. Subsequent changes remove documentation, workflow additions, and solution registration, and shorten code comments; executable code and test project configuration are unchanged. The repository's pinned SDK was not changed.

Check Result
net9.0 NativeAOT, opt-in, ILCompiler 9.0.20 98 assertions passed; zero build warnings
net10.0 NativeAOT, opt-in, ILCompiler 10.0.12 98 assertions passed; zero build warnings
net10.0 NativeAOT, legacy/default 96 assertions passed; original one IL2060 and one IL3050 remain
net10.0 ARM64 NativeAOT, opt-in Compiled and linked with zero warnings; not executed
net9.0 JIT, legacy / opt-in 96 / 98 assertions passed
Main MessagePack.Tests, net9.0 1086 passed, 6 existing skips, zero failures
Main MessagePack.Tests, net472 1055 passed, 6 existing skips, zero failures
All five library target frameworks Build passed with zero warnings/errors

The opt-in native scan graphs contain zero nodes for ObjectFallbackEqualityComparer. The legacy negative control retains that implementation and its diagnostics, rather than suppressing them.

The native fixture covers all eight enum underlying types, a boxed-only enum with no typed enum factory call, primitive hash parity, equivalent floating-point/date representations, ordinary-hash collision fixtures, custom/enum overrides, cache reuse, clone isolation, unsupported-type rejection, and an exact-wire-format untrusted map round trip. The legacy checks use ordinary typed calls to demonstrate the existing NativeAOT behavior when the native generic instantiations are available; they do not claim support for arbitrary unseen types.

Linux execution and the full solution test suite were not performed locally.

Run the standalone fixture

From the repository root:

dotnet run --project tests\MessagePack.Security.NativeAotTests -c Release -p:PublishAot=false -p:NonReflectiveSecurity=false -- legacy
dotnet run --project tests\MessagePack.Security.NativeAotTests -c Release -p:PublishAot=false -p:NonReflectiveSecurity=true -- opt-in
dotnet publish tests\MessagePack.Security.NativeAotTests -c Release -r win-x64 -p:NonReflectiveSecurity=true -warnaserror -o bin\security-optin
.\bin\security-optin\MessagePack.Security.NativeAotTests.exe opt-in

The fixture defaults to net9.0. -p:SecurityTestTargetFramework=net10.0 selects a newer executable target without retargeting the library. For the legacy native negative control, publish with -p:NonReflectiveSecurity=false, omit -warnaserror, and run with legacy. Add -p:IlcGenerateDgmlFile=true to inspect reachability.

Nathan Bu and others added 2 commits October 5, 2026 00:12
Preserve the legacy default while adding an application-wide publish-time feature switch and a non-generic comparer extension point. Retain keyed hashing and unsupported-type rejection, document custom and boxed-enum migration, and add JIT/native regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 786fa0e8-83da-46dc-a693-b17d03ded372
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 786fa0e8-83da-46dc-a693-b17d03ded372
@hydraxman

Copy link
Copy Markdown
Author

@dotnet-policy-service agree [company="microsoft"]

@hydraxman

Copy link
Copy Markdown
Author

@dotnet-policy-service agree company="microsoft"

@hydraxman
hydraxman marked this pull request as ready for review October 8, 2026 23:48
@hydraxman

Copy link
Copy Markdown
Author

@AArnott I've marked this ready for review. Could you take a look and approve the pending .NET and Unity workflow runs? The CLA check is now passing.

The production change is confined to MessagePackSecurity.cs. The feature is off by default, preserving the existing behavior; opt-in consumers can use the new Type-based hook for statically known custom and enum comparers. Keyed hashing and unsupported-type rejection remain intact.

Local Windows validation covers .NET 9/10 NativeAOT, both JIT modes, and the net9.0/net472 unit tests, with details in the PR description. The opt-in native fixture eliminates IL2060/IL3050 without suppressions. I'd particularly appreciate your feedback on the opt-in contract and the new extension point. Thanks!

Nathan Bu added 2 commits October 9, 2026 08:07
Remove added README content, workflow job, and standalone fixture solution registration. Keep production and regression test contents unchanged; retain usage and validation details in the PR description.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 786fa0e8-83da-46dc-a693-b17d03ded372
Keep only concise API, enum hashing, and legacy test intent comments. Leave executable code unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 786fa0e8-83da-46dc-a693-b17d03ded372

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The NativeAOT fixture is not executed by CI, leaving the core feature without automated regression coverage.

1 open finding
What changed in this PR

Adds an opt-in NativeAOT-compatible object comparer while preserving legacy behavior by default.

Changes:

  • Adds non-reflective primitive, enum, and custom comparer dispatch.
  • Adds regression tests and a standalone NativeAOT fixture.
  • Configures the feature switch for NativeAOT publishing.
File Description
src/​MessagePack/​MessagePackSecurity.cs Implements feature-switched comparer dispatch.
tests/​MessagePack.Tests/​MessagePackSecurityTests.cs Adds comparer contract tests.
tests/​MessagePack.Security.NativeAotTests/​Program.cs Exercises JIT and NativeAOT behavior.
tests/​MessagePack.Security.NativeAotTests/​MessagePack.Security.NativeAotTests.csproj Configures the standalone AOT fixture.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@AArnott
AArnott enabled auto-merge October 12, 2026 18:31
@AArnott AArnott added this to the v3.1 milestone Oct 12, 2026
@AArnott
AArnott merged commit 552b626 into MessagePack-CSharp:master Oct 12, 2026
4 checks passed
This was referenced Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants