serialize_typed_slice / deserialize_typed_slice for Contiguous Numeric Arrays (10X or more performance) #3057

Add optional methods to serde::Serializer and serde::Deserializer for bulk serialization/deserialization of contiguous numeric slices. This enables binary formats to memcpy typed arrays in a single operation instead of processing elements one at a time.

The lack of this support is causing Rust binary format encode/decode to perform around 10X slower than C++ libraries like Glaze when handling large, contiguous arrays of numeric data. This is critical for scientific computing. And, performance is even worse with serde for smaller numeric types.

Motivation

Binary formats like CBOR, FlatBuffers, Cap'n Proto, Avro, and Parquet can represent typed numeric arrays as contiguous byte regions. When deserializing Vec<f64> with 100K elements, the ideal operation is a single memcpy of 800KB directly into the Vec's allocation. Similar for encoding.

Today, serde forces element-by-element processing:

1. serialize_seq(Some(100000)) — serializer sees the count but not the element type
2. 100,000 calls to serialize_element -> each calls serialize_f64 -> each writes 8 bytes

This adds ~10x overhead compared to a bulk copy. Benchmarks from a BEVE implementation:

The serde path is 7x slower than a direct bulk write, and 16x slower than what the hardware can do. The overhead is pure per-element dispatch.

This gap affects every binary format crate. It's the reason serde_bytes exists for &[u8], and the same problem exists for &[f32], &[f64], &[u32], &[i64], etc. Related issues: #2028, #2120, #309, #915, #2680.

Proposed API

Marker Trait

A sealed marker trait identifies types eligible for bulk serialization. Implemented inside serde for all fixed-size numeric primitives:

/// Marker trait for types that can be serialized as contiguous typed slices.
/// Sealed — only implemented for fixed-size numeric primitives.
pub trait TypedSliceElement: Serialize + Sized + private::Sealed {
    const ELEM_TYPE: TypedSliceType;
}

#[derive(Debug, Clone, Copy)]
#[non_exhaustive]
pub enum TypedSliceType {
    U8, U16, U32, U64, U128,
    I8, I16, I32, I64, I128,
    F32, F64,
}

// Implemented for: u8, u16, u32, u64, u128, i8, i16, i32, i64, i128, f32, f64
impl TypedSliceElement for f64 {
    const ELEM_TYPE: TypedSliceType = TypedSliceType::F64;
}
// ... etc

Serializer Side

Add a default method to serde::Serializer:

pub trait Serializer: Sized {
    // ... existing methods ...

    /// Serialize a contiguous slice of fixed-size numeric values.
    ///
    /// The default implementation falls back to `collect_seq` (element-by-element).
    /// Binary formats can override this to perform a bulk write.
    ///
    /// The typed slice is passed directly — the format crate decides how to
    /// access the raw bytes. No unsafe code is required in serde itself.
    fn serialize_typed_slice<T: TypedSliceElement>(
        self,
        slice: &[T],
    ) -> Result<Self::Ok, Self::Error> {
        self.collect_seq(slice)
    }
}

The default calls collect_seq — exactly the same code path as today. No unsafe, no overhead for formats that don't override.

A binary format overrides it to access the raw bytes and memcpy:

// In a binary format crate's Serializer impl (e.g. beve, msgpack, cbor)
fn serialize_typed_slice<T: TypedSliceElement>(
    self,
    slice: &[T],
) -> Result<Self::Ok, Self::Error> {
    self.write_typed_array_header(T::ELEM_TYPE, slice.len())?;
    // The unsafe lives here, in the format crate — not in serde
    let bytes = unsafe {
        std::slice::from_raw_parts(
            slice.as_ptr() as *const u8,
            std::mem::size_of_val(slice),
        )
    };
    self.writer.write_all(bytes)?;
    Ok(())
}

Serialize Impls for Vec/Slice

Serde's existing Serialize impls for Vec<T> and [T] would call serialize_typed_slice when T implements TypedSliceElement:

// Inside serde — no unsafe needed
impl<T: Serialize + TypedSliceElement> Serialize for [T] {
    #[inline]
    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
        serializer.serialize_typed_slice(self)
    }
}

For Vec<T> where T is NOT a TypedSliceElement (e.g. Vec<String>, Vec<MyStruct>), the existing collect_seq path is used unchanged.

Formats that don't override serialize_typed_slice get the default which calls collect_seq — identical to today's behavior. No breaking changes.

Deserializer Side

Add a corresponding method to serde::Deserializer:

pub trait Deserializer<'de>: Sized {
    // ... existing methods ...

    /// Hint that the caller can accept a contiguous typed slice.
    ///
    /// If the format supports bulk reads for the given type, it may
    /// call `visitor.visit_typed_slice`. Otherwise, falls back to
    /// `deserialize_seq`.
    fn deserialize_typed_slice<T: TypedSliceElement, V: Visitor<'de>>(
        self,
        visitor: V,
    ) -> Result<V::Value, Self::Error> {
        self.deserialize_seq(visitor)
    }
}

Add a visitor method:

pub trait Visitor<'de>: Sized {
    // ... existing methods ...

    /// Called when the deserializer can provide the entire typed slice
    /// as a contiguous `&[T]`.
    ///
    /// Default: falls back to an error, causing the deserializer to
    /// use `visit_seq` instead.
    fn visit_typed_slice<T: TypedSliceElement>(
        self,
        slice: &[T],
    ) -> Result<Self::Value, E> {
        Err(de::Error::invalid_type(/* ... */))
    }
}

The Deserialize impl for Vec<f64> would call deserialize_typed_slice::<f64>(visitor). A binary format's override reads the raw bytes and constructs &[f64] (or fills a Vec<f64> directly), then calls visitor.visit_typed_slice(slice). If the format doesn't override, it falls back to deserialize_seq -> visit_seq -> element-by-element.

Design Principles

No Unsafe in Serde

All unsafe code lives in format crates that opt into the fast path. Serde's core, its derive macros, and its Serialize/Deserialize impls for standard types remain entirely safe. The serialize_typed_slice method receives &[T] (a safe typed slice); the format crate decides whether to cast it to raw bytes.

Backward Compatible

Both methods have default implementations that fall back to the existing element-by-element path. Existing format crates and Serialize/Deserialize impls continue to work unchanged. Formats opt in by overriding the methods.

Format Agnostic

The serializer receives &[T] directly, not pre-encoded bytes. Each format decides its own byte order and encoding. Little-endian formats memcpy; big-endian formats byte-swap; text formats use the default fallback. No assumptions about wire format are baked into serde.

Scope Is Narrow

This adds two methods with defaults to existing traits, one marker trait, and one small enum. It does not change the derive macros, the data model, or any existing method signatures. The TypedSliceType enum is #[non_exhaustive] so it can be extended later (e.g. F16, Complex64).

Mirrors serialize_bytes

This is the same pattern as the existing serialize_bytes / deserialize_bytes / visit_bytes methods, which exist specifically because &[u8] needed a fast path. This proposal generalizes that pattern to all fixed-size numeric types.

What This Enables

With this change, binary format crates can:

• Serialize Vec<f64> by writing a typed header + memcpy of the slice
• Deserialize Vec<f64> by reading the header + read_exact directly into the Vec's allocation
• Achieve throughput limited by memory bandwidth, not per-element dispatch

Text formats (JSON, TOML, etc.) are unaffected. They use the default fallback and behave exactly as they do today.

Alternatives Considered

serde_bytes-style wrapper crate: Works for &[u8] but doesn't scale. You'd need serde_f64s, serde_u32s, etc., each requiring wrapper types and #[serde(with = "...")] annotations. This is what the ecosystem has been doing as a workaround, and it's ergonomically poor.

Rust specialization: Would allow specializing Serialize for Vec<T> where T is numeric. But specialization is unstable, has soundness concerns, and caused type inference breakage when attempted (#309).

Custom trait per format: Format crates can define their own encode/decode traits with derive macros. This works but fragments the ecosystem. Types can't be shared across formats, and users need separate derives for each format they target.