IsNull/NotNull binding ignores ancestor nullability for required nested fields #4067

Apache Iceberg version

main (development)

Please describe the bug 🐞

Description

IsNull/NotNull binding silently drops rows/files for a required field nested under an optional ancestor struct.
Related iceberg java/rust issues/PRs:

• apache/iceberg#14270
• apache/iceberg-rust#3292

Bug:

• BoundIsNull/BoundNotNull.__new__ folds to AlwaysFalse()/AlwaysTrue() using only a field's own required flag.
• A required leaf can still be absent if an optional ancestor struct containing it is null
• plan_files() silently drops matching files once IsNull/NotNull hits a required field under an optional parent.

Example

from pyiceberg.catalog.sql import SqlCatalog
from tempfile import TemporaryDirectory
from pathlib import Path
import pyarrow as pa

with TemporaryDirectory() as tmp:
    cat = SqlCatalog("local", uri="sqlite:///:memory:", warehouse=Path(tmp).as_uri())
    cat.create_namespace("ns")

    # schema: s is optional, its only child x is required
    dtype = pa.struct([pa.field("x", pa.int64(), nullable=False)])
    data = pa.table({"s": pa.array([None], type=dtype)})
    table = cat.create_table("ns.t", schema=data.schema)
    table.append(data)  # s = None for the one row -> s.x is absent too

    import pyiceberg.expressions as e
    files = list(table.scan(row_filter=e.IsNull("s.x")).plan_files())

    print(len(files))  # 0  <-- empty, should contain the file with the s = None row

Proposed fix

• Add Schema.is_field_required_in_path(field_id) — walks the existing ancestor index to check the field and every ancestor is required.
• Remove the leaf-only fold from BoundIsNull/BoundNotNull.__new__; do the fold in IsNull.bind()/NotNull.bind() instead, where the schema is already in scope. No change to BoundReference.
• No public API changes.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time