Visitar URL original
Fix $not filters on dot-notation paths by MaxFreedomPollard · Pull Request #177 · hyparam/hyparquet · GitHub
Skip to content

Fix $not filters on dot-notation paths - #177

Merged
platypii merged 1 commit into
hyparam:masterfrom
MaxFreedomPollard:fix-not-dot-notation
Sep 11, 2026
Merged

platypii merged 1 commit into
hyparam:masterfrom
MaxFreedomPollard:fix-not-dot-notation

Conversation

@MaxFreedomPollard

Copy link
Copy Markdown
Contributor

$not is a no-op on any field addressed with dot notation. matchFilter(record, { 'person.name': { $not: { $eq: 'Ada' } } }) returns true for every row, Ada included.

matchFilter resolves a field path with resolve(), which splits it on .. The $not branch (src/filter.js line 78 on master) then rebuilds a synthetic record as { [field]: value }, using the whole dotted path as a literal object key. On re-entry resolve() splits that key again, looks for record.person.name inside { 'person.name': value }, gets undefined, and compares undefined against the target. The inner match is therefore always false and the negation always true. Flat field names never hit this because they contain no ..

The fix wraps the already-resolved value under a plain key, { value } and { value: target }, which is the shape matchesNull() uses a few lines below in the same file. Nothing else moves: pathsNeededForFilter still reports the dotted path, and canSkipStats still ignores $not, so row-group and page pruning stay as conservative as before.

Verification. On test/files/hyparquet_struct.parquet, parquetQuery({ file, filter: { 'person.name': { $not: { $eq: 'Ada' } } } }) returned all 4 rows before and returns the 3 non-Ada rows after. The new handles $not with dot-notation case in test/filter.test.js fails on master (expected true to be false) and passes with the change. npx vitest run gives 663 passed in 30 files, npm run lint and npx tsc are clean.

matchFilter resolves a field like 'person.name' with resolve(), which splits
the path on '.'. The $not branch in src/filter.js then rebuilt a synthetic
record as { [field]: value }, using the whole dotted path as a literal object
key. On re-entry, resolve() split that key again and looked for
record.person.name inside { 'person.name': value }, found undefined, and
compared that against the target. So every $not condition on a nested field
evaluated to true and $not was a no-op on those columns.

Wrap the already-resolved value under a plain key instead, the same shape
matchesNull() uses a few lines below. Flat field names are unaffected since
they contain no '.'.

@platypii platypii left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch thanks @MaxFreedomPollard ! Change looks good.

@platypii
platypii merged commit ea07519 into hyparam:master Sep 11, 2026
3 checks passed
@platypii

Copy link
Copy Markdown
Collaborator

Published hyparquet 1.30.1 to npm with this fix. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants