Visitar URL original
release v0.10.0 (#144) by georgeRobertson · Pull Request #144 · NHSDigital/data-validation-engine · GitHub
Skip to content

release v0.10.0 (#144) - #144

Draft
georgeRobertson wants to merge 47 commits into
mainfrom
release_v010
Draft

release v0.10.0 (#144)#144
georgeRobertson wants to merge 47 commits into
mainfrom
release_v010

Conversation

@georgeRobertson

@georgeRobertson georgeRobertson commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

TLDR of changes

Features

  • feat: added new section within the dischema to store entity hierarchy information
    • feat: add orphan record identification and removal (duckdb and spark)
    • feat: group rejections for mandatory primary keys
    • feat: add facility to add additional checks at end of file transformation stage
  • feat: add empty_entity check post filters for mandatory entities
  • feat: add error_report config to toggle record rejection reporting

Fixes

  • fix: add user feedback message for unsupported file type
  • fix: change record rej ct to be eq to records processed when file rejected
  • fix: add entity into data contract details to deal with duplicate column definitions
  • fix: ensure that original entity details persisted in feedback messages for further processing
  • fix: ensure network and load_dtd are disabled in xml reading
  • fix: exit processing for files that fail file transformation
  • fix: refactor the join_header method in duckdb to deal with out of bound date values
  • fix: ensure that null records are not included in record counts when determining if entity empty
  • fix: adjust regex check for timezone offsets, allowing : to be included
  • fix: total no records rejected showing when it shouldn't
  • fix: ensure error summary sheet populates entities in deterministic way

Refactors

  • refactor: remove original entities
  • refactor: removed ods udfs as they were functionally useless
  • refactor: rename of error code and messages related to empty entities post business rules
  • refactor: uppercase all entity names in error report

Build

  • build: upgrade duckdb to v1.4
  • build: upgrade urllib3 from 2.7 to 2.8
  • build: upgrade typing_extensions from 4.15.0 to 4.16.0
  • build: min python version is now 3.11

Docs

  • docs: added json schema for entity relationships
  • docs: add dev classifier
  • docs: add documentation around entity relationships
  • docs: add error_reports to json schemas

What kind of changes does this PR introduce?

Tick all that apply

  • fix: A bug fix. Correlates with PATCH in SemVer
  • feat: A new feature. Correlates with MINOR in SemVer
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
  • refactor: A code change that neither fixes a bug nor adds a feature
  • perf: A code change that improves performance
  • test: Adding missing or correcting existing tests
  • build: Changes that affect the build system or external dependencies (example scopes: pip, docker, npm)
  • ci: Changes to CI configuration files and scripts (example scopes: GitLabCI)

Please check if the PR fulfills these requirements

  • I have read and followed the Contributing guidance
  • Docs have been added / updated
  • Tests and Linting in the CI are passing
  • Changes have been reviewed and approved by a Project Maintainer

@stevenhsd
feature: add entity hierarchy handling (#143)
de08575 
* feat: added proposals for new models and objects to store entity hierarchy information

* docs: added json schema for entity relationships
@georgeRobertson georgeRobertson changed the title feature: add entity hierarchy handling (#143) release v0.10.0 (#144) Aug 25, 2026
georgeRobertson and others added 28 commits August 27, 2026 11:41
@georgeRobertson
build: upgrade duckdb to v1.4
42fb3e4
@georgeRobertson
docs: add dev classifier
a9c0872
@georgeRobertson
build: upgrade duckdb to v1.4 (#147)
5c1e11b 
build: upgrade duckdb to v1.4
@georgeRobertson
feat: add orphan record identification and removal (#150)
a3cafe7
@stevenhsd
refactor: unify HierarchyNode and ChildHierarchyNode also tweaked ba…
de4e44d 
…ckground writer queue type and use in orphaned records implementation (#153)
@georgeRobertson
fix: add user feedback message for unsupported file type (#142)
516e5e4
@georgeRobertson
fix: change record rej ct to be eq to records processed when file rej…
c2b8cf5 
…ected (#154)
@stevenhsd
refactor: tweak hierarchy node design to allow iteration both directi…
a6b6154 
…ons on hierarchies (#155)
@georgeRobertson
feat: group rejections for mandatory primary keys (#152)
7792d41 
* docs: update jsonschema for entity relationships to include group rej code+message
@georgeRobertson
fix: revert change to SubStats population in error reports (#156)
b63c2ac
@georgeRobertson
build: upgrade java runtime for dev (#158)
a817759
@stevenhsd
fix: ensure missing parent and group rejection records are being remo…
49290bb 
…ved (#160)
@georgeRobertson
fix: add message templating to error messages on orph+group rej (#161)
41b9020
@georgeRobertson
fix: add entity into data contract details to deal with duplicate co…
51a9c07 
…lumn definitions (#159)
@stevenhsd
docs: add entity relationships detail to docs (#157)
3eca2f1
@stevenhsd
fix: raise processing error where filter is materialising null values (…
43c428f 
…#162)
@georgeRobertson
feat: add empty_entity check post filters for mandatory entities (#163)
21043db
@georgeRobertson
refactor: remove original entities (#164)
1a0beb6 
redundant in the grand scheme of things and was increasing the I/O overhead
@georgeRobertson
perf: only check for orphaned records on entities with at least one r…
a51254d 
…ecord (#166)
@georgeRobertson
fix: ensure network and load_dtd are disabled in xml reading (#167)
b3e03b4
@stevenhsd
fix: ensure that original entity detials persisted in feedback messag…
8f5e1ae 
…es for further processing (#168)
@georgeRobertson
docs: update docs around entity relationships for root entities (#169)
e2a9c16 
* build: bump zensical to 0.0.63
@stevenhsd
feat: add facility to add additional checks at end of file transforma… (
2092879 
#165)
@georgeRobertson
perf: remove orphan tracker as it's causing significant performance d…
e4dab57 
…egradation (#170)
@georgeRobertson
fix: exit processing for files that fail file transformation (#171)
9415d2a 
* fix: exit processing xml files that fail XSD

* refactor: add CriticalFeedbackMessage which exits the pipeline asap

number of feedback messages to reduce and removal of specific error code+messages for certain readers to become generic ft_error_code/message
@stevenhsd
perf: cache entities during orphan and group rejections to avoid reco…
acaf522 
…mputing multiple times (#172)
@georgeRobertson
feat: add orphan and group rejections to spark backend (#174)
df8bbe4
@georgeRobertson
fix: group rejection not filtering out invalid records correctly (#175)
a342af7
georgeRobertson and others added 18 commits October 9, 2026 10:43
@georgeRobertson
build: upgrade urllib3 from 2.7 to 2.8 (#178)
ddfdf64
@georgeRobertson
fix: refactor the join_header method in duckdb to deal with out of bo…
a32a01b 
…und date values (#177)
@georgeRobertson
build: bump virtualenv version to 21.7.13 to resolve vulns (#179)
06eaee6
@georgeRobertson
docs: add udfs to docs
5bc3eda 
refactor the advanced docs section
@stevenhsd
* fix: ensure that null records are not included in record counts whe…
72d15be 
…n determining if entity empty (#182)

* test: additional tests with record index for entity empty check functions
@georgeRobertson
docs: fix incorrect text in some of the indexes
562ca2b
@georgeRobertson
refactor: removed ods udfs as they were functionally useless
3c71d18
@georgeRobertson
Merge pull request #183 from NHSDigital/docs/gr-ndsp-683-add_udfs_to_…
986a03b 
…docs

docs: add udfs to docs
@stevenhsd
fix: adjust regex check for timezone offsets, allowing : to be includ…
54f704b 
…ed (#181)
@stevenhsd
refactor: rename of error code and messages related to empty entities…
665e1fa 
… post business rules (#187)
@georgeRobertson
refactor: uppercase all entity names in error report (#190)
17dc21b
@georgeRobertson @vaibhavtiwari16
feat: add error_report config to toggle record rejection reporting (#189
ec64b42 
)

Co-authored-by: vaibhavtiwari16 <249828730+vaibhavtiwari16@users.noreply.github.com>
@georgeRobertson
build: update various packages for security patching (#193)
2dc7a48
@georgeRobertson
fix: total no records rejected showing when it shouldn't
56be7c8
@georgeRobertson
docs: add error_reports to json schemas
910fc14
@georgeRobertson
fix: total no records rejected showing when it shouldn't (#194)
101d309
@stevenhsd
fix: ensure error summary sheet populates entities in deterministic w…
4652abe 
…ay (#191)
@georgeRobertson
docs: update docs related to v0.10.0 release (#195)
3a7994a 
* docs: update docs related to v0.10.0 release

* docs: fix typo in upperbound python version for v0.10.0
@sonarqubecloud

sonarqubecloud Bot commented Oct 29, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
30 New issues
2 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@georgeRobertson @stevenhsd