• add error_report config to toggle record rejection reporting (#189)
• add orphan and group rejections to spark backend (#174)
• add facility to add additional checks at end of file transforma… (#165)
• add empty_entity check post filters for mandatory entities (#163)
• group rejections for mandatory primary keys (#152)
• add orphan record identification and removal (#150)

• ensure error summary sheet populates entities in deterministic way (#191)
• total no records rejected showing when it shouldn't (#194)
• adjust regex check for timezone offsets, allowing : to be included (#181)
• refactor the join_header method in duckdb to deal with out of bound date values (#177)
• group rejection not filtering out invalid records correctly (#175)
• exit processing for files that fail file transformation (#171)
• ensure that original entity detials persisted in feedback messages for further processing (#168)
• ensure network and load_dtd are disabled in xml reading (#167)
• raise processing error where filter is materialising null values (#162)
• add entity into data contract details to deal with duplicate column definitions (#159)
• add message templating to error messages on orph+group rej (#161)
• ensure missing parent and group rejection records are being removed (#160)
• revert change to SubStats population in error reports (#156)
• change record rej ct to be eq to records processed when file rejected (#154)
• add user feedback message for unsupported file type (#142)

• uppercase all entity names in error report (#190)
• rename of error code and messages related to empty entities post business rules (#187)
• removed ods udfs as they were functionally useless
• remove original entities (#164)
• tweak hierarchy node design to allow iteration both directions on hierarchies (#155)
• unify HierarchyNode and ChildHierarchyNode also tweaked background writer queue type and use in orphaned records implementation (#153)

• cache entities during orphan and group rejections to avoid recomputing multiple times (#172)
• remove orphan tracker as it's causing significant performance degradation (#170)
• only check for orphaned records on entities with at least one record (#166)

• alphanumeric and identifier type (#136)

• inherit is_mandatory with get_type_and_validators
• adjusted the is_type_complex function to work with subclasses of 'simple' types

• upgrade pydantic to v2 (#107)

• ensure that format for times can be accessed and supplied in duckdb casting (#135)

• bump cryptography from 48.0.1 to 50.0.0 (#133)
• bump pymdown-extensions from 10.21.3 to 11.0.1 (#134)

• fix issue with non iso date/datetime values in duck db casting after data contract validations (#129)
• implement non iso timestamp formats in spark. constrain ddb and spark parsers to only cast where format exactly as stipulated (#129)

• bump pyspark to 3.5.5 (#129)

• add record count into error report

• ensure that data contract errors can be assigned an error level (#118)
• default the value for submission failure in the error report
• filter out record that are record rejection and failed the data contract
• adjust error reports to differentiate between submission and record rejection

• add rejected records to sub status object (#122)
• change submission rejection to file rejection in the error reports (#119)

• Update lxml from v4.6.4 to 6.1.1
• Update pyarrow from 17.0.0 to 23.0.1

• add additional fields check into csv readers (#109)

• add greater error handling around polars and duckdb csv reader (#112)
• adjust csv header check feedback message to be more detailed

• ddb xml reader connection args consistent with other ddb readers
• remove default connection from ddb json reader
• remove default connection in csv reader causing hanging in multithreaded setups
• configured refdata loader to be instantiated when required without class vars #99

• add in backend kwargs for readers to allow reader args not determinable at config write time to be passed

• configured refdata loader to be instantiated when required without need for class vars (#98)

• fix failures not being properly audited in foundry pipeline (#94)

• make time format more strict to stop invalid time date flowing (#92)
• enhance duckdb casting to be less permissive of poorly formatted dates and trim whitespace

• integrated duckdb casting into data contract and added initial spark casting

• amend messaging for missing CSV fields check in ddb reader (#83)
• ensure publish audit only captures information about the indivdual submission (#77)

• add extra clarification to the units presented in the file size row of the error report (#81)

• enable passing of original_entity_override to fix instance where id cannot be sourced from current entity

• add row index (#57)
• add option in csv readers to clean and null empty strings (#64)

• add ability to strictly enforce date format in conformatteddate (#70)
• add postcode type to model gen (#69)

• fix issue where formattedtime type not triggering validate correctly (#54)
• fix postcode type to raise error when postcode submitted without space (#53)
• amend arrow file read to use streams
• deal with case sensitivity on file extension derivation (#50)

• ensure that captured errors during business rule evaluation are being captured and logged
• included submission status (with additional processing failure check) in error report population to reduce chance of incorrect status
• issue with si filename handling when the filename contains special chars

• added reference data loading of arrow ipc files including enhanced test coverage for reference data loaders
• Add read of arrow ipc files to reference data loaders
• Change how error messages are generated (by writing in batches). Duckdb no long relies on pandas- use of pyarrow, multiprocessing and background thread batch writing to avoid memory pressure

• removal of processing pool for duckdb data contract
• amended process pool to use executor supplied to duckdb pipeline/data contract rather than always instantiating new pool
• address review comments
• merging develop v06
• merging logging additions from release branch
• merge in main and resolve conflicts and linting issues
• Modified business rule step to write feedback messages in batches to increase tolerance to large files with large numbers of validation errors

• allow passing of custom loggers into pipeline objects
• ensure traceback in broad exceptions
• improve the logging around dve processing errors and align reporting to module name rather than legacy name
• add sense check for text based file (#32)

• deal with pathing assumption that file had been moved to processed_file_path during file transformation

• added entity name override option in data contract error details to align with business rules

• Amend relation to python dictionaries approach as using polars (… (#25)
• fix issue where reporting_entity resulted in key fields being removed from error reports (#23)

• added reporting_period_start and end attribute to submission_info model (#28)
• rename "Grouping" to "Group"
• rename the column headers for elements of the error report

• add persistance of error aggregates to pipeline
• add Foundry pipeline

• issue where templated error messages would not correctly format when passing in parameter values

• include submission status for services passthrough

• new domain type formattedtime for time only data

• small tweak to allow use of dynamic fields in select rules

• ensure dve working on python 3.10
• ensure dve working on python 3.11

• Numerous typing updates that will make this codebase unusable below python 3.9

note - this does not mean the package will work on python 3.9. Minimum working version is 3.10.

• added functionality to allow error messages in business rules t… (#8)

• bump pylint to work correctly with py3.11 and fix numerous linting issues

NB - This was previously v1.0.0 and v1.1.0 but has been rolled back into a 0.1.0 release to reflect lack of package stability.

• Added ability to define custom error codes and templated messages for data contract feedback messages
• Added new JSON readers
• Added SparkCSVReader
• Added PolarsToDuckDBCSVReader and DuckDBCSVRepeatingReader
• Added quotechar option to DuckDBCSVReader

• Fixed issues with refdata loader table implementations
• Fixed duckdb try_cast statements in data contract phase
• Allowed use of entity type in file transformation

• release initial dve source code