Glue table Parameters are never pruned: removed table properties persist indefinitely #3977

Activity

1. sungwy

   added

   bugSomething isn't workingSomething isn't working

   on Sep 16, 2026
2. ArulJerald commented on Sep 17, 2026

   

   ArulJerald

   on Sep 17, 2026

   Contributor

   More actions

   I'd like to pick it up and raise the PR.

   @sungwy - Could you please clarify ?

   While working on this I hit a design question worth settling before opening a PR.

   The straightforward fix is to rebuild Parameters from the reserved keys (table_type, metadata_location, previous_metadata_location) plus the current table properties, rather than layering onto whatever Glue already holds. That prunes removed properties correctly.

   It also drops keys PyIceberg never wrote, though. Injecting externally-managed parameters into a moto-backed Glue table and then running an ordinary property commit:

   key	today	with rebuild
   classification	preserved	deleted
   UPDATED_BY_CRAWLER	preserved	deleted
   recordCount	preserved	deleted

   So parameters written by Athena or a Glue crawler would disappear on the next commit.

   The alternative is to prune surgically — remove only keys that were in the previous table properties and are absent from the new ones, leaving anything else alone. Both current_glue_table and current_table.properties are already in scope at the commit_table call site, so it stays small:

   new_parameters = dict(glue_table.get("Parameters", {})) if glue_table else {}
   for key in set(prev_metadata_properties) - set(metadata_properties or {}):
       new_parameters.pop(key, None)
   

   Which behavior do you want?

   1. Full rebuild — Glue Parameters becomes an exact mirror of the Iceberg table properties; anything else isn't PyIceberg's to keep.
   2. Surgical prune — only properties the table itself dropped are removed; externally managed keys survive.

   Happy either way. (2) seemed the safer default, but (1) may well be intended if Parameters is meant to mirror table properties exactly.
3. David-Banquet commented on Oct 5, 2026

   

   David-Banquet

   on Oct 5, 2026

   More actions

   The Java Glue catalog might help with the design question. GlueTableOperations.prepareProperties starts from the Parameters already in Glue and only sets table_type, metadata_location and previous_metadata_location (GlueTableOperations.java#L291-L300). It never copies the Iceberg table properties into Parameters: setTableInputInformation reads them only for the description and the additional locations (IcebergToGlueConverter.java#L245-L264).

   So Java keeps every key it didn't write, like the Athena and crawler keys in @ArulJerald's table, and it has no drift because it doesn't mirror table properties at all. Of the two options, the surgical prune keeps that guarantee. A full rebuild would make PyIceberg drop keys that Java leaves alone on the same table.