Visitar URL original
Fix ADLSFileIO listPrefix FileInfo location. by tom-s-powell · Pull Request #18400 · apache/iceberg · GitHub
Skip to content

Fix ADLSFileIO listPrefix FileInfo location. - #18400

Open
tom-s-powell wants to merge 1 commit into
apache:mainfrom
tom-s-powell:tom-s-powell/fix-adls-location
Open

tom-s-powell wants to merge 1 commit into
apache:mainfrom
tom-s-powell:tom-s-powell/fix-adls-location

Conversation

@tom-s-powell

Copy link
Copy Markdown
Contributor

SupportsPrefixOperations.listPrefix is a generic API whose callers feed each FileInfo.location() straight back into a FileIO (or use it to compare against Iceberg metadata locations). The two other cloud implementations honour this contract by returning fully-qualified URIs:

  • S3FileIO.listPrefix → s3://bucket/key
  • GCSFileIO.listPrefix → gs://bucket/blob

ADLSFileIO.listPrefix, however, was returning the raw PathItem.getName() from the Azure SDK, which is only the container-relative path (e.g. dir/file). That means the location it emits:

  • cannot be re-parsed by ADLSLocation (its regex requires an abfs[s]/wasb[s] scheme, so anything passed back through newInputFile/deleteFile throws a ValidationException),
  • is not round-trippable through deletePrefix the way S3/GCS are,
  • silently misbehaves in generic prefix walkers like FileSystemWalker.listDirRecursivelyWithFileIO and downstream actions like DeleteOrphanFilesSparkAction / RemoveOrphanFilesProcedure, which rely on FileInfo.location() being a usable location string.

What changed

  • ADLSLocation now captures and exposes the URI scheme (abfs, abfss, wasb, wasbs) so it can be reconstituted on the way out.
  • ADLSFileIO.listPrefix reconstructs a fully-qualified URI (<scheme>://<container>@<host>/<path>, or <scheme>://<host>/<path> when no container is present) from the input prefix's scheme/host/container and the PathItem name, bringing it in line with S3FileIO and GCSFileIO.
  • Tests updated to assert the fully-qualified form and to cover ADLSLocation#scheme().

Compatibility

This is a behaviour change for any caller that was relying on the previous container-relative return value. Given the existing behaviour was inconsistent with the interface contract and with every other DelegateFileIO implementation - and would break in-tree callers such as orphan-file cleanup on ADLS - this is a bug fix rather than a semantic break.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants