[Feature] Expose the data file being indexed to FileIndexer writers #10314

Search before asking

• I searched in the issues and found nothing similar.

Motivation

PIP-41 proposes file-path-based global indexes to speed up lookups on append tables. File index can offer a complementary capability at file granularity: while a data file is written, a FileIndexWriter receives the indexed value of every row in row order, so an index implementation could produce key -> (data file, row position) entries during index construction and feed them into a cross-file lookup structure. This issue only asks for the missing building block, not for an implementation of PIP-41.

This is not possible today: the writer is created by FileIndexer#createWriter() without knowing which data file it indexes, although DataFileIndexWriter is always constructed right after the data file path is decided.

Current workarounds are fragile:

• passing the path through a ThreadLocal, which depends on internal construction order;
• wrapping the file format to get FileAwareFormatWriter#setFile, which changes the data file extension and requires registering the custom format on every reader;
• resolving files in a CommitCallback, which only sees the files after commit and needs an extra token-to-file indirection.

Paimon already exposes the file location at write time elsewhere: FileAwareFormatWriter#setFile for format writers, and TableWrite#withBlobConsumer (#7074), which reports the uri, offset and length of each blob as it is written.

Solution

Add a small context class and a default method. Existing FileIndexer implementations keep working through the default fallback:

public class FileIndexWriterContext {
    public Path dataFilePath();  // the data file indexed by the writer
    public long schemaId();      // the table schema the data file is written with
}

public interface FileIndexer {
    FileIndexWriter createWriter();

    default FileIndexWriter createWriter(FileIndexWriterContext context) {
        return createWriter();
    }
}

The schema id lets plugins associate these file references with the table schema of the data file, including when the indexes of older files are rebuilt.

DataFileIndexWriter passes the context to all index writers, including nested map index writers. Every call site provides it: RowDataFileWriter, KeyValueDataFileWriter, KeyValueClusteringFileWriter and FileIndexProcessor, so rewrite_file_index also provides it for existing files.

• Built-in indexes and the index file format are unchanged.
• Existing FileIndexer implementations are unaffected; only the internal DataFileIndexWriter constructor and factory methods now take the context.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!