paimon-eslib provides the es-index global index: a Lucene-backed, multi-column, multi-modal index. A single es-index file indexes a primary column (typically a vector) together with optional companion columns — full-text, keyword, scalar, and date — so vector search and full-text search can be served from the same index, and companion columns stay available for scoring and filtering inside one index definition.

Unlike the single-purpose vector and full-text global indexes — each of which indexes one column with one dedicated engine — es-index groups a primary column plus optional companion columns into one index. The primary column determines the main modality (usually a vector); companion columns are indexed according to their data type and per-field options.

Plugin required. es-index is provided by this paimon-eslib module. Make sure paimon-eslib is on the classpath of your Spark or Flink job (and of any reader) before building or querying an es-index. The module and its ESLib/Lucene dependencies require Java 11 or newer. It is excluded from the default Maven reactor; build it explicitly with -Ppaimon-eslib on JDK 11 or newer.

paimon-eslib is distributed as a thin JAR and does not embed ESLib or Lucene classes. Maven and Gradle resolve these dependencies transitively. When installing JARs manually, place eslib-core-lucene9 and eslib-simdvec-lucene9 1.0.7, together with their Lucene 9.12 dependencies, in the same runtime classloader as paimon-eslib.

• ESLib source code: CrownChu/es-paimon-lib
• Published ESLib JARs: CrownChu/es-paimon-lib-releases

The ESLib source and binary repositories are maintained independently of the Apache Software Foundation. The exact ESLib artifact versions used by this module are declared in pom.xml.

See the general Global Index documentation for the required Data Evolution table properties, coverage/freshness behavior, and shared build options.

For each indexed column, es-index decides the sub-index type from the column data type, unless you override it with fields.<field>.type:

Text columns always provide both capabilities. A fulltext primary field gets a <column>.keyword exact-match sub-field; a keyword primary field gets a <column>.fulltext analyzed sub-field.

Create a Data Evolution table with the required properties:

CREATE TABLE my_table (
    id INT,
    embedding ARRAY<FLOAT>,
    content STRING,
    category STRING,
    price INT
) TBLPROPERTIES (
    'bucket' = '-1',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'global-index.enabled' = 'true'
);

Build the index with sys.create_global_index and index_type => 'es-index'. To index several columns in one es-index, pass a comma-separated index_column list: the first column is the primary column, the rest are companion columns. Per-column behavior is configured through options under the global-index.es-index. prefix.

-- Single vector column with an HNSW index
CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=hnsw,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine'
);

-- Multi-column: vector (primary) + full-text + keyword + scalar in one index.
-- 'content' becomes full-text because an analyzer is set; 'category' stays keyword; 'price' scalar.
CALL sys.create_global_index(
    table => 'db.my_table',
    index_column => 'embedding,content,category,price',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=diskbbq,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine,global-index.es-index.fields.content.analyzer=standard,global-index.es-index.fields.category.type=keyword'
);

-- Single vector column with an HNSW index
CALL sys.create_global_index(
    `table` => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=hnsw,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine'
);

-- Multi-column: vector (primary) + full-text + keyword + scalar in one index.
CALL sys.create_global_index(
    `table` => 'db.my_table',
    index_column => 'embedding,content,category,price',
    index_type => 'es-index',
    options => 'global-index.es-index.fields.embedding.algorithm=diskbbq,global-index.es-index.fields.embedding.dimension=768,global-index.es-index.fields.embedding.metric=cosine,global-index.es-index.fields.content.analyzer=standard,global-index.es-index.fields.category.type=keyword'
);

You can also set the same options in TBLPROPERTIES at table-creation time instead of passing them per build. Options are resolved with field-level keys taking precedence over index-type-level keys: global-index.es-index.fields.<field>.<key> overrides global-index.es-index.<key>.

Index-type-level keys use the global-index.es-index.<key> prefix and apply to every column of that kind; field-level keys use global-index.es-index.fields.<field>.<key> and override them for one column.

es-index implements the same search API as the other global indexes, so querying does not depend on which engine built the index — only the build step differs.

-- Vector search: top-5 nearest neighbors on the primary vector column
SELECT * FROM vector_search('my_table', 'embedding', array(1.0f, 2.0f, 3.0f), 5);

-- Full-text search on a companion full-text column, using the JSON query DSL
SELECT * FROM full_text_search(
    'my_table',
    'content',
    '{"match":{"query":"paimon lake format"}}',
    10
);

Vector routes and full-text routes over an es-index can also be combined with the hybrid_search(...) table-valued function.

Flink exposes vector search as a CALL procedure. The procedure returns JSON-serialized rows as strings.

-- Vector search: top-5 nearest neighbors on the primary vector column
CALL sys.vector_search(
    `table` => 'db.my_table',
    vector_column => 'embedding',
    query_vector => '1.0,2.0,3.0',
    top_k => 5
);

-- Optionally restrict the returned columns
CALL sys.vector_search(
    `table` => 'db.my_table',
    vector_column => 'embedding',
    query_vector => '1.0,2.0,3.0',
    top_k => 5,
    projection => 'id,content'
);

Note. Flink SQL currently exposes vector search only. For full-text search and hybrid search over an es-index, use Spark SQL (the full_text_search / hybrid_search table-valued functions) or the Java API (Table.newFullTextSearchBuilder() / Table.newHybridSearchBuilder()).

CALL sys.drop_global_index(
    table => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index'
);

CALL sys.drop_global_index(
    `table` => 'db.my_table',
    index_column => 'embedding',
    index_type => 'es-index'
);