[Bug] Paimon table-valued functions fail with NoSuchMethodError on Spark 4.0 #10396

Search before asking

• I searched in the issues and found no similar issues.

Paimon version

2.0.0. master is affected as well.

Compute Engine

Spark 4.0 (checked against 4.0.3) with paimon-spark-4.0.

Minimal reproduce step

With the Paimon SQL extensions enabled:

CREATE TABLE t (id INT) USING paimon;
INSERT INTO t VALUES 1;
INSERT INTO t VALUES 2;
SELECT * FROM paimon_incremental_query('t', 1, 2);

What doesn't meet your expectations?

The query fails during analysis:

java.lang.NoSuchMethodError: 'scala.Option org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create$default$5()'

Every Paimon table-valued function hits it: paimon_incremental_query, paimon_incremental_between_timestamp, paimon_incremental_to_auto_tag, vector_search, hybrid_search and full_text_search.

Anything else?

Since #7648 the spark4 profile compiles paimon-spark-common against Spark 4.1.2, and paimon-spark-4.0 ships those classes to Spark 4.0. Spark 4.1 added a fifth timeTravelSpec parameter with a default to DataSourceV2Relation.create, so the four-argument calls in PaimonTableValuedFunctions compile to create$default$5() plus the five-argument overload. Spark 4.0 has neither. javap on the released paimon-spark-4.0_2.13-2.0.0.jar shows five such call sites in PaimonTableValuedFunctions$.

The shared TableValuedFunctionsTest only runs in paimon-spark-ut, against the newest Spark in the profile, so CI never runs these functions on 4.0. 1.4.x compiled the shared module against Spark 4.0.2 and does not emit these calls.

Are you willing to submit a PR?

• I'm willing to submit a PR!