Sandbox::https://github.com/GoogleCloudDataproc/spark-bigquery-connector/releases

@AntinPhillips

spark-4.2-bigquery is generally available! Like Spark 4.2, this connector requires at
least Java 17 runtime.
⚠️ Starting the next connector version, 0.47.0, the DataSource V1 connectors
(a.k.a spark-bigquery-with-dependencies) will not be released as part of the regular
connector release.
Issue #1520: Fixed ClassCastException when reading BigQuery DATETIME columns with
readDataFormat=AVRO on Spark 3.4+.
Issue #1525: Fixed count() returning an incorrect value on DataSource
V2 reads when the optimized empty projection path splits more than
Integer.MAX_VALUE rows into a single partition. Thanks @AntinPhillips !
PR #1522: Reading a table that does not exist with the DataSource V2 connector now fails
with Table <name> not found instead of an [INTERNAL_ERROR] NullPointerException,
matching the DataSource V1 behavior.

spark-4.1-bigquery is generally available!
Issue #1501: Preserved detailed row errors from failed direct writes
Issue #1503: Corrected the documented billing project option to parentProject.
PR #1494: Added Support for Change Data Capture (CDC)
PR #1510: Handle Arrow StructVectors with 0 child vectors in ArrowColumnBatchPartitionReaderContext
PR #1512: Fixed a schema-resolution error on Spark 3.5 when reading a nested
STRUCT column with a filter Spark cannot push down (e.g. struct_col IS NOT NULL). The DataSource V2 reader now honors Spark's nested schema pruning and
pushes matching nested selected_fields to the BigQuery Storage Read API.
PR #1512: Kept reader-owned Arrow vectors open between reusable columnar
batches on Spark 4.
BigQuery API has been upgraded to version 2.69.0
BigQuery Storage API has been upgraded to version 3.30.0
GAX has been upgraded to version 2.82.0
gRPC has been upgraded to version 1.83.1
Guava has been upgraded to version 33.6.0-jre
Netty has been upgraded to version 4.2.17.Final
Protocol Buffers has been upgraded to version 4.35.1

Issue #1389: Fixed Timestamp Conversion from Julian to Gregorian conversion
Issue #1476: Fixed BigQuery MERGE statement ambiguity on overwrite when table columns are named target or source.
PR #1479: Fixing NotSerializableException thrown on the executors
PR #1488: Added timing and Job ID logging to write path operations

Added new connector, spark-4.1-bigquery aimed to be used in Spark 4.1. Like Spark 4.1, this connector requires at
least Java 17 runtime. It is currently in preview mode.
spark-4.0-bigquery is generally available!
PR #1445: Add streaming support for Spark DS v2 indirect write.
PR #1452: Improved the performance of the dynamic partition overwrite for RANGE_BUCKET partitioned tables.
PR #1434: Support per-catalog configuration for project and location for BigQueryCatalog.

Added new connector, spark-4.0-bigquery aimed to be used in Spark 4.0. Like Spark 4.0, this connector requires at
least Java 17 runtime. It is currently in preview mode.
PR #1367: Query Pushdown is no longer supported.
PR #1369: Catalog enhancements
PR #1374: Ensure TableId includes project ID if not explicitly set
PR #1376: materializationDataset is now optional to read from views or queries.
PR #1380: Fixed ImpersonatedCredentials serialization
PR #1381: Added the option to set custom credentials scopes
PR #1411: Added Support for SparkSession#executeCommand
BigQuery API has been upgraded to version 2.54.0
BigQuery Storage API has been upgraded to version 3.16.1
GAX has been upgraded to version 2.68.2
gRPC has been upgraded to version 1.74.0

@ddebowczyk92

Releases: GoogleCloudDataproc/spark-bigquery-connector