Visitar URL original
feat(bigquery): support maxResults in createQueryArrowStream by ohmayr · Pull Request #9605 · googleapis/google-cloud-node · GitHub
Skip to content

feat(bigquery): support maxResults in createQueryArrowStream - #9605

Merged
ohmayr merged 2 commits into
feat/bigquery-arrow-serialization-optionsfrom
feat/bigquery-arrow-max-results
Oct 21, 2026
Merged

feat(bigquery): support maxResults in createQueryArrowStream#9605
ohmayr merged 2 commits into
feat/bigquery-arrow-serialization-optionsfrom
feat/bigquery-arrow-max-results

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

maxResults was ignored by createQueryArrowStream, so callers always got every row. This enforces it across the inline page and ReadRows.

  • Sends maxResults on jobs.query and keeps it out of jobs.insert.
  • Counts rows across batches, slices the batch that reaches the limit with batch.slice(), cancels ReadRows, and ends the stream.
  • maxResults: 0 emits one 0-row RecordBatch with the query's schema without opening ReadRows, reading the schema from the job's destination table when it isn't known yet.
  • Rejects a maxResults that isn't a non-negative integer before the query runs.

@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 13, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 13, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the maxResults option in createQueryArrowStream to limit the number of returned rows. It implements row tracking, batch slicing, and early stream termination when the limit is reached, accompanied by comprehensive unit and system tests. The review feedback highlights a critical issue where stream.end() is called on a Readable stream, which is non-idiomatic and can cause runtime errors; it is recommended to use stream.push(null) instead to signal the end of the stream.

Comment thread handwritten/bigquery/src/arrow.ts Resolved
Comment thread handwritten/bigquery/src/arrow.ts Resolved
@ohmayr
ohmayr added this pull request to stack #9597 October 13, 2026 07:08
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch 2 times, most recently from 6cc7542 to 575632c Compare October 13, 2026 09:33
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch 2 times, most recently from 4b8d1ac to a256f90 Compare October 13, 2026 10:42
@ohmayr ohmayr mentioned this pull request Oct 13, 2026
Closed
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from a256f90 to 1d7cf4e Compare October 13, 2026 11:30
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch 2 times, most recently from b6b6853 to c678a65 Compare October 13, 2026 19:26
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from c678a65 to 7460922 Compare October 15, 2026 09:55
quirogas
quirogas reviewed Oct 15, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Resolved
Comment thread handwritten/bigquery/system-test/arrow.ts Resolved
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from 7460922 to cdba3c1 Compare October 15, 2026 18:15
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from cdba3c1 to d2a4c5d Compare October 15, 2026 18:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch 3 times, most recently from 4c9c511 to 9cbe84c Compare October 16, 2026 02:19
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch 2 times, most recently from 74f9c54 to 9cbe84c Compare October 16, 2026 21:50
@ohmayr
ohmayr marked this pull request as ready for review October 16, 2026 21:53
@ohmayr
ohmayr requested review from a team as code owners October 16, 2026 21:53
quirogas
quirogas reviewed Oct 20, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Resolved
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from 9cbe84c to 511b61d Compare October 20, 2026 22:23
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from 8d3f151 to b5e5803 Compare October 21, 2026 21:28
quirogas
quirogas approved these changes Oct 21, 2026
View reviewed changes

@quirogas quirogas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment thread handwritten/bigquery/src/arrow.ts
Comment on lines 1016 to 1028
if (!hasRemainingRows || inlineRowsCount > 0n) {
pushBatches(
stream,
decodeArrowPayload(
arrow,
res?.arrowSchema?.serializedSchema,
res?.arrowRecordBatch?.serializedRecordBatch,
limitRows(
ctx,
decodeArrowPayload(
arrow,
res?.arrowSchema?.serializedSchema,
res?.arrowRecordBatch?.serializedRecordBatch,
),
),
);
}

@quirogas quirogas Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: When inlineRowsCount > 0n and limitRows reaches ctx.remainingRows === 0 within the inline first page, returning early before streamDefaultJobStream (e.g., if (!hasRemainingRows || (inlineRowsCount > 0n && ctx.remainingRows === 0) || stream.destroyed) return;) would avoid calling streamDefaultJobStream when a positive maxResults limit was already satisfied inline.

@ohmayr ohmayr Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

streamReadRows already checks ctx.remainingRows !== 0 (src/arrow.ts:584), so ReadRows isn't opened when the inline page satisfies maxResults, and streamDefaultJobStream returns immediately since offset !== 0. Keeping that check in one place inside streamReadRows.

ohmayr added 2 commits October 21, 2026 16:04
@ohmayr
feat(bigquery): support maxResults in createQueryArrowStream
aafd774 
Enforce options.maxResults across inline jobs.query batches and ReadRows
streams by tracking remaining rows, slicing the RecordBatch that reaches
the limit with batch.slice(0, remainingRows), and cancelling the active
ReadRows gRPC stream once the limit is reached. Reject a maxResults that
isn't a non-negative integer before sending the query.

Strip maxResults from the query before building the jobs.insert
configuration and send it as jobs.query maxResults instead. A limit of
0 emits a single 0-row RecordBatch that keeps the schema without
opening ReadRows; when the schema isn't known yet, it's read from the
job's destination table.
@ohmayr
test(bigquery): cover Arrow maxResults on jobs.insert queries
ba64a33 
Add a system test for maxResults on queries that run via jobs.insert:
rows read from streams/_default starting at offset 0, rows read from
an explicit destination table via CreateReadSession, and maxResults 0
returning one 0-row batch that keeps the schema.
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-max-results branch from b5e5803 to ba64a33 Compare October 21, 2026 23:05
@ohmayr
ohmayr merged commit a7fdd05 into main Oct 21, 2026
32 checks passed
@ohmayr
ohmayr deleted the feat/bigquery-arrow-max-results branch October 21, 2026 23:23
@release-please release-please Bot mentioned this pull request Oct 21, 2026
Merged
shivanee-p pushed a commit that referenced this pull request Oct 23, 2026
@release-please
chore: release main (#9684)
b3a9687 
🤖 I have created a release *beep* *boop*
---


<details><summary>common: 8.2.2</summary>

##
[8.2.2](common-v8.2.1...common-v8.2.2)
(2026-10-23)


### Bug Fixes

* **common:** Preserve {{projectId}} token during URI path encoding
([#9668](#9668))
([14b0cc6](14b0cc6))
</details>

<details><summary>gapic-generator: 5.7.2</summary>

##
[5.7.2](gapic-generator-v5.7.1...gapic-generator-v5.7.2)
(2026-10-23)


### Bug Fixes

* **generator:** Compile protos before running tests in generator-tests
([#9701](#9701))
([cde8ae3](cde8ae3))
* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>gaxios: 8.3.2</summary>

##
[8.3.2](gaxios-v8.3.1...gaxios-v8.3.2)
(2026-10-23)


### Bug Fixes

* **gaxios:** Narrow fetch types to call signature for @types/bun
([#9699](#9699))
([2a34528](2a34528))
</details>

<details><summary>bigquery: 9.5.0</summary>

##
[9.5.0](bigquery-v9.4.0...bigquery-v9.5.0)
(2026-10-23)


### Features

* **bigquery:** Allow WRITE_APPEND in createQueryArrowStream
([#9648](#9648))
([2062403](2062403))
* **bigquery:** Resume interrupted Arrow ReadRows streams
([#9602](#9602))
([8372c9e](8372c9e))
* **bigquery:** Stream multi-page Arrow results from default job stream
([#9598](#9598))
([1aa6eb4](1aa6eb4))
* **bigquery:** Support arrowSerializationOptions in
createQueryArrowStream
([#9603](#9603))
([2449fe4](2449fe4))
* **bigquery:** Support explicit destination tables in
createQueryArrowStream
([#9601](#9601))
([f22e2dd](f22e2dd))
* **bigquery:** Support jobs.insert in createQueryArrowStream
([#9600](#9600))
([4c414c6](4c414c6))
* **bigquery:** Support maxResults in createQueryArrowStream
([#9605](#9605))
([a7fdd05](a7fdd05))
* **bigquery:** Wait for unfinished jobs before streaming Arrow results
([#9599](#9599))
([5e465b7](5e465b7))


### Bug Fixes

* **bigquery:** Preserve timestamp nanoseconds when microseconds are
zero
([#9650](#9650))
([f6ca61d](f6ca61d))
* **bigquery:** Raise when an Arrow batch arrives before its schema
([#9681](#9681))
([576a42e](576a42e))
</details>

<details><summary>datastore: 11.3.1</summary>

##
[11.3.1](datastore-v11.3.0...datastore-v11.3.1)
(2026-10-23)


### Bug Fixes

* **datastore:** Align Transaction JSDoc
[@extends](https://github.com/extends) tag with DatastoreRequest
([#9671](#9671))
([733e9f1](733e9f1))
</details>

<details><summary>datamanager: 1.4.0</summary>

##
[1.4.0](datamanager-v1.3.0...datamanager-v1.4.0)
(2026-10-23)


### Features

* Update API sources and regenerate
([#9472](#9472))
([514e2f6](514e2f6))


### Bug Fixes

* Update package.json and .gitignore across packages
([#9378](#9378))
([9c54cec](9c54cec))
</details>

<details><summary>agentidentity: 0.3.1</summary>

##
[0.3.1](agentidentity-v0.3.0...agentidentity-v0.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>agentregistry: 0.6.1</summary>

##
[0.6.1](agentregistry-v0.6.0...agentregistry-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>apptopology: 0.4.1</summary>

##
[0.4.1](apptopology-v0.4.0...apptopology-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>bigquery-storage-api: 0.6.1</summary>

##
[0.6.1](bigquery-storage-api-v0.6.0...bigquery-storage-api-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>bigtable-api: 0.11.1</summary>

##
[0.11.1](bigtable-api-v0.11.0...bigtable-api-v0.11.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>commerceproducer: 0.7.1</summary>

##
[0.7.1](commerceproducer-v0.7.0...commerceproducer-v0.7.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>datastore-api: 0.5.1</summary>

##
[0.5.1](datastore-api-v0.5.0...datastore-api-v0.5.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>firestore-api: 0.7.1</summary>

##
[0.7.1](firestore-api-v0.7.0...firestore-api-v0.7.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>ftp: 0.4.1</summary>

##
[0.4.1](ftp-v0.4.0...ftp-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>logging-api: 0.5.1</summary>

##
[0.5.1](logging-api-v0.5.0...logging-api-v0.5.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>policy-troubleshooter-iam: 0.11.1</summary>

##
[0.11.1](policy-troubleshooter-iam-v0.11.0...policy-troubleshooter-iam-v0.11.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>productregistry: 2.2.1</summary>

##
[2.2.1](productregistry-v2.2.0...productregistry-v2.2.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>spanner-api: 0.6.1</summary>

##
[0.6.1](spanner-api-v0.6.0...spanner-api-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>speech: 8.2.1</summary>

##
[8.2.1](speech-v8.2.0...speech-v8.2.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>tasks: 7.3.1</summary>

##
[7.3.1](tasks-v7.3.0...tasks-v7.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>workloadidentity: 0.4.1</summary>

##
[0.4.1](workloadidentity-v0.4.0...workloadidentity-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>isochrones: 0.3.1</summary>

##
[0.3.1](isochrones-v0.3.0...isochrones-v0.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>inventories: 0.15.1</summary>

##
[0.15.1](inventories-v0.15.0...inventories-v0.15.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>loyaltycustomers: 0.4.1</summary>

##
[0.4.1](loyaltycustomers-v0.4.0...loyaltycustomers-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>reports: 0.18.1</summary>

##
[0.18.1](reports-v0.18.0...reports-v0.18.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>reviews: 0.9.0</summary>

##
[0.9.0](reviews-v0.8.0...reviews-v0.9.0)
(2026-10-23)


### Features

* **google/shopping/merchant/reviews/v1:** Onboard a new library
([#9691](#9691))
([f369d0b](f369d0b))
</details>

<details><summary>storage-control: 0.13.1</summary>

##
[0.13.1](storage-control-v0.13.0...storage-control-v0.13.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@quirogas quirogas quirogas approved these changes
@shivanee-p shivanee-p shivanee-p approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ohmayr @quirogas @shivanee-p