Visitar URL original
feat(storage): emit Storage.UploadChunk spans for gRPC appendable uploads by sahusneha2004 · Pull Request #20886 · googleapis/google-cloud-go · GitHub
Skip to content

feat(storage): emit Storage.UploadChunk spans for gRPC appendable uploads - #20886

Draft
sahusneha2004 wants to merge 2 commits into
googleapis:mainfrom
sahusneha2004:storage-appendable-uploadchunk-spans
Draft

sahusneha2004 wants to merge 2 commits into
googleapis:mainfrom
sahusneha2004:storage-appendable-uploadchunk-spans

Conversation

@sahusneha2004

@sahusneha2004 sahusneha2004 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Emit Storage.UploadChunk child spans for gRPC appendable object uploads (BidiWriteObject) when Writer.ChunkSize > 0, matching resumable uploads. This follows up on #20750, which excluded appendable uploads from chunk tracing.

Addresses item 3 of b/581841769.

Changes

storage/grpc_writer.go

  • pickBufferSender now attaches the upload chunk size to the context on both appendable paths (new object and takeover), so attemptStatsHandler installs a grpcUploadChunkTracker.
  • A new chunkingRequested field (params.chunkSize > 0) records whether the application set a positive ChunkSize. w.chunkSize can't be used because it is always at least 256 KiB. w.forceOneShot can't be used either, because gatherFirstBuffer also sets it for small objects. With this field, ChunkSize == 0 stays untraced.

storage/trace.go (grpcUploadChunkTracker)

  • A response that carries the object resource now confirms Resource.Size bytes instead of MaxInt64. This matches completion() in grpc_writer.go. The size is the persisted size: every chunk once the object is finalized, or the bytes persisted so far for an unfinalized appendable object. So chunk spans still in flight are not ended early, for example by the object returned in the first response of an appendable stream. The tracker does not try to detect appendable streams from their first message; it trusts the size for every upload type.
  • A flush or finish_write that carries no data now marks the open chunk span as flushed. This covers Writer.Flush, or Close after all buffered data was already sent.
  • Chunk numbers stay offset/chunkSize + 1. A comment now explains that this is the 1-based index of the chunkSize window holding the chunk's first byte, so spans can share a number after a mid-chunk Flush or an unaligned takeover. gcp.storage.payload.offset tells them apart.
  • Comments updated to mention appendable uploads.

Tests (storage/trace_test.go)

  • appendChunkTraceTestServer is a fake appendable server. The first response on each stream returns the object (size 0, or the takeover offset) with a write handle. Later flushes are acknowledged with persisted_size, and finalizing returns the finalized object.
  • TestAppendableWriterUploadChunkSpans checks:
    • several chunks with finalize
    • a final partial chunk without finalize
    • a payload smaller than one chunk
    • ChunkSize == 0 produces no spans
    • takeover at a non-zero offset, with chunk numbers and offset attributes
    • a retry after UNAVAILABLE mid-upload: the in-flight chunk span ends with the error, and new OK spans with the same numbers are recorded on the next stream
  • TestAppendableWriterUploadChunkSpansHeldAcks (new object and takeover) holds the flush acks while two or more chunks are in flight, after the first-response object has been received. It checks that no chunk span ends while acks are held, and that all end afterwards. It fails if the resource is treated as MaxInt64 again (checked locally with that change applied).
  • TestAppendableWriterUploadChunkSpansFlush checks that an empty Writer.Flush ends the open chunk span before Close, and the numbering after a mid-chunk flush (256 KiB chunks; write 100 KiB, flush, write 300 KiB, close → numbers 1, 1, 2 at offsets 0, 102400, 364544).
  • TestGRPCUploadChunkTrackerResourceAck checks that a resource ack ends only the spans within Resource.Size: unfinalized mid-stream (new object and takeover), finalized appendable, resumable, and a resource that arrives before the first request.
  • TestGRPCUploadChunkTrackerEmptyFlush checks that a flush with no data ends the open span only once it is acknowledged.
  • The expectation of the appendable case of TestWriterTraceWriteModeEmulated changed: it now expects one chunk span. This test needs the emulator and was not run locally.

Validation (local):

  • gofmt -l and goimports -l report no files.
  • go build ./... and go vet . pass.
  • go test -race -short -count=3 -run 'UploadChunk|GRPCUploadChunkTracker|Appendable|PayloadTracing|AttemptStatsHandler' . passes.
  • go test -race -short . in storage passes.

Merge only after the kokoro check is green on the head commit.

Open questions (resolved)

  1. Chunk numbering. Keep offset/chunkSize + 1, as in resumable and HTTP uploads. This is now documented in the code. After a mid-chunk Writer.Flush or an unaligned takeover, spans can repeat a number; the offset attribute tells them apart. Repeats in flush-heavy workloads will be raised with the tracing spec owner.
  2. Small appendable uploads. Accepted as is: with a positive ChunkSize, a small appendable upload emits one UploadChunk span, because it still uses an upload session. A resumable upload of the same size becomes oneshot and is not traced.
  3. Resumable side effect. Accepted as is: an empty finish_write marks the last span as flushed, so for resumable uploads the last span ends when the finalize ack arrives instead of when the stream ends. The timing is effectively the same.
  4. Retries and redirects. Accepted as is: each retry or redirect opens a new stream with its own TagRPC and tracker, so chunk spans are deduplicated only within a stream.

…oads

Record Storage.UploadChunk child spans for gRPC appendable object uploads (BidiWriteObject) when Writer.ChunkSize > 0, matching resumable uploads. Unfinalized appendable flush acks that return the object resource now confirm only Resource.Size bytes instead of completing all in-flight chunk spans, and a flush without data completes the open chunk span.

[Generated-by: AI]
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Nov 11, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds tracing support for appendable uploads in the gRPC storage client, enabling the recording of Storage.UploadChunk spans which were previously only recorded for resumable uploads. It introduces tracking for appendable streams, handles empty flushes, and manages unfinalized object resource acknowledgments within the grpcUploadChunkTracker. Additionally, comprehensive unit tests have been added to verify these tracing behaviors. There are no review comments, so I have no feedback to provide.

Confirm Resource.Size bytes for every response that carries the object
instead of detecting appendable streams from their first message, and
document chunk numbering. Replace requestedChunkSize with
chunkingRequested. Make the fake appendable server return the object
with a write handle only in the first response and acknowledge later
flushes with persisted_size, and add tests for held flush acks, empty
and mid-chunk flushes, and an appendable retry.

[Generated-by: AI]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant